Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

2016-10-20 Thread Lefty Leverenz 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/#review153357
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 1766 - 1772)


Looks good, thanks.  Just one more nit (on 2 lines):  Zookeeper -> 
ZooKeeper.

See https://zookeeper.apache.org if you don't believe me.  ;)


- Lefty Leverenz


On Oct. 19, 2016, 11:50 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52923/
> ---
> 
> (Updated Oct. 19, 2016, 11:50 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, 
> namit jain, Sergio Pena, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-14979
> https://issues.apache.org/jira/browse/HIVE-14979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding a new configuration option to HiveConf to signal whether stale lock 
> removal is requested on startup.
> Adding a new method to ZooKeeperHiveLockManager to remove stale locks
> Modifying the HiveServer2 to instantiate a lock manager and call the new 
> method if defined by the configuration.
> 
> Please take extra care when reviewing these:
> - Modifying the lock fetching method to use the clientIp from the lock, and 
> not update with the current ip - Not sure why it was done before
> - Instantiation of the lock manager - I might not chose the best method for it
> 
> Open for any suggestions :)
> 
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
>  14d0ef4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 
> 
> Diff: https://reviews.apache.org/r/52923/diff/
> 
> 
> Testing
> ---
> 
> Created 2 unit test cases:
> - Removing own locks
> - Not removing other server's locks
> 
> Manually tested the Lock manager instantiation method on HiveServer2
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Review Request 53021: HIVE-14984: Hive-WebUI access results in Request is a replay (34) attack

2016-10-20 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53021/
---

Review request for hive, Aihua Xu and Mohit Sabharwal.


Repository: hive-git


Description
---

HIVE-14984: Hive-WebUI access results in Request is a replay (34) attack


Diffs
-

  common/src/java/org/apache/hive/http/HttpServer.java 
c4e2e33c6627be979daec5e7afa2ed82a039dde0 
  service/src/resources/hive-webapps/hiveserver2/index.html 
f18ba53e91518379b2f08a096fe08be899b293e3 
  service/src/test/org/apache/hive/service/server/TestHS2HttpServer.java 
c9e0ac3a751b8824224bda4c5a0487d286ab069a 

Diff: https://reviews.apache.org/r/53021/diff/


Testing
---

Tested that in a secured cluster the hiveserver2.jsp is still only accessible 
with correct credentials.
Tested that the replay attack is not triggered when the context root is called.
Added unit test comparing the contents of the html response for a query 
requesting the context root and the jsp.


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-15024) LLAP: ClassCastException: org.apache.hadoop.hive.common.io.DiskRangeList cannot be cast to org.apache.orc.impl.BufferChunk

2016-10-20 Thread Gopal V (JIRA) 
Gopal V created HIVE-15024:
--

 Summary: LLAP: ClassCastException: 
org.apache.hadoop.hive.common.io.DiskRangeList cannot be cast to 
org.apache.orc.impl.BufferChunk
 Key: HIVE-15024
 URL: https://issues.apache.org/jira/browse/HIVE-15024
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 2.2.0
Reporter: Gopal V
Priority: Critical


{code}
Caused by: java.io.IOException: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.io.DiskRangeList cannot be cast to 
org.apache.orc.impl.BufferChunk
at 
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:383)
at 
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:338)
at 
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:278)
at 
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 23 more
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.io.DiskRangeList cannot be cast to 
org.apache.orc.impl.BufferChunk
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.prepareRangesForCompressedRead(EncodedReaderImpl.java:728)
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:616)
at 
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:397)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:424)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:227)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:224)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:224)
at 
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:93)
... 6 more

2016-10-20T00:48:45,354 WARN  [TezTaskRunner 
(1475017598908_0410_15_00_20_0)] 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask: Ignoring exception when 
closing input part(cleanup). Exception class
=java.io.IOException, message=java.lang.ClassCastException: 
org.apache.hadoop.hive.common.io.DiskRangeList cannot be cast to 
org.apache.orc.impl.BufferChunk
2016-10-20T00:48:45,416 WARN  [TaskHeartbeatThread 
(1475017598908_0410_15_00_20_0)] 
org.apache.hadoop.hive.llap.daemon.impl.LlapTaskReporter: Exiting TaskReporter 
thread with pending queue size=2
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Start HiveServer2 with Kerberos meet FATAL error

2016-10-20 Thread Vivek Shrivastava 
make sense.. glad to know that it has resolved your problem.

On Thu, Oct 20, 2016 at 12:11 AM, Micro dong  wrote:

> It's done.
> Because  hadoop client which hive use didn't integrated with Kerberos.
> thanks for your  suggestion.
>
> 2016-10-19 11:03 GMT+08:00 Vivek Shrivastava :
>
> > How are you starting the service? Do you have Hadoop setup done and
> > integrated with Kerberos?
> >
> > On Tue, Oct 18, 2016 at 10:33 PM, Micro dong 
> > wrote:
> >
> > > It seems  to me it's trying to login into HiveServer with unix username
> > > instead of  Kerberos keytab
> > >
> > > 2016-10-19 10:30 GMT+08:00 Micro dong :
> > >
> > > > it also meet the same error too.
> > > >
> > > > *2016-10-19 10:28:07,300 FATAL [Thread-9]: thrift.ThriftCLIService
> > > > (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2:
> > > could
> > > > not start ThriftBinaryCLIService*
> > > > *javax.security.auth.login.LoginException: Kerberos principal should
> > > have
> > > > 3 parts: work*
> > > > * at
> > > > org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(
> > > HiveAuthFactory.java:147)*
> > > > * at
> > > > org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
> > > run(ThriftBinaryCLIService.java:58)*
> > > > * at java.lang.Thread.run(Thread.java:722)*
> > > >
> > > > 2016-10-19 10:22 GMT+08:00 Vivek Shrivastava <
> vivshrivast...@gmail.com
> > >:
> > > >
> > > >> This looks good. Please run this one and then try starting the
> > service.
> > > >>
> > > >> kinit -kt /home/work/software/hive/conf/hive.keytab  hive2/
> > > >> temp...@hadoop.com
> > > >>
> > > >>
> > > >> On Tue, Oct 18, 2016 at 10:16 PM, Micro dong <
> microle.d...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > [work@tempt48 ~]$ klist -ket /home/work/software/hive/conf/
> > > hive.keytab
> > > >> > Keytab name: FILE:/home/work/software/hive/conf/hive.keytab
> > > >> > KVNO Timestamp Principal
> > > >> >  -
> > > >> > 
> > > >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com
> > > >> (aes128-cts-hmac-sha1-96)
> > > >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des3-cbc-sha1)
> > > >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (arcfour-hmac)
> > > >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-hmac-sha1)
> > > >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-cbc-md5)
> > > >> > [work@tempt48 ~]$ hostname
> > > >> > tempt48
> > > >> > [work@tempt48 ~]$ hostname -f
> > > >> > tempt48
> > > >> >
> > > >> >
> > > >> > 2016-10-18 17:48 GMT+08:00 Vivek Shrivastava <
> > > vivshrivast...@gmail.com
> > > >> >:
> > > >> >
> > > >> > > What is the output of
> > > >> > > klist -ket /home/work/software/hive/conf/hive.keytab
> > > >> > > and
> > > >> > > hostname
> > > >> > > and
> > > >> > > hostname -f
> > > >> > >
> > > >> > > On Tue, Oct 18, 2016 at 4:18 AM, Micro dong <
> > microle.d...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > tempt48  is the local
> machine's
> > > >> > hostname
> > > >> > > > which is configured in /etc/hosts.
> > > >> > > >
> > > >> > > > 2016-10-18 15:33 GMT+08:00 Chen, Junjie J <
> > > junjie.j.c...@intel.com
> > > >> >:
> > > >> > > >
> > > >> > > > > Please try to use full qualified name of host,  such as
> hive2/
> > > >> > > > > tempt48.hadoop@hadoop.com
> > > >> > > > >
> > > >> > > > > -Original Message-
> > > >> > > > > From: Micro dong [mailto:microle.d...@gmail.com]
> > > >> > > > > Sent: Tuesday, October 18, 2016 2:37 PM
> > > >> > > > > To: dev@hive.apache.org
> > > >> > > > > Subject: Re: Start HiveServer2 with Kerberos meet FATAL
> error
> > > >> > > > >
> > > >> > > > > I follow your suggestion and it also meet the same error
> too.
> > > >> > > > >
> > > >> > > > > *start hiveservice*
> > > >> > > > > /home/work/software/hive/bin/hive --service hiveserver2
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *klist output*
> > > >> > > > > Ticket cache: FILE:/tmp/krb5cc_31100
> > > >> > > > > Default principal: hive2/temp...@hadoop.com
> > > >> > > > >
> > > >> > > > > Valid starting ExpiresService principal
> > > >> > > > > 10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/
> > > >> hadoop@hadoop.com
> > > >> > > renew
> > > >> > > > > until 10/28/16 14:23:04
> > > >> > > > >
> > > >> > > > > 2016-10-18 12:07 GMT+08:00 Vivek Shrivastava <
> > > >> > vivshrivast...@gmail.com
> > > >> > > >:
> > > >> > > > >
> > > >> > > > > > can you please change this line
> > > >> > > > > >
> > > >> > > > > > hive2/_h...@hadoop.com *
> > with
> > > >> > > > > > hive2/_h...@hadoop.com*
> > > >> > > > > >
> > > >> > > > > > How are you starting the service, What is the klist
> output?
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Mon, Oct 17, 2016 at 11:09 PM, Micro dong <
> > > >> >

Re: Review Request 53021: HIVE-14984: Hive-WebUI access results in Request is a replay (34) attack

2016-10-20 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53021/#review153392
---


Ship it!




LGTM

Nicely done :)

- Peter Vary


On Oct. 20, 2016, 9:47 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53021/
> ---
> 
> (Updated Oct. 20, 2016, 9:47 a.m.)
> 
> 
> Review request for hive, Aihua Xu and Mohit Sabharwal.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14984: Hive-WebUI access results in Request is a replay (34) attack
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hive/http/HttpServer.java 
> c4e2e33c6627be979daec5e7afa2ed82a039dde0 
>   service/src/resources/hive-webapps/hiveserver2/index.html 
> f18ba53e91518379b2f08a096fe08be899b293e3 
>   service/src/test/org/apache/hive/service/server/TestHS2HttpServer.java 
> c9e0ac3a751b8824224bda4c5a0487d286ab069a 
> 
> Diff: https://reviews.apache.org/r/53021/diff/
> 
> 
> Testing
> ---
> 
> Tested that in a secured cluster the hiveserver2.jsp is still only accessible 
> with correct credentials.
> Tested that the replay attack is not triggered when the context root is 
> called.
> Added unit test comparing the contents of the html response for a query 
> requesting the context root and the jsp.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-15026) Option to not merge the views

2016-10-20 Thread Carlos Martinez Moller (JIRA) 
Carlos Martinez Moller created HIVE-15026:
-

 Summary: Option to not merge the views
 Key: HIVE-15026
 URL: https://issues.apache.org/jira/browse/HIVE-15026
 Project: Hive
  Issue Type: Improvement
  Components: Logical Optimizer, Physical Optimizer
Reporter: Carlos Martinez Moller


Note: I am trying to simplify a real case scenario we are having, but 
simplifying the queries for the example. Hope they make sense and that the 
problem itself can be explained. The real query is a lot more complex and long.

When performing a query of this type:

--
SELECT COLUMNA, COLUMNB, MAX (COLUMNC)
FROM TABLE_A
WHERE COLUMNA=1 AND COLUMND='Case 1'
UNION ALL
SELECT COLUMNA, COLUMNB, MAX (COLUMNC)
FROM TABLE_A
WHERE COLUMNA=10 AND COLUMNE='Case 2'
--

This creates Three Stages. First Stage is FULL SCAN of TABLE_A + Filter 
(COLUMNA=1/COLUMND='Case 1'),  Second Stage is FULL SCAN of TABLE_A again + 
Filter (COLUMNA=10/COLUMNE='Case 2'), and third stage is the UNION ALL.

TABLE_A has 2TB data of information.
But COLUMNA=1 and COLUMNA=10 filter all together only 2GB of information.

So I thought to use:

--
WITH TEMP_VIEW AS
(SELECT COLUMNA,COLUMNB,COLUMNC,COLUMND
FROM TABLE_A
WHERE COLUMNA=1 AND COLUMNA=10)
SELECT COLUMNA, COLUMNB, MAX (COLUMNC)
FROM TEMP_VIEW
WHERE COLUMNA=1 AND COLUMND='Case 1'
UNION ALL
SELECT COLUMNA, COLUMNB, MAX (COLUMNC)
FROM TEMP_VIEW
WHERE COLUMNA=10 AND COLUMNE='Case 2'
---

I thought that with this it would create 4 Stages:
- Stage 1: Full Scan of TABLE_A and generate intermediate data
- Stage 2: In the data of Stage 1 Filter (COLUMNA=1/COLUMND='Case 1')
- Stage 3: In the data of Stage 1 Filter (COLUMNA=10/COLUMNE='Case 2')
- Stage 4: Union ALL

With this instead of 4TB being read from disk, only 2TB+4GB (twice going 
through the view) would be read (In our case complexity is even bigger and we 
will be saving 20TB reading)

But it does the same than in the original query. It internally pushes the 
predicates of the "WITH" query in the two parts of the UNION.

It would be good to have a control on this, or for the optimizer to choose the 
best approach using histogram/statistics information.

For those knowing Oracle RDBMS this is equivalent to the MERGE/NO_MERGE and 
NEST behaviour:
http://www.dba-oracle.com/t_hint_no_merge.htm as an explanation...

Other approaches for my example could apply, as partitioning by COLUMNA of 
BUCKETING. But are not applicable in our case as COLUMNA is not commonly used 
when accessing this table.

The point of this JIRA is to add a functionality similar to the one of Oracle 
(not Merging the query, but generating an in-memory/disk temporary view) both 
for "WITH" clauses and VIEWS.

This is very very commonly used in Data Ware Houses managing big amounts of 
data and provides big performance benefits.







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15025) Secure-Socket-Layer (SSL) support for HMS

2016-10-20 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-15025:
---

 Summary: Secure-Socket-Layer (SSL) support for HMS
 Key: HIVE-15025
 URL: https://issues.apache.org/jira/browse/HIVE-15025
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Affects Versions: 2.2.0
Reporter: Aihua Xu
Assignee: Aihua Xu


HMS server should support SSL encryption. When the server is keberos enabled, 
the encryption can be enabled. But if keberos is not enabled, then there is no 
encryption between HS2 and HMS. 

Similar to HS2, we should support encryption in both cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52493: HIVE-13589 : beeline support prompt for password with '-p' option

2016-10-20 Thread Vihang Karajgaonkar 


> On Oct. 9, 2016, 1:42 a.m., cheng xu wrote:
> > beeline/src/java/org/apache/hive/beeline/Commands.java, line 1059
> > 
> >
> > This error msg is not user-friendly. Can you give some potential reason 
> > for such NPE like the comments do?

I think we should throw a RuntimeException here since there is nothing that 
Beeline can do here. I have updated the error message and change it to throwing 
RuntimeException. Let me know if you have any thoughts.


> On Oct. 9, 2016, 1:42 a.m., cheng xu wrote:
> > beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java, line 
> > 114
> > 
> >
> > Do we have to change the original format with "key=value;" pair?

Yes, this is required since now the password be optional. The connect(String 
line) method expects connect[driver] where the 
driver value can be present or not. If the password is also optional there is 
no way to distinguish between connectand  
  case. That was the reason I changed the implementation of 
constructCmd method in the first version of the patch.

Now, I have kept the original implementation of the method and added a new 
method which gets called only when -p is used without providing a password. 
This limits the scope of the change and also takes care of both the cases. Let 
me know if you have any thoughts.


> On Oct. 9, 2016, 1:42 a.m., cheng xu wrote:
> > beeline/src/test/resources/hive-site.xml, line 48
> > 
> >
> > Do we need this change?

Yes, this property is used by the testcases to add dummy data in test tables. 
This is similar to what we do in other modules too.


> On Oct. 9, 2016, 1:42 a.m., cheng xu wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeelinePasswordOption.java,
> >  line 59
> > 
> >
> > Why we need this configuration?

removed it.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52493/#review151931
---


On Oct. 20, 2016, 6:02 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52493/
> ---
> 
> (Updated Oct. 20, 2016, 6:02 p.m.)
> 
> 
> Review request for hive, cheng xu and Sergio Pena.
> 
> 
> Bugs: HIVE-13589
> https://issues.apache.org/jira/browse/HIVE-13589
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13589 : beeline support prompt for password with '-p' option
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 
> fdbe0a2c6c37688ab511710bd43517908996f158 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 
> 6c3e7f70c811ad81aa8f3ee2ec2e42d98da7d330 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeelinePasswordOption.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52493/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

[jira] [Created] (HIVE-15027) make sure export takes MM information into account

2016-10-20 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-15027:
---

 Summary: make sure export takes MM information into account
 Key: HIVE-15027
 URL: https://issues.apache.org/jira/browse/HIVE-15027
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52493: HIVE-13589 : beeline support prompt for password with '-p' option

2016-10-20 Thread Vihang Karajgaonkar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52493/
---

(Updated Oct. 20, 2016, 6:02 p.m.)


Review request for hive, cheng xu and Sergio Pena.


Changes
---

Added review comments and made minor changes after rebasing to the latest 
changes.


Summary (updated)
-

HIVE-13589 : beeline support prompt for password with '-p' option


Bugs: HIVE-13589
https://issues.apache.org/jira/browse/HIVE-13589


Repository: hive-git


Description (updated)
---

HIVE-13589 : beeline support prompt for password with '-p' option


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 
fdbe0a2c6c37688ab511710bd43517908996f158 
  beeline/src/java/org/apache/hive/beeline/Commands.java 
6c3e7f70c811ad81aa8f3ee2ec2e42d98da7d330 
  
itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeelinePasswordOption.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/52493/diff/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 52079: HIVE-14753: Track the number of open/closed/abandoned sessions in HS2

2016-10-20 Thread Szehon Ho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52079/#review153432
---


Ship it!




Sorry for the delay, lgtm

- Szehon Ho


On Oct. 17, 2016, 10:57 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52079/
> ---
> 
> (Updated Oct. 17, 2016, 10:57 a.m.)
> 
> 
> Review request for hive, Gabor Szadovszky, Peter Vary, and Sergio Pena.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14753: Track the number of open/closed/abandoned sessions in HS2
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 
> ba2267b29eaeb497e9582ca3ff3de4ad63bf8482 
>   common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 
> 9b263d91321f141adf73e5421f3d659f80081471 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
>  c9d4087c5aeaf7f1ca0e9b5860b898b5766cd911 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
>  9525b452eada8ab6a2dd56895f9c9d4eb50db894 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/MetricVariableRatioGauge.java
>  PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java 
> 46676589e6656d0f13f1931bfe67a63dd1920042 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestMetricVariableRatioGauge.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 76e759f23fdbc60629397121fd2522e6e9251a85 
>   
> service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
>  82126c053f13309013035c6f900bd5d37170bdae 
> 
> Diff: https://reviews.apache.org/r/52079/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests in the common and the ql subprojects.
> Manually verified the metrics using the HS2 webui metric dump.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

multi-insert and dynamic partitioning

2016-10-20 Thread Sergey Shelukhin 
Should we disable dynamic partitioning for multi-insert when one table is
mentioned several times?


create table intermediate (key int, p int);

.. insert some data
create table test (key int, key2 int) partitioned by (p int);


This is explicitly not allowed:
from intermediate
insert into table test partition(p=1) select p, key
insert into table test partition(p=1) select key, p;


But this is allowed (as is regular dynamic partitioning that could result
in the same effect without us knowing):
from intermediate
insert into table test partition(p) select p, key, 1
insert into table test partition(p=1) select key, p;

You could even do this:
from intermediate
insert overwrite table test partition(p) select p, key, 1
insert overwrite table test partition(p=1) select key, p;

What the last one SHOULD produce I do not know.

Re: Review Request 52981: HIVE-14679: csv2/tsv2 output format disables quoting by default and it's difficult to enable

2016-10-20 Thread Jianguo Tian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52981/
---

(Updated Oct. 21, 2016, 12:48 a.m.)


Review request for hive and cheng xu.


Bugs: HIVE-14679
https://issues.apache.org/jira/browse/HIVE-14679


Repository: hive-git


Description (updated)
---

1. Enabled double quoting by default for csv2, tsv2 and dsv.
2. Disabling quoting using beeline argument --outputformat.
3. No quote character at all when disable quoting.


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 79922d2 
  beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 57b9c46 
  beeline/src/java/org/apache/hive/beeline/Commands.java 039e354 
  beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java 
66d9fd0 
  beeline/src/main/resources/BeeLine.properties ad79c01 
  beeline/src/test/org/apache/hive/beeline/TestOutputFormatWithQuotes.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/52981/diff/


Testing (updated)
---

Ran JUnit test to validate csv2,tsv2,dsv,csv and tsv output format with 
enabling and disabling double quotes.


Thanks,

Jianguo Tian

Re: Review Request 52493: HIVE-13589 : beeline support prompt for password with '-p' option

2016-10-20 Thread cheng xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52493/#review153482
---



Thank you for updating the patch. LGTM overall and just leave some minor 
comments.


beeline/src/java/org/apache/hive/beeline/BeeLine.java (line 680)


hint: perfer HIVE_VAR_PREFIX.equals(arg)



beeline/src/java/org/apache/hive/beeline/BeeLine.java (line 926)


2 space indent



beeline/src/java/org/apache/hive/beeline/BeeLine.java (line 946)


Let's make it simple as following:
return url.indexOf('/', "jdbc:hive2://".length() < 0;



itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeelinePasswordOption.java
 (line 48)


Please move it before Line 45.



itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeelinePasswordOption.java
 (line 69)


hint: better to call "testPromptPasswordOptionAsFirst" in consist with the 
later one.



itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeelinePasswordOption.java
 (line 243)


2 space indent


- cheng xu


On Oct. 21, 2016, 2:02 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52493/
> ---
> 
> (Updated Oct. 21, 2016, 2:02 a.m.)
> 
> 
> Review request for hive, cheng xu and Sergio Pena.
> 
> 
> Bugs: HIVE-13589
> https://issues.apache.org/jira/browse/HIVE-13589
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13589 : beeline support prompt for password with '-p' option
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 
> fdbe0a2c6c37688ab511710bd43517908996f158 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 
> 6c3e7f70c811ad81aa8f3ee2ec2e42d98da7d330 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeelinePasswordOption.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52493/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 52981: HIVE-14679: csv2/tsv2 output format disables quoting by default and it's difficult to enable

2016-10-20 Thread Jianguo Tian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52981/
---

(Updated Oct. 21, 2016, 1:10 a.m.)


Review request for hive and cheng xu.


Bugs: HIVE-14679
https://issues.apache.org/jira/browse/HIVE-14679


Repository: hive-git


Description (updated)
---

1. Enabled double quoting by default for csv2, tsv2 and dsv.
2. Disabling quoting using beeline argument --disablequoting.
3. No quote character at all when disable quoting.


Diffs
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 79922d2 
  beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 57b9c46 
  beeline/src/java/org/apache/hive/beeline/Commands.java 039e354 
  beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java 
66d9fd0 
  beeline/src/main/resources/BeeLine.properties ad79c01 
  beeline/src/test/org/apache/hive/beeline/TestOutputFormatWithQuotes.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/52981/diff/


Testing
---

Ran JUnit test to validate csv2,tsv2,dsv,csv and tsv output format with 
enabling and disabling double quotes.


Thanks,

Jianguo Tian

Re: Review Request 51755: Support Intersect Except

2016-10-20 Thread pengcheng xiong 


> On Oct. 18, 2016, 11:09 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java,
> >  line 956
> > 
> >
> > Does this need to be public?

Yes, it is used by two rules and it may be reused in the future.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51755/#review153186
---


On Oct. 20, 2016, 6:12 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51755/
> ---
> 
> (Updated Oct. 20, 2016, 6:12 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12765
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 09833ff 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java f308832 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> c527e58 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> cf93ed8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExcept.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectMergeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectOverIntersectRemoveRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
>  3ec9dac 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  63aa086 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java d32a0a7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 5d3fa6a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 5c16c55 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 50987c3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBExpr.java cccf0f6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 9d58193 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
>  a427803 
>   ql/src/test/queries/clientpositive/except_all.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/except_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_all.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_merge.q PRE-CREATION 
>   ql/src/test/results/clientpositive/except_all.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/except_distinct.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/intersect_all.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/intersect_distinct.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/intersect_merge.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51755/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 51755: Support Intersect Except

2016-10-20 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51755/
---

(Updated Oct. 20, 2016, 6:12 p.m.)


Review request for hive and Ashutosh Chauhan.


Summary (updated)
-

Support Intersect Except


Repository: hive-git


Description
---

HIVE-12765


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 09833ff 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java f308832 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
c527e58 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
cf93ed8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExcept.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectMergeRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectOverIntersectRemoveRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
 3ec9dac 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
 63aa086 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java d32a0a7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 5d3fa6a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 5c16c55 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 50987c3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBExpr.java cccf0f6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 9d58193 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
 a427803 
  ql/src/test/queries/clientpositive/except_all.q PRE-CREATION 
  ql/src/test/queries/clientpositive/except_distinct.q PRE-CREATION 
  ql/src/test/queries/clientpositive/intersect_all.q PRE-CREATION 
  ql/src/test/queries/clientpositive/intersect_distinct.q PRE-CREATION 
  ql/src/test/queries/clientpositive/intersect_merge.q PRE-CREATION 
  ql/src/test/results/clientpositive/except_all.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/except_distinct.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/intersect_all.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/intersect_distinct.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/intersect_merge.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/51755/diff/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 53021: HIVE-14984: Hive-WebUI access results in Request is a replay (34) attack

2016-10-20 Thread Aihua Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53021/#review153388
---


Ship it!




Ship It!

- Aihua Xu


On Oct. 20, 2016, 9:47 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53021/
> ---
> 
> (Updated Oct. 20, 2016, 9:47 a.m.)
> 
> 
> Review request for hive, Aihua Xu and Mohit Sabharwal.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14984: Hive-WebUI access results in Request is a replay (34) attack
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hive/http/HttpServer.java 
> c4e2e33c6627be979daec5e7afa2ed82a039dde0 
>   service/src/resources/hive-webapps/hiveserver2/index.html 
> f18ba53e91518379b2f08a096fe08be899b293e3 
>   service/src/test/org/apache/hive/service/server/TestHS2HttpServer.java 
> c9e0ac3a751b8824224bda4c5a0487d286ab069a 
> 
> Diff: https://reviews.apache.org/r/53021/diff/
> 
> 
> Testing
> ---
> 
> Tested that in a secured cluster the hiveserver2.jsp is still only accessible 
> with correct credentials.
> Tested that the replay attack is not triggered when the context root is 
> called.
> Added unit test comparing the contents of the html response for a query 
> requesting the context root and the jsp.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>