Remote debugging of HIVE code from Intellij

[Ranjan Banerjee]

Hi,
   I am trying to remote debug HIVE code from Intellij. Here are my steps:
 1)In HIVE folder: mvn clean install -Dskiptests
  2)cd itests/qtest
  3)mvn clean install -Dskiptests
   4)mvn -Dmaven.surefire.debug -Dtest=TestPositiveCliDriver 
-Dqfile=acid_bucket_pruning.q
   I have setup the remote debugging config in Intellij and have put breakpoint 
in sessionstate.java
   However I see that the test completes successfully and does not listen at 
the 5005 port. Can someone give some pointers as to why the mvn test is not 
stopping to listen at this port?

Thanks
Ranjan

[jira] [Created] (HIVE-14718) spark download is not ignored to a sufficient degree

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-14718:
---

 Summary: spark download is not ignored to a sufficient degree
 Key: HIVE-14718
 URL: https://issues.apache.org/jira/browse/HIVE-14718
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51695: HIVE-5867: JDBC driver and beeline should support executing an initial SQL script

[cheng xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51695/#review148124
---




jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 128)


Extra Space



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 150)


Extra Space



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 160)


Extra Space



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 116)


2 space indent from L116 to L166



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 127)


Will this handle the following case?
{noformat}
show
 tables;
{noformat}



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 136)


Can we move it to the finally block?



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 147)


LOG.error



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 154)


Remove this line.



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 156)


Extra space



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 159)


Extra space



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 161)


Extra space



jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java (line 167)


No ident needed here.



jdbc/src/test/org/apache/hive/jdbc/TestHiveDriver.java (line 14)


Can this test pass without starting any server?



jdbc/src/test/org/apache/hive/jdbc/TestHiveDriver.java (line 19)


Extra space.



jdbc/src/test/resources/init.sql (line 3)


Remove this line since it's commented out.


- cheng xu


On Sept. 8, 2016, 8:52 a.m., Jianguo Tian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51695/
> ---
> 
> (Updated Sept. 8, 2016, 8:52 a.m.)
> 
> 
> Review request for hive and cheng xu.
> 
> 
> Bugs: HIVE-5867
> https://issues.apache.org/jira/browse/HIVE-5867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-5867: JDBC driver and beeline should support executing an initial SQL 
> script
> 
> 
> Diffs
> -
> 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
> ad96a6466dd1aadab71fc261f55be4639dcbe2bf 
>   jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 
> a349f8bdad09f526f66a71768ecd2ca019671167 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 
> 3161566994d6c6e01de9d88a6e87295684619ffa 
>   jdbc/src/test/org/apache/hive/jdbc/TestHiveDriver.java PRE-CREATION 
>   jdbc/src/test/resources/init.sql PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51695/diff/
> 
> 
> Testing
> ---
> 
> TestHiveDriver.java is JUnit test class which will print the rusult of 
> executing initial SQL(init.sql locate in src/test/resources).
> 
> 
> Thanks,
> 
> Jianguo Tian
> 
>

Re: Review Request 51695: HIVE-5867: JDBC driver and beeline should support executing an initial SQL script

[Jianguo Tian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51695/
---

(Updated Sept. 8, 2016, 12:52 a.m.)


Review request for hive and cheng xu.


Bugs: HIVE-5867
https://issues.apache.org/jira/browse/HIVE-5867


Repository: hive-git


Description
---

HIVE-5867: JDBC driver and beeline should support executing an initial SQL 
script


Diffs (updated)
-

  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
ad96a6466dd1aadab71fc261f55be4639dcbe2bf 
  jdbc/src/java/org/apache/hive/jdbc/HiveDriver.java 
a349f8bdad09f526f66a71768ecd2ca019671167 
  jdbc/src/java/org/apache/hive/jdbc/Utils.java 
3161566994d6c6e01de9d88a6e87295684619ffa 
  jdbc/src/test/org/apache/hive/jdbc/TestHiveDriver.java PRE-CREATION 
  jdbc/src/test/resources/init.sql PRE-CREATION 

Diff: https://reviews.apache.org/r/51695/diff/


Testing
---

TestHiveDriver.java is JUnit test class which will print the rusult of 
executing initial SQL(init.sql locate in src/test/resources).


Thanks,

Jianguo Tian

Re: Review Request 51695: HIVE-5867: JDBC driver and beeline should support executing an initial SQL script

[Jianguo Tian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51695/
---

(Updated Sept. 8, 2016, 12:51 a.m.)


Review request for hive and cheng xu.


Bugs: HIVE-5867
https://issues.apache.org/jira/browse/HIVE-5867


Repository: hive-git


Description
---

HIVE-5867: JDBC driver and beeline should support executing an initial SQL 
script


Diffs
-


Diff: https://reviews.apache.org/r/51695/diff/


Testing
---

TestHiveDriver.java is JUnit test class which will print the rusult of 
executing initial SQL(init.sql locate in src/test/resources).


File Attachments (updated)


HIVE-5867.1.patch
  
https://reviews.apache.org/media/uploaded/files/2016/09/08/72ebe7f1-4a30-489c-9b6c-8d5e79fdf500__HIVE-5867.1.patch


Thanks,

Jianguo Tian

Review Request 51695: HIVE-5867: JDBC driver and beeline should support executing an initial SQL script

[Jianguo Tian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51695/
---

Review request for hive and cheng xu.


Summary (updated)
-

HIVE-5867: JDBC driver and beeline should support executing an initial SQL 
script


Bugs: HIVE-5867
https://issues.apache.org/jira/browse/HIVE-5867


Repository: hive-git


Description
---

HIVE-5867: JDBC driver and beeline should support executing an initial SQL 
script


Diffs
-


Diff: https://reviews.apache.org/r/51695/diff/


Testing
---

TestHiveDriver.java is JUnit test class which will print the rusult of 
executing initial SQL(init.sql locate in src/test/resources).


Thanks,

Jianguo Tian

Re: Review Request 51468: HIVE-14532 - enable qtests from ide - eclipse

[Zoltan Haindrich]

> On Aug. 27, 2016, 12:25 a.m., Gabor Szadovszky wrote:
> > pom.xml, line 195
> > 
> >
> > Wouldn't it infect the other modules (production) as well?
> 
> Zoltan Haindrich wrote:
> I don't think so...this property will affect everything where its 
> used...when the ide profile is inactive it will work as before.
> 
> Gabor Szadovszky wrote:
> Then, the comment "this will enable by default the shade plugin" in the 
> root pom is a bit misleading to me.
> 
> Zoltan Haindrich wrote:
> disabling the shade plugin was not easy...contolling its execution phase 
> seemed the most straight forward way to control its skip...
> it's not good to have a comment which is misleading...
> 
> * do you have any suggestion?
> * sould i remove the comment? I think the property name is self 
> explaining about what it does - and that way you may look up its uses and 
> figure out its function from the actual configuration..
> 
> Gabor Szadovszky wrote:
> For me the comment means that the shade plugin will run for all the 
> submodules which is not true. I would say something like "Used for 
> enabling/disabling the shade plugin where it is relevant. Enabled by 
> default." What do you think?

i've changed the note...makes more sense ;)


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51468/#review147046
---


On Sept. 7, 2016, 8:39 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51468/
> ---
> 
> (Updated Sept. 7, 2016, 8:39 p.m.)
> 
> 
> Review request for hive, Balint Molnar, Lefty Leverenz, and Prasanth_J.
> 
> 
> Bugs: HIVE-14532
> https://issues.apache.org/jira/browse/HIVE-14532
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> how to execute qtests for ide wikipage draft:
> v1: http://hastebin.com/paxicutive.vhdl
> 
> the patch itself contains:
> 
> * some automatic property settings to configure qtest related things to be 
> able to execute
> * maven profile to avoid shading plugin invocation during IDE project 
> generation - to void classpath/compile errors i've encountered
> * some test.src.tables related changes - defaulting is now done at a 
> different place
> * eclipse is ok for me
> 
> patch#2 here is patch#3 in the jira
> 
> 
> Diffs
> -
> 
>   itests/hive-jmh/pom.xml f1417fd7e8b70a7a3dd255c25d3909013f02df67 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java
>  efbd4657f22e856b9c9ba5f74472ad5fd9f9a5b5 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 69c4974105c6b47cc8e8051cd980274c9c381775 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 4d4a929c159c61f9f4af3238d4b7baff146d346e 
>   jdbc/pom.xml b29739b3f8577c6e363b5c8ee39b63e53a17c907 
>   pom.xml 4c41200ffc8e2c9a1d207f8676f252b94e80e4fd 
>   ql/pom.xml 02ddb805a228ed23694c8a81953dd2400d7308c6 
> 
> Diff: https://reviews.apache.org/r/51468/diff/
> 
> 
> Testing
> ---
> 
> I've tested the draft using eclipse:
> 3.0 -> 3.1 -> TestCliDriver(combine2,alter_concatenate_indexed_table)
> 3.0 -> 3.2 -> TestCliDriver(combine2)
> 
> idea:
> althrough I was able to create a few working setups...I think it will need 
> more work
> 
> 
> Thanks,
> 
> Zoltan Haindrich
> 
>

Re: Review Request 51468: HIVE-14532 - enable qtests from ide - eclipse

[Zoltan Haindrich]

> On Aug. 28, 2016, 3:07 p.m., Peter Vary wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java,
> >  lines 80-84
> > 
> >
> > It is mostly just a question, which appeared to me in the morning, 
> > waking up:
> > - There are several other places which are using QTestUtil's query 
> > running framework - do any of them uses these properties? If so, then it 
> > might be good idea to put these defaults to the QTestUtil constructor, so 
> > these initialized there as well, and could be run from browser too.
> > 
> > I could see several valid answers why you did not do this, like:
> > - These are only used by qtest queries
> > - We concentrate on the qtests now, and do not now enough of the other 
> > test to do this change
> > 
> > This morning ideas are sometimes very good, sometimes not so much, just 
> > "stored" here to check when realy aweaken :)
> > 
> > Another one of these ideas:
> > The next develeper, who adds a new variable, he might forget to add it 
> > here, so the test will only work with maven, and not with eclipse, or idea. 
> > I think we should add a comment to the pom.xml, to remaind him to add a 
> > default value here as well. For example after this comment:
> > 
> > 
> > 
> > ${basedir}/${hive.path.to.root}/data/files
> > 
> > What do you think?

i think most of these extra properties should be removed later and/or use some 
convention instead of setting them(warehousedir/tmpdir)

test.data.files - is a property which has no real alternative value...that 
directory contains so many specific files that you can't really change it...


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51468/#review147100
---


On Sept. 7, 2016, 8:39 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51468/
> ---
> 
> (Updated Sept. 7, 2016, 8:39 p.m.)
> 
> 
> Review request for hive, Balint Molnar, Lefty Leverenz, and Prasanth_J.
> 
> 
> Bugs: HIVE-14532
> https://issues.apache.org/jira/browse/HIVE-14532
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> how to execute qtests for ide wikipage draft:
> v1: http://hastebin.com/paxicutive.vhdl
> 
> the patch itself contains:
> 
> * some automatic property settings to configure qtest related things to be 
> able to execute
> * maven profile to avoid shading plugin invocation during IDE project 
> generation - to void classpath/compile errors i've encountered
> * some test.src.tables related changes - defaulting is now done at a 
> different place
> * eclipse is ok for me
> 
> patch#2 here is patch#3 in the jira
> 
> 
> Diffs
> -
> 
>   itests/hive-jmh/pom.xml f1417fd7e8b70a7a3dd255c25d3909013f02df67 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java
>  efbd4657f22e856b9c9ba5f74472ad5fd9f9a5b5 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 69c4974105c6b47cc8e8051cd980274c9c381775 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 4d4a929c159c61f9f4af3238d4b7baff146d346e 
>   jdbc/pom.xml b29739b3f8577c6e363b5c8ee39b63e53a17c907 
>   pom.xml 4c41200ffc8e2c9a1d207f8676f252b94e80e4fd 
>   ql/pom.xml 02ddb805a228ed23694c8a81953dd2400d7308c6 
> 
> Diff: https://reviews.apache.org/r/51468/diff/
> 
> 
> Testing
> ---
> 
> I've tested the draft using eclipse:
> 3.0 -> 3.1 -> TestCliDriver(combine2,alter_concatenate_indexed_table)
> 3.0 -> 3.2 -> TestCliDriver(combine2)
> 
> idea:
> althrough I was able to create a few working setups...I think it will need 
> more work
> 
> 
> Thanks,
> 
> Zoltan Haindrich
> 
>

Re: Review Request 51468: HIVE-14532 - enable qtests from ide - eclipse

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51468/
---

(Updated Sept. 7, 2016, 8:39 p.m.)


Review request for hive, Balint Molnar, Lefty Leverenz, and Prasanth_J.


Changes
---

removed change in enforcereadonlytables; separated eclipse build dir


Summary (updated)
-

HIVE-14532 - enable qtests from ide - eclipse


Bugs: HIVE-14532
https://issues.apache.org/jira/browse/HIVE-14532


Repository: hive-git


Description (updated)
---

how to execute qtests for ide wikipage draft:
v1: http://hastebin.com/paxicutive.vhdl

the patch itself contains:

* some automatic property settings to configure qtest related things to be able 
to execute
* maven profile to avoid shading plugin invocation during IDE project 
generation - to void classpath/compile errors i've encountered
* some test.src.tables related changes - defaulting is now done at a different 
place
* eclipse is ok for me

patch#2 here is patch#3 in the jira


Diffs (updated)
-

  itests/hive-jmh/pom.xml f1417fd7e8b70a7a3dd255c25d3909013f02df67 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java
 efbd4657f22e856b9c9ba5f74472ad5fd9f9a5b5 
  itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
69c4974105c6b47cc8e8051cd980274c9c381775 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
4d4a929c159c61f9f4af3238d4b7baff146d346e 
  jdbc/pom.xml b29739b3f8577c6e363b5c8ee39b63e53a17c907 
  pom.xml 4c41200ffc8e2c9a1d207f8676f252b94e80e4fd 
  ql/pom.xml 02ddb805a228ed23694c8a81953dd2400d7308c6 

Diff: https://reviews.apache.org/r/51468/diff/


Testing (updated)
---

I've tested the draft using eclipse:
3.0 -> 3.1 -> TestCliDriver(combine2,alter_concatenate_indexed_table)
3.0 -> 3.2 -> TestCliDriver(combine2)

idea:
althrough I was able to create a few working setups...I think it will need more 
work


Thanks,

Zoltan Haindrich

Review Request 51699: HIVE-12812: Enable mapred.input.dir.recursive by default to support union with aggregate function

[Chaoyu Tang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51699/
---

Review request for hive.


Bugs: HIVE-12812
https://issues.apache.org/jira/browse/HIVE-12812


Repository: hive-git


Description
---

mapred.input.dir.recursive itself is not Hive property, and it is often ignored 
when user runs some queries (e.g. union) which needs it to be enabled to 
process some intermediate data files. This patch is to enable this property by 
default. Change only happens for MR since Tez has done similar in its 
TezComplier and Spark does not need it.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 5b08ed2 
  ql/src/test/queries/clientnegative/truncate_column_list_bucketing.q 90f6cf7 
  ql/src/test/queries/clientpositive/infer_bucket_sort_list_bucket.q 64dd05d 
  ql/src/test/queries/clientpositive/lb_fs_stats.q 5ad3ff3 
  ql/src/test/queries/clientpositive/list_bucket_dml_1.q e38930f 
  ql/src/test/queries/clientpositive/list_bucket_dml_10.q f25c174 
  ql/src/test/queries/clientpositive/list_bucket_dml_11.q 8ac1627 
  ql/src/test/queries/clientpositive/list_bucket_dml_12.q 9facfa5 
  ql/src/test/queries/clientpositive/list_bucket_dml_13.q 0fe7f61 
  ql/src/test/queries/clientpositive/list_bucket_dml_14.q abe5ea7 
  ql/src/test/queries/clientpositive/list_bucket_dml_2.q c6dceab 
  ql/src/test/queries/clientpositive/list_bucket_dml_3.q ef01529 
  ql/src/test/queries/clientpositive/list_bucket_dml_4.q 950409d 
  ql/src/test/queries/clientpositive/list_bucket_dml_5.q fce8e2e 
  ql/src/test/queries/clientpositive/list_bucket_dml_6.q 631c938 
  ql/src/test/queries/clientpositive/list_bucket_dml_7.q e2af1b3 
  ql/src/test/queries/clientpositive/list_bucket_dml_8.q 6d73896 
  ql/src/test/queries/clientpositive/list_bucket_dml_9.q d2e24af 
  ql/src/test/queries/clientpositive/list_bucket_query_multiskew_1.q 1a4fbc2 
  ql/src/test/queries/clientpositive/list_bucket_query_multiskew_2.q 996dce6 
  ql/src/test/queries/clientpositive/list_bucket_query_multiskew_3.q f09026f 
  ql/src/test/queries/clientpositive/list_bucket_query_oneskew_1.q cc597dc 
  ql/src/test/queries/clientpositive/list_bucket_query_oneskew_2.q fb4cf6d 
  ql/src/test/queries/clientpositive/list_bucket_query_oneskew_3.q 702de64 
  ql/src/test/queries/clientpositive/recursive_dir.q d703b8e 
  ql/src/test/queries/clientpositive/skewjoin_union_remove_1.q 5e688d7 
  ql/src/test/queries/clientpositive/skewjoin_union_remove_2.q 3e2610f 
  ql/src/test/queries/clientpositive/truncate_column_list_bucket.q 476120a 
  ql/src/test/queries/clientpositive/union_remove_1.q 0a69068 
  ql/src/test/queries/clientpositive/union_remove_10.q 7892e15 
  ql/src/test/queries/clientpositive/union_remove_11.q fd41648 
  ql/src/test/queries/clientpositive/union_remove_12.q b665666 
  ql/src/test/queries/clientpositive/union_remove_13.q 11077fd 
  ql/src/test/queries/clientpositive/union_remove_14.q b559b35 
  ql/src/test/queries/clientpositive/union_remove_15.q 43c7834 
  ql/src/test/queries/clientpositive/union_remove_16.q eca8aeb 
  ql/src/test/queries/clientpositive/union_remove_17.q 59a3a9c 
  ql/src/test/queries/clientpositive/union_remove_18.q 98ee7d0 
  ql/src/test/queries/clientpositive/union_remove_19.q 9e47254 
  ql/src/test/queries/clientpositive/union_remove_2.q 2ebc1ad 
  ql/src/test/queries/clientpositive/union_remove_20.q 700ee4d 
  ql/src/test/queries/clientpositive/union_remove_21.q 1b1472e 
  ql/src/test/queries/clientpositive/union_remove_22.q d4d3cbc 
  ql/src/test/queries/clientpositive/union_remove_23.q 98ae503 
  ql/src/test/queries/clientpositive/union_remove_24.q 7ed80d1 
  ql/src/test/queries/clientpositive/union_remove_25.q 78fa9aa 
  ql/src/test/queries/clientpositive/union_remove_26.q 4e77d73 
  ql/src/test/queries/clientpositive/union_remove_3.q b0f63fc 
  ql/src/test/queries/clientpositive/union_remove_4.q 9ee6282 
  ql/src/test/queries/clientpositive/union_remove_5.q 5d3c482 
  ql/src/test/queries/clientpositive/union_remove_6.q 23eb760 
  ql/src/test/queries/clientpositive/union_remove_6_subq.q 3607a52 
  ql/src/test/queries/clientpositive/union_remove_7.q 5e0d14b 
  ql/src/test/queries/clientpositive/union_remove_8.q 1e049ba 
  ql/src/test/queries/clientpositive/union_remove_9.q b819560 

Diff: https://reviews.apache.org/r/51699/diff/


Testing
---

Manual tests
Precommit tests


Thanks,

Chaoyu Tang

Re: Review Request 51593: HIVE-14063 : Add beeline connection configuration file to automatically connect to HS2 using beeline

[Vihang Karajgaonkar]

> On Sept. 7, 2016, 2:38 p.m., Peter Vary wrote:
> > beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java,
> >  line 88
> > 
> >
> > I might prefer not to expose the file location on the interface, if it 
> > is not absolutely required.
> > 
> > What do you think of using instead of this a new public method:
> > 
> > public boolean isConfigExists()?

Good idea. Added a new interface method configExists() instead of 
getFileLocation.


> On Sept. 7, 2016, 2:38 p.m., Peter Vary wrote:
> > beeline/src/java/org/apache/hive/beeline/BeeLine.java, lines 981-982
> > 
> >
> > Public + VisibleForTesting

This is a marker annotation for documetation purposes only. The method needs to 
be more visible than required only for test coverage. I have added a comment 
saying the same


> On Sept. 7, 2016, 2:38 p.m., Peter Vary wrote:
> > beeline/src/java/org/apache/hive/beeline/BeeLine.java, lines 976-977
> > 
> >
> > Public + VisibleForTesting

This is a marker annotation for documetation purposes only. The method needs to 
be more visible than required only for test coverage. I have added a comment 
saying the same


> On Sept. 7, 2016, 2:38 p.m., Peter Vary wrote:
> > beeline/src/java/org/apache/hive/beeline/BeeLine.java, lines 955-957
> > 
> >
> > I am not sure about this, and this is more like a requirement question, 
> > than a coding question:
> > - We might like to use the hive-site.xml based connection data, even 
> > there is no user specific connection data.
> > 
> > For example:
> > - We have an unsecured cluster, so we do not mind, if everyone uses the 
> > same username, password using hive.
> > 
> > This might be a valid usecase. We still can chose not to support this, 
> > since this is a bad practice :)
> > 
> > So this is just a question, nothing more!

I had thought about this particular case. The issue is this change needs to be 
backwards compatible which means that previous methods of connecting using url 
should work. That means Beeline should know using some flag to use 
hive-site.xml or not. The presence of this file indicates that hive-site.xml is 
used or not. I had initially added an explicit flag for the same in user conf 
file, but seemed redundant to me. So if user would like to use only 
hive-site.xml for connection info, the can provide an empty file and it will 
serve as a flag to look for hive-site.xml. This may be true in a secure, 
kerberos enabled cluster where hive-site.xml has all the information that we 
need to connect. But there is a bug where beeline still asks for username, 
password even in kerberos enabled cluster, so currently, even in that case 
there needs to be some user-specific information which needs to be provided. I 
am planning to fix that case in a separate change. For all the other changes, 
there needs t
 o be a non-empty user-specific file which atleast provides user/password since 
hive-site.xml doesn't have that information.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51593/#review148007
---


On Sept. 7, 2016, 6:22 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51593/
> ---
> 
> (Updated Sept. 7, 2016, 6:22 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal, Sergio Pena, and Szehon Ho.
> 
> 
> Bugs: HIVE-14063
> https://issues.apache.org/jira/browse/HIVE-14063
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This change adds a new optional configuration file for Beeline. If this file 
> is present at predefined locations, Beeline will attempt to create the 
> connection url using the hive-site.xml found in classpath and another 
> user-specific configuration file. Beeline then connects automatically using 
> the url generated based on these configuration files. The main objective of 
> the change is to provide user another way to connect to the HiveServer2 
> without providing the connection url everytime. The configuration file uses 
> hadoop xml format so that we can support encryption/obfuscation using hadoop 
> credential manager API in the future.
> 
> Properties in the user-specific configuration file override the properties 
> derived from hive-site.xml.
> 
> Tested using newly added unit tests and itests in various Hiveserver2 
> configurations.
> 
> 
> Diffs
> -
> 

Re: Review Request 51593: HIVE-14063 : Add beeline connection configuration file to automatically connect to HS2 using beeline

[Vihang Karajgaonkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51593/
---

(Updated Sept. 7, 2016, 6:22 p.m.)


Review request for hive, Mohit Sabharwal, Sergio Pena, and Szehon Ho.


Changes
---

Added changes suggested by Peter.


Bugs: HIVE-14063
https://issues.apache.org/jira/browse/HIVE-14063


Repository: hive-git


Description
---

This change adds a new optional configuration file for Beeline. If this file is 
present at predefined locations, Beeline will attempt to create the connection 
url using the hive-site.xml found in classpath and another user-specific 
configuration file. Beeline then connects automatically using the url generated 
based on these configuration files. The main objective of the change is to 
provide user another way to connect to the HiveServer2 without providing the 
connection url everytime. The configuration file uses hadoop xml format so that 
we can support encryption/obfuscation using hadoop credential manager API in 
the future.

Properties in the user-specific configuration file override the properties 
derived from hive-site.xml.

Tested using newly added unit tests and itests in various Hiveserver2 
configurations.


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 
8e65e3987398531cce5c65c383762cf49a52c578 
  beeline/src/java/org/apache/hive/beeline/Commands.java 
2f3ec134098dfa3767bab9545438d1f38f11697c 
  
beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineHS2ConnectionFileParseException.java
 PRE-CREATION 
  
beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
 PRE-CREATION 
  
beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileUtils.java
 PRE-CREATION 
  
beeline/src/java/org/apache/hive/beeline/hs2connection/HiveSiteHS2ConnectionFileParser.java
 PRE-CREATION 
  
beeline/src/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
 PRE-CREATION 
  
beeline/src/test/org/apache/hive/beeline/hs2connection/TestUserHS2ConnectionFileParser.java
 PRE-CREATION 
  beeline/src/test/resources/hive-site.xml 
5f310d68245275ac9dc24df45579784019eea332 
  beeline/src/test/resources/test-hs2-conn-conf-kerberos-http.xml PRE-CREATION 
  beeline/src/test/resources/test-hs2-conn-conf-kerberos-nossl.xml PRE-CREATION 
  beeline/src/test/resources/test-hs2-conn-conf-kerberos-ssl.xml PRE-CREATION 
  beeline/src/test/resources/test-hs2-connection-conf-list.xml PRE-CREATION 
  beeline/src/test/resources/test-hs2-connection-config-noauth.xml PRE-CREATION 
  beeline/src/test/resources/test-hs2-connection-multi-conf-list.xml 
PRE-CREATION 
  beeline/src/test/resources/test-hs2-connection-zookeeper-config.xml 
PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineConnectionUsingHiveSite.java
 PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineWithHS2ConnectionFile.java
 PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineWithUserHs2ConnectionFile.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/51593/diff/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Error when running the test TestBeeLineWithArgs

[Vihang Karajgaonkar]

Yes, you are right. I have created a JIRA for the same.

https://issues.apache.org/jira/browse/HIVE-14717 

> On Sep 6, 2016, at 7:02 AM, Marta Kuczora  wrote:
> 
> Hi,
> 
> I have noticed recently that I cannot run the TestBeeLineWithArgs test on
> my local machine.
> I wasn't sure if something went wrong on my local machine or this test was
> broken, so I checked the result of the last few PreCommit-HIVE-MASTER-Build
> and it seems to me that this test failed on those builds as well.
> For example: I checked the logs of the build # and I found the error
> written below [1] in the maven-test.txt of the TestBeeLineWithArgs test.
> 
> Has anybody run into the same problem?
> Is this a known issue? If not, I will create a Jira for it.
> 
> 
> Thanks and regards,
> Marta
> 
> 
> [1]
> ---
> T E S T S
> ---
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=512m; support was removed in 8.0
> Running org.apache.hive.beeline.TestBeeLineWithArgs
> 
> Results :
> 
> Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> 
> [INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Integration - Parent .. SUCCESS [
> 1.279 s]
> [INFO] Hive Integration - Custom Serde  SUCCESS [
> 4.705 s]
> [INFO] Hive Integration - Custom udfs . SUCCESS [
> 1.447 s]
> [INFO] Hive Integration - Custom UDFs - udf-classloader-util SUCCESS [
> 1.342 s]
> [INFO] Hive Integration - Custom UDFs - udf-classloader-udf1 SUCCESS [
> 1.299 s]
> [INFO] Hive Integration - Custom UDFs - udf-classloader-udf2 SUCCESS [
> 1.131 s]
> [INFO] Hive Integration - Custom UDFs - udf-vectorized-badexample SUCCESS [
> 0.923 s]
> [INFO] Hive Integration - HCatalog Unit Tests . SUCCESS [
> 5.420 s]
> [INFO] Hive Integration - Testing Utilities ... SUCCESS [
> 5.626 s]
> [INFO] Hive Integration - Unit Tests .. FAILURE [
> 48.933 s]
> [INFO] Hive Integration - Test Serde .. SKIPPED
> [INFO] Hive Integration - QFile Tests . SKIPPED
> [INFO] Hive Integration - QFile Accumulo Tests  SKIPPED
> [INFO] JMH benchmark: Hive  SKIPPED
> [INFO] Hive Integration - Unit Tests - Hadoop 2 ... SKIPPED
> [INFO] Hive Integration - Unit Tests with miniKdc . SKIPPED
> [INFO] Hive Integration - QFile Spark Tests ... SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 01:13 min
> [INFO] Finished at: 2016-09-05T20:57:36-04:00
> [INFO] Final Memory: 83M/877M
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test)
> on project hive-it-unit: ExecutionException The forked VM terminated
> without properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd
> /home/hiveptest/50.18.132.165-hiveptest-0/apache-github-source-source/itests/hive-unit
> && /usr/java/jdk1.8.0_25/jre/bin/java -Xmx2048m -XX:MaxPermSize=512m -jar
> /home/hiveptest/50.18.132.165-hiveptest-0/apache-github-source-source/itests/hive-unit/target/surefire/surefirebooter2766135795402840392.jar
> /home/hiveptest/50.18.132.165-hiveptest-0/apache-github-source-source/itests/hive-unit/target/surefire/surefire5846457031220411379tmp
> /home/hiveptest/50.18.132.165-hiveptest-0/apache-github-source-source/itests/hive-unit/target/surefire/surefire_0622126075359686603tmp
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn  -rf :hive-it-unit

[jira] [Created] (HIVE-14717) Beeline tests failing

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-14717:
--

 Summary: Beeline tests failing
 Key: HIVE-14717
 URL: https://issues.apache.org/jira/browse/HIVE-14717
 Project: Hive
  Issue Type: Bug
Reporter: Vihang Karajgaonkar
Priority: Critical


If you run mvn clean test in beeline you see the following errors:

{noformat}
---
 T E S T S
---
Running org.apache.hive.beeline.cli.TestHiveCli
Running org.apache.hive.beeline.TestBeelineArgParsing
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.584 sec - in 
org.apache.hive.beeline.TestBeelineArgParsing
Running org.apache.hive.beeline.TestBeeLineHistory
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.198 sec - in 
org.apache.hive.beeline.TestBeeLineHistory
Running org.apache.hive.beeline.TestBeeLineOpts
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.338 sec - in 
org.apache.hive.beeline.TestBeeLineOpts
Running org.apache.hive.beeline.TestBufferedRows
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.382 sec - in 
org.apache.hive.beeline.TestBufferedRows
Running org.apache.hive.beeline.TestClientCommandHookFactory
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.546 sec - in 
org.apache.hive.beeline.TestClientCommandHookFactory
Running org.apache.hive.beeline.TestIncrementalRowsWithNormalization
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.379 sec - in 
org.apache.hive.beeline.TestIncrementalRowsWithNormalization
Running org.apache.hive.beeline.TestTableOutputFormat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.402 sec - in 
org.apache.hive.beeline.TestTableOutputFormat

Results :

Tests run: 44, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:03 min
[INFO] Finished at: 2016-09-07T10:57:28-07:00
[INFO] Final Memory: 65M/949M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on 
project hive-beeline: ExecutionException The forked VM terminated without 
properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/vihang/work/src/upstream/hive/beeline 
&& /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre/bin/java 
-Xmx2048m -XX:MaxPermSize=512m -jar 
/Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefirebooter8133887423099901737.jar
 
/Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefire6528142441948588259tmp
 
/Users/vihang/work/src/upstream/hive/beeline/target/surefire/surefire_05570572112194455658tmp
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
{noformat}

My guess is this is related to recent fix for HIVE-13589 but needs more 
investigation.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14716) Duplicate pom.xml entries for mockito

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-14716:
--

 Summary: Duplicate pom.xml entries for mockito
 Key: HIVE-14716
 URL: https://issues.apache.org/jira/browse/HIVE-14716
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar
Priority: Trivial


When you build beeline there is a warning which says there are duplicate 
pom.xml entries for mockito dependency.

[WARNING]
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hive:hive-beeline:jar:2.2.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must 
be unique: org.mockito:mockito-all:jar -> duplicate declaration of version (?) 
@ org.apache.hive:hive-beeline:[unknown-version], 
/Users/vihang/work/src/upstream/hive/beeline/pom.xml, line 126, column 17
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must 
be unique: org.mockito:mockito-all:jar -> duplicate declaration of version (?) 
@ org.apache.hive:hive-beeline:[unknown-version], 
/Users/vihang/work/src/upstream/hive/beeline/pom.xml, line 137, column 17



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14715) Hive throws NumberFormatException with query with Null value

[Yongzhi Chen (JIRA)]

Yongzhi Chen created HIVE-14715:
---

 Summary: Hive throws NumberFormatException with query with Null 
value
 Key: HIVE-14715
 URL: https://issues.apache.org/jira/browse/HIVE-14715
 Project: Hive
  Issue Type: Bug
Reporter: Yongzhi Chen


The java.lang.NumberFormatException will throw with following reproduce:
set hive.cbo.enable=false;
CREATE TABLE `paqtest`(
`c1` int,
`s1` string,
`s2` string,
`bn1` bigint)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat';

insert into paqtest values (58, '', 'ABC', 0);

SELECT
'Pricing mismatch' AS category,
c1,
NULL AS itemtype_used,
NULL AS acq_itemtype,
s2,
NULL AS currency_used_avg,
NULL AS acq_items_avg,
sum(bn1) AS cca
FROM paqtest
WHERE (s1 IS NULL OR length(s1) = 0)
GROUP BY 'Pricing mismatch', c1, NULL, NULL, s2, NULL, NULL;

The stack like following:
java.lang.NumberFormatException: ABC
GroupByOperator.process(Object, int) line: 773  
ExecReducer.reduce(Object, Iterator, OutputCollector, Reporter) line: 236   
ReduceTask.runOldReducer(JobConf, TaskUmbilicalProtocol, TaskReporter, 
RawKeyValueIterator, RawComparator, Class, Class) line: 
444   
ReduceTask.run(JobConf, TaskUmbilicalProtocol) line: 392
LocalJobRunner$Job$ReduceTaskRunnable.run() line: 319   
Executors$RunnableAdapter.call() line: 471   

It works fine when hive.cbo.enable = true




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14714) Finishing Hive on Spark causes "java.io.IOException: Stream closed"

[Gabor Szadovszky (JIRA)]

Gabor Szadovszky created HIVE-14714:
---

 Summary: Finishing Hive on Spark causes "java.io.IOException: 
Stream closed"
 Key: HIVE-14714
 URL: https://issues.apache.org/jira/browse/HIVE-14714
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 1.1.0
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky


After execute hive command with Spark, finishing the beeline session or
even switch the engine causes IOException. The following executed Ctrl-D to
finish the session but "!quit" or even "set hive.execution.engine=mr;" causes
the issue.

>From HS2 log:
{code}
2016-09-06 16:15:12,291 WARN  org.apache.hive.spark.client.SparkClientImpl: 
[HiveServer2-Handler-Pool: Thread-106]: Timed out shutting down remote driver, 
interrupting...
2016-09-06 16:15:12,291 WARN  org.apache.hive.spark.client.SparkClientImpl: 
[Driver]: Waiting thread interrupted, killing child process.
2016-09-06 16:15:12,296 WARN  org.apache.hive.spark.client.SparkClientImpl: 
[stderr-redir-1]: Error in redirector thread.
java.io.IOException: Stream closed
at 
java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at 
org.apache.hive.spark.client.SparkClientImpl$Redirector.run(SparkClientImpl.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51593: HIVE-14063 : Add beeline connection configuration file to automatically connect to HS2 using beeline

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51593/#review148007
---



Hi Vihang,

Finally, I had time to read through the code.
One question about the requirements, and few smaller ones.

Otherwise LGTM.

Thanks,
Peter


beeline/src/java/org/apache/hive/beeline/BeeLine.java (lines 936 - 938)


I am not sure about this, and this is more like a requirement question, 
than a coding question:
- We might like to use the hive-site.xml based connection data, even there 
is no user specific connection data.

For example:
- We have an unsecured cluster, so we do not mind, if everyone uses the 
same username, password using hive.

This might be a valid usecase. We still can chose not to support this, 
since this is a bad practice :)

So this is just a question, nothing more!



beeline/src/java/org/apache/hive/beeline/BeeLine.java (lines 957 - 958)


Public + VisibleForTesting



beeline/src/java/org/apache/hive/beeline/BeeLine.java (lines 962 - 963)


Public + VisibleForTesting



beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
 (line 88)


I might prefer not to expose the file location on the interface, if it is 
not absolutely required.

What do you think of using instead of this a new public method:

public boolean isConfigExists()?


- Peter Vary


On Sept. 6, 2016, 11:53 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51593/
> ---
> 
> (Updated Sept. 6, 2016, 11:53 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal, Sergio Pena, and Szehon Ho.
> 
> 
> Bugs: HIVE-14063
> https://issues.apache.org/jira/browse/HIVE-14063
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This change adds a new optional configuration file for Beeline. If this file 
> is present at predefined locations, Beeline will attempt to create the 
> connection url using the hive-site.xml found in classpath and another 
> user-specific configuration file. Beeline then connects automatically using 
> the url generated based on these configuration files. The main objective of 
> the change is to provide user another way to connect to the HiveServer2 
> without providing the connection url everytime. The configuration file uses 
> hadoop xml format so that we can support encryption/obfuscation using hadoop 
> credential manager API in the future.
> 
> Properties in the user-specific configuration file override the properties 
> derived from hive-site.xml.
> 
> Tested using newly added unit tests and itests in various Hiveserver2 
> configurations.
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 
> 8e65e3987398531cce5c65c383762cf49a52c578 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 
> 2f3ec134098dfa3767bab9545438d1f38f11697c 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineHS2ConnectionFileParseException.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileUtils.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HiveSiteHS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/test/org/apache/hive/beeline/hs2connection/TestUserHS2ConnectionFileParser.java
>  PRE-CREATION 
>   beeline/src/test/resources/hive-site.xml 
> 5f310d68245275ac9dc24df45579784019eea332 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-http.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-nossl.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-ssl.xml PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-conf-list.xml PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-config-noauth.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-multi-conf-list.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-zookeeper-config.xml 
> PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineConnectionUsingHiveSite.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineWithHS2ConnectionFile.java
>  

Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests

[Illya Yalovyy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51694/
---

Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and 
Szehon Ho.


Repository: hive-git


Description
---

Currently LdapAuthenticationProviderImpl class is not covered with unit tests. 
To make this class testable some minor refactoring will be required.


Diffs
-

  service/pom.xml ecea719 
  
service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java
 efd5393 
  service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java 
PRE-CREATION 
  
service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java 
PRE-CREATION 
  
service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java
 089a059 
  
service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java
 f276906 
  service/src/test/org/apache/hive/service/auth/ldap/Credentials.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/LdapTestUtils.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestChainFilter.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestCustomQueryFilter.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestLdapUtils.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestQuery.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 
PRE-CREATION 
  
service/src/test/org/apache/hive/service/auth/ldap/TestSearchResultHandler.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestUserFilter.java 
PRE-CREATION 
  service/src/test/org/apache/hive/service/auth/ldap/TestUserSearchFilter.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/51694/diff/


Testing
---

...hive/service> mvn clean test

...

Results :

Tests run: 123, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 04:18 min
[INFO] Finished at: 2016-09-06T08:46:04-07:00
[INFO] Final Memory: 66M/984M
[INFO] 


Thanks,

Illya Yalovyy

Re: Review Request 51593: HIVE-14063 : Add beeline connection configuration file to automatically connect to HS2 using beeline

[Gabor Szadovszky]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51593/#review148013
---


Ship it!




LGTM

- Gabor Szadovszky


On Sept. 6, 2016, 11:53 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51593/
> ---
> 
> (Updated Sept. 6, 2016, 11:53 p.m.)
> 
> 
> Review request for hive, Mohit Sabharwal, Sergio Pena, and Szehon Ho.
> 
> 
> Bugs: HIVE-14063
> https://issues.apache.org/jira/browse/HIVE-14063
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This change adds a new optional configuration file for Beeline. If this file 
> is present at predefined locations, Beeline will attempt to create the 
> connection url using the hive-site.xml found in classpath and another 
> user-specific configuration file. Beeline then connects automatically using 
> the url generated based on these configuration files. The main objective of 
> the change is to provide user another way to connect to the HiveServer2 
> without providing the connection url everytime. The configuration file uses 
> hadoop xml format so that we can support encryption/obfuscation using hadoop 
> credential manager API in the future.
> 
> Properties in the user-specific configuration file override the properties 
> derived from hive-site.xml.
> 
> Tested using newly added unit tests and itests in various Hiveserver2 
> configurations.
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 
> 8e65e3987398531cce5c65c383762cf49a52c578 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 
> 2f3ec134098dfa3767bab9545438d1f38f11697c 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineHS2ConnectionFileParseException.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileUtils.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HiveSiteHS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
>  PRE-CREATION 
>   
> beeline/src/test/org/apache/hive/beeline/hs2connection/TestUserHS2ConnectionFileParser.java
>  PRE-CREATION 
>   beeline/src/test/resources/hive-site.xml 
> 5f310d68245275ac9dc24df45579784019eea332 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-http.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-nossl.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-conn-conf-kerberos-ssl.xml PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-conf-list.xml PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-config-noauth.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-multi-conf-list.xml 
> PRE-CREATION 
>   beeline/src/test/resources/test-hs2-connection-zookeeper-config.xml 
> PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineConnectionUsingHiveSite.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineWithHS2ConnectionFile.java
>  PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/TestBeelineWithUserHs2ConnectionFile.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51593/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

[jira] [Created] (HIVE-14713) LDAP Authentication Provider should be covered with unit tests

[Illya Yalovyy (JIRA)]

Illya Yalovyy created HIVE-14713:


 Summary: LDAP Authentication Provider should be covered with unit 
tests
 Key: HIVE-14713
 URL: https://issues.apache.org/jira/browse/HIVE-14713
 Project: Hive
  Issue Type: Test
  Components: Authentication, Tests
Affects Versions: 2.1.0
Reporter: Illya Yalovyy
Assignee: Illya Yalovyy


Currently LdapAuthenticationProviderImpl class is not covered with unit tests. 
To make this class testable some minor refactoring will be required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51395: HIVE-14626: Support Trash in Truncate Table

[Chaoyu Tang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51395/
---

(Updated Sept. 7, 2016, 12:49 p.m.)


Review request for hive.


Changes
---

Revised the patch based on Sergio's review - only to clean the directory 
contents when it is protected by encryption zone, for other cases, we still do 
trash/delete the directory followed by recreation which might have a better 
performance especially for S3 cases.


Bugs: HIVE-14626
https://issues.apache.org/jira/browse/HIVE-14626


Repository: hive-git


Description
---

Truncate table/partition is implemented to clean the contents of table/partiton 
directory using multi-threaded trashFiles instead of deleting/recreating the 
directory, so the encryption level is preserved if any. In addition, it 
supports the HDFS Trash.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a59b781 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 5f53aef 
  ql/src/test/queries/clientpositive/encryption_drop_partition.q 57dfabd 
  ql/src/test/queries/clientpositive/encryption_drop_table.q 2ae3c69 
  ql/src/test/results/clientpositive/encrypted/encryption_drop_partition.q.out 
2643006 
  ql/src/test/results/clientpositive/encrypted/encryption_drop_table.q.out 
c5007ee 

Diff: https://reviews.apache.org/r/51395/diff/


Testing
---

Manual tests
New ptests
precommit tests


Thanks,

Chaoyu Tang

[jira] [Created] (HIVE-14712) Insert fails when using union all

[BALAMURUGAN (JIRA)]

BALAMURUGAN created HIVE-14712:
--

 Summary: Insert fails when using union all
 Key: HIVE-14712
 URL: https://issues.apache.org/jira/browse/HIVE-14712
 Project: Hive
  Issue Type: Bug
  Components: Database/Schema
Reporter: BALAMURUGAN






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)