Re: Review Request 51695: HIVE-5867: JDBC driver and beeline should support executing an initial SQL script

[cheng xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51695/#review149793
---




jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 237)


No need for "this."



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 250)


LOG.error



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 251)


Please throw SQLException.



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 256)


public method?



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 289)


private method?



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 295)


Please use if-else since you have only switch case.



jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java (line 747)


Do we need this method? It's private.



jdbc/src/test/org/apache/hive/jdbc/TestInitSQL.java (line 48)


Two space indents please.



jdbc/src/test/org/apache/hive/jdbc/TestInitSQL.java (line 53)


This is not a negative case.
"#negative cases:"->"#Some comments"



jdbc/src/test/org/apache/hive/jdbc/TestInitSQL.java (line 55)


Please add the case "#show tables; show/n tables"



jdbc/src/test/org/apache/hive/jdbc/TestInitSQL.java (line 75)


Assert.fail("Test was failed due to " + e);


- cheng xu


On Sept. 21, 2016, 11:47 a.m., Jianguo Tian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51695/
> ---
> 
> (Updated Sept. 21, 2016, 11:47 a.m.)
> 
> 
> Review request for hive and cheng xu.
> 
> 
> Bugs: HIVE-5867
> https://issues.apache.org/jira/browse/HIVE-5867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-5867: JDBC driver and beeline should support executing an initial SQL 
> script
> 
> 
> Diffs
> -
> 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
> ad96a6466dd1aadab71fc261f55be4639dcbe2bf 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 
> 3161566994d6c6e01de9d88a6e87295684619ffa 
>   jdbc/src/test/org/apache/hive/jdbc/TestInitSQL.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51695/diff/
> 
> 
> Testing
> ---
> 
> TestInitSQL.java is JUnit test class which will test method initSql() in 
> HiveConnection.java.
> 
> 
> Thanks,
> 
> Jianguo Tian
> 
>

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

[cheng xu]

> On Sept. 21, 2016, 8:54 a.m., Szehon Ho wrote:
> > This looks straight-forward and good to me (once 2.0.0 is the version in 
> > pom)

Thanks Sezhon for your review. I have updated some versions required by Spark 
side.


- cheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149771
---


On Sept. 21, 2016, 1:27 p.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated Sept. 21, 2016, 1:27 p.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   itests/pom.xml a452db3 
>   pom.xml 2fb78cd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   spark-client/pom.xml 6cf3b17 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

[cheng xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/
---

(Updated Sept. 21, 2016, 1:27 p.m.)


Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-14029
https://issues.apache.org/jira/browse/HIVE-14029


Repository: hive-git


Description
---

There are quite some new optimizations in Spark 2.0.0. We need to bump up Spark 
to 2.0.0 to benefit those performance improvements.


Diffs (updated)
-

  itests/pom.xml a452db3 
  pom.xml 2fb78cd 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
 5b65036 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 53c5c0e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
f6595f1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java a6350d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
 09c54c1 
  ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
ee9f9b7 
  spark-client/pom.xml 6cf3b17 
  
spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java 
e77aa78 
  spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
e3b88d1 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
 e46b67d 
  spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
a7305cf 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
 be14c06 
  
spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
 4420e4d 
  
spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
 5146e91 

Diff: https://reviews.apache.org/r/48839/diff/


Testing
---


Thanks,

cheng xu

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

[cheng xu]

> On Sept. 21, 2016, 3:44 a.m., Sahil Takiar wrote:
> > pom.xml, line 179
> > 
> >
> > Can this be changed to `2.0.0` instead of `2.0.0-preview`
> 
> Sahil Takiar wrote:
> Looked at your updated patch, seems like you already did this.

I forgot to update the review board entry. Reattach file to update it.


- cheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149717
---


On Sept. 21, 2016, 1:27 p.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated Sept. 21, 2016, 1:27 p.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   itests/pom.xml a452db3 
>   pom.xml 2fb78cd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   spark-client/pom.xml 6cf3b17 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 51695: HIVE-5867: JDBC driver and beeline should support executing an initial SQL script

[Jianguo Tian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51695/
---

(Updated Sept. 21, 2016, 3:47 a.m.)


Review request for hive and cheng xu.


Bugs: HIVE-5867
https://issues.apache.org/jira/browse/HIVE-5867


Repository: hive-git


Description
---

HIVE-5867: JDBC driver and beeline should support executing an initial SQL 
script


Diffs (updated)
-

  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
ad96a6466dd1aadab71fc261f55be4639dcbe2bf 
  jdbc/src/java/org/apache/hive/jdbc/Utils.java 
3161566994d6c6e01de9d88a6e87295684619ffa 
  jdbc/src/test/org/apache/hive/jdbc/TestInitSQL.java PRE-CREATION 

Diff: https://reviews.apache.org/r/51695/diff/


Testing
---

TestInitSQL.java is JUnit test class which will test method initSql() in 
HiveConnection.java.


Thanks,

Jianguo Tian

Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests

[Chaoyu Tang]

> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/Query.java, line 122
> > 
> >
> > Will it improve the performance to set the search limit? I did not see 
> > it is used.
> 
> Illya Yalovyy wrote:
> I will be used for different filters. Do you think we should use it for 
> existing filters? Which one in particular? Or you would prefer me to remove 
> this option?
> 
> Please keep in mind that this CR is not about performance.

I thought in the existing implementation, the search limits in some methods 
like findGroupDNByName, findUserDNByName are set to 2 to reduce the returned 
results in case there are many, is not it?


- Chaoyu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51694/#review148634
---


On Sept. 20, 2016, 7:39 p.m., Illya Yalovyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51694/
> ---
> 
> (Updated Sept. 20, 2016, 7:39 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and 
> Szehon Ho.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently LdapAuthenticationProviderImpl class is not covered with unit 
> tests. To make this class testable some minor refactoring will be required.
> 
> 
> Diffs
> -
> 
>   service/pom.xml ecea719 
>   
> service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java
>  efd5393 
>   service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java 
> PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java 
> PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java
>  PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java
>  089a059 
>   
> service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java
>  f276906 
>   service/src/test/org/apache/hive/service/auth/ldap/Credentials.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/LdapTestUtils.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestChainFilter.java 
> PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/auth/ldap/TestCustomQueryFilter.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestLdapUtils.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestQuery.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 
> PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/auth/ldap/TestSearchResultHandler.java
>  PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestUserFilter.java 
> PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/auth/ldap/TestUserSearchFilter.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51694/diff/
> 
> 
> Testing
> ---
> 
> ...hive/service> mvn clean test
> 
> ...
> 
> Results :
> 
> Tests run: 123, Failures: 0, Errors: 0, Skipped: 0
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 04:18 min
> [INFO] Finished at: 

Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests

[Chaoyu Tang]

> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java, line 105
> > 
> >
> > This method might throw out runtime exception such as NPE, 
> > IndexOutOfBoundsException, should we check the passed in parameter rdn? 
> > We might not run into this situation in old code, but since this line 
> > of code is refactored as a separate API, I think we should do the check. 
> > Same for the other methods like patternToBaseDn etc.
> 
> Illya Yalovyy wrote:
> Agree. The DN parsing in general implemented quite poorly. I have a task 
> already to re-implement it completely. There are many problems with current 
> one. Handling incorrect format or invalid input is only one of them. My 
> intention is to use RDN java class to do a correct parsing. 
> 
> https://docs.oracle.com/javase/7/docs/api/javax/naming/ldap/Rdn.html
> 
> I think we can leave it for now, and I will submit another CR that 
> addresses this concern sortly.

OK


> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java, line 108
> > 
> >
> > can getSingleLdapName be used to enforce only one returned entry? that 
> > API in SearachResultHandler is never used.
> 
> Illya Yalovyy wrote:
> I was trying to copy existing logic. At the moment, I don't want to do 
> any changes to that. This particular code should be improved in separate CR. 
> Does it make sense?

OK


> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java, line 159
> > 
> >
> > I am not sure if there is any precedence for these configurations, but 
> > here it seems that the GUIDKEY/BASEDN takes precedence over DNPATTERN, 
> > which is different from the existing implementation and cause the behavior 
> > change.
> 
> Illya Yalovyy wrote:
> Could you please give more details on the case when the behavior will be 
> different. 
> The logic seems to be same: It uses GUIDKEY/BASEDN only when DNPATTERN is 
> not configured:
> 
> if (StringUtils.isBlank(patternsString)) {
> ...
> 
> Which means *only* if patternsString is blank, try to use GUIDKEY/BASEDN.
> 
> Please let me know if I did not get it correctly.

Never mind.


- Chaoyu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51694/#review148634
---


On Sept. 20, 2016, 7:39 p.m., Illya Yalovyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51694/
> ---
> 
> (Updated Sept. 20, 2016, 7:39 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and 
> Szehon Ho.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently LdapAuthenticationProviderImpl class is not covered with unit 
> tests. To make this class testable some minor refactoring will be required.
> 
> 
> Diffs
> -
> 
>   service/pom.xml ecea719 
>   
> service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java
>  efd5393 
>   service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java 
> PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java 
> PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java
>  PRE-CREATION 
>   
> 

[jira] [Created] (HIVE-14803) S3: Stats gathering for insert queries can be expensive for partitioned dataset

[Rajesh Balamohan (JIRA)]

Rajesh Balamohan created HIVE-14803:
---

 Summary: S3: Stats gathering for insert queries can be expensive 
for partitioned dataset
 Key: HIVE-14803
 URL: https://issues.apache.org/jira/browse/HIVE-14803
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Priority: Minor


StatsTask's aggregateStats populates stats details for all partitions by 
checking the file sizes which turns out to be expensive when larger number of 
partitions are inserted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

[Szehon Ho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149771
---


Ship it!




This looks straight-forward and good to me (once 2.0.0 is the version in pom)

- Szehon Ho


On June 17, 2016, 8:52 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated June 17, 2016, 8:52 a.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   pom.xml 63a5ae1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4b34ebf 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests

[Szehon Ho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51694/#review149769
---


Ship it!




This looks like a great refactoring to me. This class was always hard to 
understand, and this makes it a little easier.  I'll defer to Chaoyu to the 
comments.

- Szehon Ho


On Sept. 20, 2016, 7:39 p.m., Illya Yalovyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51694/
> ---
> 
> (Updated Sept. 20, 2016, 7:39 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and 
> Szehon Ho.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently LdapAuthenticationProviderImpl class is not covered with unit 
> tests. To make this class testable some minor refactoring will be required.
> 
> 
> Diffs
> -
> 
>   service/pom.xml ecea719 
>   
> service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java
>  efd5393 
>   service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java 
> PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java 
> PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java
>  PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java
>  089a059 
>   
> service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java
>  f276906 
>   service/src/test/org/apache/hive/service/auth/ldap/Credentials.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/LdapTestUtils.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestChainFilter.java 
> PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/auth/ldap/TestCustomQueryFilter.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestLdapUtils.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestQuery.java 
> PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 
> PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/auth/ldap/TestSearchResultHandler.java
>  PRE-CREATION 
>   service/src/test/org/apache/hive/service/auth/ldap/TestUserFilter.java 
> PRE-CREATION 
>   
> service/src/test/org/apache/hive/service/auth/ldap/TestUserSearchFilter.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51694/diff/
> 
> 
> Testing
> ---
> 
> ...hive/service> mvn clean test
> 
> ...
> 
> Results :
> 
> Tests run: 123, Failures: 0, Errors: 0, Skipped: 0
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 04:18 min
> [INFO] Finished at: 2016-09-06T08:46:04-07:00
> [INFO] Final Memory: 66M/984M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Illya Yalovyy
> 
>

[jira] [Created] (HIVE-14802) bogus HiveMetaStore.getDelegationToken calls in HS2 session opening

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-14802:
---

 Summary: bogus HiveMetaStore.getDelegationToken calls in HS2 
session opening
 Key: HIVE-14802
 URL: https://issues.apache.org/jira/browse/HIVE-14802
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Not sure what's going on. Tons of these in some log, looks like an internal 
error {noformat}
2016-09-07 07:57:27,226 ERROR [HiveServer2-HttpHandler-Pool: Thread-22520]: 
metadata.Hive (Hive.java:getDelegationToken(2890)) - 
java.lang.UnsupportedOperationException: getDelegationToken() can be called 
only in thrift (non local) mode
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:1664)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
at com.sun.proxy.$Proxy8.getDelegationToken(Unknown Source)
at 
org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2888)
at 
org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:468)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.getDelegationToken(ThriftCLIService.java:340)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:321)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:235)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
at 
org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:101)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:349)
at 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
at 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at 
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14801) improve TestPartitionNameWhitelistValidation stability

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-14801:


 Summary: improve TestPartitionNameWhitelistValidation stability
 Key: HIVE-14801
 URL: https://issues.apache.org/jira/browse/HIVE-14801
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair
Assignee: Thejas M Nair


TestPartitionNameWhitelistValidation uses remote metastore. However, there can 
be multiple issues around startup of remote metastore, including race 
conditions in finding available port. In addition, all the initialization done 
at startup of remote metastore is likely to make the test case take more time.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

[Sahil Takiar]

> On Sept. 20, 2016, 7:44 p.m., Sahil Takiar wrote:
> > pom.xml, line 179
> > 
> >
> > Can this be changed to `2.0.0` instead of `2.0.0-preview`

Looked at your updated patch, seems like you already did this.


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149717
---


On June 17, 2016, 8:52 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated June 17, 2016, 8:52 a.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   pom.xml 63a5ae1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4b34ebf 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 48839: HIVE-14029: Update Spark version to 2.0.0

[Sahil Takiar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48839/#review149717
---




pom.xml (line 179)


Can this be changed to `2.0.0` instead of `2.0.0-preview`


- Sahil Takiar


On June 17, 2016, 8:52 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48839/
> ---
> 
> (Updated June 17, 2016, 8:52 a.m.)
> 
> 
> Review request for hive, Rui Li, Sergio Pena, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-14029
> https://issues.apache.org/jira/browse/HIVE-14029
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> There are quite some new optimizations in Spark 2.0.0. We need to bump up 
> Spark to 2.0.0 to benefit those performance improvements.
> 
> 
> Diffs
> -
> 
>   pom.xml 63a5ae1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
>  5b65036 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 
> 53c5c0e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 
> f6595f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SortByShuffler.java 
> a6350d3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobMetricsListener.java
>  09c54c1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4b34ebf 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java 
> ee9f9b7 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/MetricsCollection.java
>  e77aa78 
>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 
> e3b88d1 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/InputMetrics.java
>  e46b67d 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/Metrics.java 
> a7305cf 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleReadMetrics.java
>  be14c06 
>   
> spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java
>  4420e4d 
>   
> spark-client/src/test/java/org/apache/hive/spark/client/TestMetricsCollection.java
>  5146e91 
> 
> Diff: https://reviews.apache.org/r/48839/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> cheng xu
> 
>

[jira] [Created] (HIVE-14800) Handle off by 3 in ORC split generation based on split strategy used

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-14800:
-

 Summary: Handle off by 3 in ORC split generation based on split 
strategy used
 Key: HIVE-14800
 URL: https://issues.apache.org/jira/browse/HIVE-14800
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


BI will apparently generate splits starting at offset 0.
ETL will skip the ORC header and generate a split starting at offset 3.

There's a workaround in the HiveSplitGenreator to handle this for consistent 
splits. Ideally, Orc split generation should take care of this.


cc [~prasanth_j], [~gopalv]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests

[Illya Yalovyy]

> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, 
> > line 37
> > 
> >
> > Do we really need an extra factory layer and have a factory for each 
> > filter?
> > In Hive, actaully each session instantiates its own 
> > LdapAuthenticationProviderImpl, which now contains different factories with 
> > each one generating only one instance of its filter.

I think so. Factories encapsulate logic related to choosing whether particular 
filter is required or not based on the provided configuration.


> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java, line 108
> > 
> >
> > can getSingleLdapName be used to enforce only one returned entry? that 
> > API in SearachResultHandler is never used.

I was trying to copy existing logic. At the moment, I don't want to do any 
changes to that. This particular code should be improved in separate CR. Does 
it make sense?


> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java, line 105
> > 
> >
> > This method might throw out runtime exception such as NPE, 
> > IndexOutOfBoundsException, should we check the passed in parameter rdn? 
> > We might not run into this situation in old code, but since this line 
> > of code is refactored as a separate API, I think we should do the check. 
> > Same for the other methods like patternToBaseDn etc.

Agree. The DN parsing in general implemented quite poorly. I have a task 
already to re-implement it completely. There are many problems with current 
one. Handling incorrect format or invalid input is only one of them. My 
intention is to use RDN java class to do a correct parsing. 

https://docs.oracle.com/javase/7/docs/api/javax/naming/ldap/Rdn.html

I think we can leave it for now, and I will submit another CR that addresses 
this concern sortly.


> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java, line 159
> > 
> >
> > I am not sure if there is any precedence for these configurations, but 
> > here it seems that the GUIDKEY/BASEDN takes precedence over DNPATTERN, 
> > which is different from the existing implementation and cause the behavior 
> > change.

Could you please give more details on the case when the behavior will be 
different. 
The logic seems to be same: It uses GUIDKEY/BASEDN only when DNPATTERN is not 
configured:

if (StringUtils.isBlank(patternsString)) {
...

Which means *only* if patternsString is blank, try to use GUIDKEY/BASEDN.

Please let me know if I did not get it correctly.


> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/Query.java, line 122
> > 
> >
> > Will it improve the performance to set the search limit? I did not see 
> > it is used.

I will be used for different filters. Do you think we should use it for 
existing filters? Which one in particular? Or you would prefer me to remove 
this option?

Please keep in mind that this CR is not about performance.


- Illya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51694/#review148634
---


On Sept. 7, 2016, 2:24 p.m., Illya Yalovyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51694/
> ---
> 
> (Updated Sept. 7, 2016, 2:24 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and 
> Szehon Ho.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently LdapAuthenticationProviderImpl class is not covered with unit 
> tests. To make this class testable some minor refactoring will be required.
> 
> 
> Diffs
> -
> 
>   service/pom.xml ecea719 
>   
> service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java
>  efd5393 
>   service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java 
> PRE-CREATION 
>   
> service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java 
> PRE-CREATION 
>   

Re: Review Request 50525: HIVE-14341: Altered skewed location is not respected for list bucketing

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50525/#review149671
---




ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java (line 234)


Actually the original one is also not overwrite but move from a temp file 
to the final location. While that would fail if an previous file already 
exists. Then we should try to rename to '_copy_1' and so.



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java (line 899)


Yes. That's where this issue comes from. Without this change, even if you 
alter the skew location which will be saved in locationMap, it would get the 
old location and save it to overwrite the newer location. That is not correct.

The new logic is: if we have the mapped location for the skew keys, then 
use it. Otherwise, get the default skew location.


- Aihua Xu


On Sept. 19, 2016, 9:02 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50525/
> ---
> 
> (Updated Sept. 19, 2016, 9:02 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14341: Altered skewed location is not respected for list bucketing
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java e386717 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java da46854 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
>  ba4f6a7 
>   ql/src/test/queries/clientpositive/create_alter_list_bucketing_table1.q 
> bf89e8f 
>   ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out 
> 216d3be 
> 
> Diff: https://reviews.apache.org/r/50525/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

[jira] [Created] (HIVE-14799) Query operation are not thread safe during its cancellation

[Chaoyu Tang (JIRA)]

Chaoyu Tang created HIVE-14799:
--

 Summary: Query operation are not thread safe during its 
cancellation
 Key: HIVE-14799
 URL: https://issues.apache.org/jira/browse/HIVE-14799
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


When a query is cancelled either via Beeline (Ctrl-C) or API call 
TCLIService.Client.CancelOperation, SQLOperation.cancel is invoked in a 
different thread from that running the query to close/destroy its encapsulated 
Driver object. Both SQLOperation and Driver are not thread-safe which could 
sometimes result in Runtime exceptions like NPE. The errors from the running 
query are not handled properly therefore probably causing some stuffs (files, 
locks etc) not being cleaned after the query termination.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14798) MSCK REPAIR TABLE throws null pointer exception

[Anbu Cheeralan (JIRA)]

Anbu Cheeralan created HIVE-14798:
-

 Summary: MSCK REPAIR TABLE throws null pointer exception
 Key: HIVE-14798
 URL: https://issues.apache.org/jira/browse/HIVE-14798
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.1.0
Reporter: Anbu Cheeralan


MSKC REPAIR TABLE statement throws null pointer exception in Hive 2.1
I have tested the same against external/internal tables created both in HDFS 
and in Google Cloud.

The error shown in beeline/sql client 
"FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask"

Hive Logs:

2016-09-14T04:08:02,433 ERROR [main]: metadata.HiveMetaStoreChecker (:()) - 
java.lang.NullPointerException
2016-09-14T04:08:02,434 WARN  [main]: exec.DDLTask (:()) - Failed to run 
metacheck:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:444)
at 
org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:448

Here are the steps to recreate this issue:
use default
DROP TABLE IF EXISTS repairtable
CREATE TABLE repairtable(col STRING) PARTITIONED BY (p1 STRING, p2 STRING)
MSCK REPAIR TABLE default.repairtable



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52079: HIVE-14753: Track the number of open/closed/abandoned sessions in HS2

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52079/#review149648
---



LGTM,
Thanks for the patch

- Peter Vary


On Sept. 20, 2016, 10:37 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52079/
> ---
> 
> (Updated Sept. 20, 2016, 10:37 a.m.)
> 
> 
> Review request for hive, Gabor Szadovszky, Peter Vary, and Sergio Pena.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14753: Track the number of open/closed/abandoned sessions in HS2
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 
> 9be9b50aa02ff88816eb92079eaff9afa3e1be90 
>   common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 
> 4297233ed12a7d9a2fa03ac3204e8335c0aed821 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
>  9dc96f9c6412720a891b5c55e2074049c893d780 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
>  4c433678bd62ea74b80babce9856681192deb25f 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/MetricVariableRatioGauge.java
>  PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java 
> 46676589e6656d0f13f1931bfe67a63dd1920042 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestMetricVariableRatioGauge.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 15bab0660fcb9a997d66f6ff0a5dbc0e39c37ae7 
>   
> service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
>  5511c54ff431211f7f72deaa017c915b839dfb2a 
> 
> Diff: https://reviews.apache.org/r/52079/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests in the common and the ql subprojects.
> Manually verified the metrics using the HS2 webui metric dump.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 52079: HIVE-14753: Track the number of open/closed/abandoned sessions in HS2

[Gabor Szadovszky]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52079/#review149643
---


Ship it!




Thanks for the patch. LGTM.

- Gabor Szadovszky


On Sept. 20, 2016, 10:37 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52079/
> ---
> 
> (Updated Sept. 20, 2016, 10:37 a.m.)
> 
> 
> Review request for hive, Gabor Szadovszky, Peter Vary, and Sergio Pena.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14753: Track the number of open/closed/abandoned sessions in HS2
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 
> 9be9b50aa02ff88816eb92079eaff9afa3e1be90 
>   common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 
> 4297233ed12a7d9a2fa03ac3204e8335c0aed821 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
>  9dc96f9c6412720a891b5c55e2074049c893d780 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
>  4c433678bd62ea74b80babce9856681192deb25f 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/MetricVariableRatioGauge.java
>  PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java 
> 46676589e6656d0f13f1931bfe67a63dd1920042 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestMetricVariableRatioGauge.java
>  PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 15bab0660fcb9a997d66f6ff0a5dbc0e39c37ae7 
>   
> service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
>  5511c54ff431211f7f72deaa017c915b839dfb2a 
> 
> Diff: https://reviews.apache.org/r/52079/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests in the common and the ql subprojects.
> Manually verified the metrics using the HS2 webui metric dump.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 50525: HIVE-14341: Altered skewed location is not respected for list bucketing

[Yongzhi Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50525/#review149640
---




ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java (line 234)


Any reason you change the logic from replace(overwrite) to something 
like(insert into)?


- Yongzhi Chen


On Sept. 19, 2016, 9:02 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50525/
> ---
> 
> (Updated Sept. 19, 2016, 9:02 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14341: Altered skewed location is not respected for list bucketing
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java e386717 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java da46854 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
>  ba4f6a7 
>   ql/src/test/queries/clientpositive/create_alter_list_bucketing_table1.q 
> bf89e8f 
>   ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out 
> 216d3be 
> 
> Diff: https://reviews.apache.org/r/50525/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Review Request 52079: HIVE-14753: Track the number of open/closed/abandoned sessions in HS2

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52079/
---

Review request for hive, Gabor Szadovszky, Peter Vary, and Sergio Pena.


Repository: hive-git


Description
---

HIVE-14753: Track the number of open/closed/abandoned sessions in HS2


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/metrics/LegacyMetrics.java 
9be9b50aa02ff88816eb92079eaff9afa3e1be90 
  common/src/java/org/apache/hadoop/hive/common/metrics/common/Metrics.java 
4297233ed12a7d9a2fa03ac3204e8335c0aed821 
  
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
 9dc96f9c6412720a891b5c55e2074049c893d780 
  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
 4c433678bd62ea74b80babce9856681192deb25f 
  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/MetricVariableRatioGauge.java
 PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/metrics/MetricsTestUtils.java 
46676589e6656d0f13f1931bfe67a63dd1920042 
  
common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestMetricVariableRatioGauge.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
15bab0660fcb9a997d66f6ff0a5dbc0e39c37ae7 
  
service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
 5511c54ff431211f7f72deaa017c915b839dfb2a 

Diff: https://reviews.apache.org/r/52079/diff/


Testing
---

Ran the unit tests in the common and the ql subprojects.
Manually verified the metrics using the HS2 webui metric dump.


Thanks,

Barna Zsombor Klara

Re: Review Request 50525: HIVE-14341: Altered skewed location is not respected for list bucketing

[Yongzhi Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50525/#review149635
---




ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java (line 899)


You change old logic here a little bit in following case:
When locationMap has skewedValsCandidate, but
allSkewedVals.contains(skewedValsCandidate) == false

Before your change, it uses defaultKey in locationMap while after the 
change, skewedValsCandidate is used. 
Is that safe?


- Yongzhi Chen


On Sept. 19, 2016, 9:02 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50525/
> ---
> 
> (Updated Sept. 19, 2016, 9:02 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14341: Altered skewed location is not respected for list bucketing
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java e386717 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java da46854 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
>  ba4f6a7 
>   ql/src/test/queries/clientpositive/create_alter_list_bucketing_table1.q 
> bf89e8f 
>   ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out 
> 216d3be 
> 
> Diff: https://reviews.apache.org/r/50525/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

[jira] [Created] (HIVE-14797) reducer number estimating may lead to data skew

[roncenzhao (JIRA)]

roncenzhao created HIVE-14797:
-

 Summary: reducer number estimating may lead to data skew
 Key: HIVE-14797
 URL: https://issues.apache.org/jira/browse/HIVE-14797
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: roncenzhao
Assignee: roncenzhao


HiveKey's hash code is generated by multipling by 31 key by key which is 
implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
for (int i = 0; i < bucketFields.length; i++) {
  int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i], 
bucketFieldInspectors[i]);
  hashCode = 31 * hashCode + fieldHash;
}

The follow example will lead to data skew:

I hava two table called tbl1 and tbl2 and they have the same column: a int, b 
string. The values of column 'a' in both two tables are not skew, but values of 
column 'b' in both two tables are skew.

When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and 
tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data 
skew.

As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`. 
When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the 
result, the job will be skew.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14796) MetastoreEventListener - OnGrant() and OnRevoke() Events required for capturing the event on grant and revoke operation on the table in hive.

[Rahul Dhote (JIRA)]

Rahul Dhote created HIVE-14796:
--

 Summary: MetastoreEventListener - OnGrant() and OnRevoke() Events 
required for capturing the event on grant and revoke operation on the table in 
hive. 
 Key: HIVE-14796
 URL: https://issues.apache.org/jira/browse/HIVE-14796
 Project: Hive
  Issue Type: New Feature
  Components: Authorization
Affects Versions: 1.2.1
 Environment: RHEL6 and RHEL7
Reporter: Rahul Dhote


During granting and revoking privileges on the table, OnGrant and OnRevoke 
method is required inside the MetastoreEventListener for capturing the events.
It would be useful for doing certain operation on this basic authorization 
events. 

Ex:

 /**
   * @param OnGrantEvent grant event
   * @throws MetaException
   */
  public void onAddIndex(GrantEvent grantEvent) throws MetaException {
  }

  /**
   * @param OnRevoke revoke event
   * @throws MetaException
   */
  public void onDropIndex(RevokeEvent revokeEvent) throws MetaException {
  }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)