[jira] [Created] (HIVE-16158) Correct mistake in documentation for ALTER TABLE … ADD/REPLACE COLUMNS CASCADE
Illya Yalovyy created HIVE-16158: Summary: Correct mistake in documentation for ALTER TABLE … ADD/REPLACE COLUMNS CASCADE Key: HIVE-16158 URL: https://issues.apache.org/jira/browse/HIVE-16158 Project: Hive Issue Type: Bug Components: Documentation Affects Versions: 1.0.0 Reporter: Illya Yalovyy Current documentation says that key word CASCADE was introduced in Hive 0.15 release. That information is incorrect and confuses users. The feature was actually released in Hive 1.1.0. (HIVE-8839) https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Add/ReplaceColumns -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 56688: HIVE-15881: Use new thread count variable name instead of mapred.dfsclient.parallelism.max
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56688/#review166384 --- ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java (line 2080) <https://reviews.apache.org/r/56688/#comment238311> I feel like we need a unit test for this method. - Illya Yalovyy On Feb. 22, 2017, 4:28 p.m., Sergio Pena wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56688/ > --- > > (Updated Feb. 22, 2017, 4:28 p.m.) > > > Review request for hive. > > > Bugs: HIVE-15881 > https://issues.apache.org/jira/browse/HIVE-15881 > > > Repository: hive-git > > > Description > --- > > Deprecates the use of mapred.dfsclient.parallelism.max, and instead use > hive.exec.input.listing.max.threads > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > 1af59ba5879d3ad24148fa86f8ce725acef2bb63 > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > e81cbce3e333d44a4088c10491f399e92a505293 > ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java > 5a9d83ca4890a68a63f8f6c145d30906a58c7e73 > > Diff: https://reviews.apache.org/r/56688/diff/ > > > Testing > --- > > Added tests to TestUtilities. > Waiting for HiveQA > > > Thanks, > > Sergio Pena > >
Re: Review Request 54711: HIVE-1555 [currently support only reading primitive types from oracle db]
6>
Style:
It seems like in Hive code base annotations are on a separate line. Please
make sure your files pass checkstyle.
jdbc-handler/src/java/org/apache/hive/storagehandler/db/oracle/HiveOracleVendorBridge.java
(line 25)
<https://reviews.apache.org/r/54711/#comment230238>
It is better to use try-catch with resource:
http://stackoverflow.com/questions/9260159/java-7-automatic-resource-management-jdbc-try-with-resources-statement
jdbc-handler/src/java/org/apache/hive/storagehandler/db/oracle/HiveOracleVendorBridge.java
(line 38)
<https://reviews.apache.org/r/54711/#comment230239>
Allocated resources are not released.
jdbc-handler/src/java/org/apache/hive/storagehandler/db/oracle/HiveOracleVendorBridge.java
(line 74)
<https://reviews.apache.org/r/54711/#comment230240>
return String.format("SELECT \* FROM %s WHERE ROWNUM = 0", tableName);
The name should be escaped.
For example:
http://stackoverflow.com/questions/11629966/how-to-handle-table-column-named-with-reserved-sql-keyword
- Illya Yalovyy
On Dec. 13, 2016, 5:14 p.m., Dmitry Zagorulkin wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54711/
> ---
>
> (Updated Dec. 13, 2016, 5:14 p.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> Please review my aproach for HIVE-1555 implementation. any feedback are
> welcome.
>
>
> Diffs
> -
>
> jdbc-handler/pom.xml PRE-CREATION
>
> jdbc-handler/src/java/org/apache/hive/storagehandler/JDBCStorageHandler.java
> PRE-CREATION
>
> jdbc-handler/src/java/org/apache/hive/storagehandler/db/HiveJDBCTypeBridge.java
> PRE-CREATION
>
> jdbc-handler/src/java/org/apache/hive/storagehandler/db/HiveJDBCTypeBridgeUtil.java
> PRE-CREATION
>
> jdbc-handler/src/java/org/apache/hive/storagehandler/db/HiveJDBCVendorBridge.java
> PRE-CREATION
>
> jdbc-handler/src/java/org/apache/hive/storagehandler/db/oracle/HiveOracleJDBCDBWritable.java
> PRE-CREATION
>
> jdbc-handler/src/java/org/apache/hive/storagehandler/db/oracle/HiveOracleTypeBridge.java
> PRE-CREATION
>
> jdbc-handler/src/java/org/apache/hive/storagehandler/db/oracle/HiveOracleVendorBridge.java
> PRE-CREATION
> jdbc-handler/src/java/org/apache/hive/storagehandler/serde/JDBCSerde.java
> PRE-CREATION
>
> jdbc-handler/src/java/org/apache/hive/storagehandler/serde/TypeHiveJDBCConversion.java
> PRE-CREATION
>
> jdbc-handler/src/test/org/apache/hive/storagehandler/serde/HiveOracleJDBCDBWritableTest.java
> PRE-CREATION
> packaging/pom.xml 76e0cffdcfac4c4c6aed73a1ca479716857cc659
> packaging/src/main/assembly/src.xml
> f27911235b995850a444c9640a0e3b2090551665
> pom.xml 3d8fa1a044e9da94efef5bef2e01d9959f3d8e92
>
> Diff: https://reviews.apache.org/r/54711/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Dmitry Zagorulkin
>
>
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/ --- (Updated Dec. 12, 2016, 5:51 p.m.) Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Changes --- 1. added INFO log record for successful authentication 2. brought back WARN log records for unexpected LDAP exceptions Repository: hive-git Description --- HIVE-15076 Improve scalability of LDAP authentication provider group filter https://issues.apache.org/jira/browse/HIVE-15076 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 152c4b2 service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java e9172d3 service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java cd62935 service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java 4fad755 service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java acde8c1 service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 0cc2ead service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 499b624 service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 3054e33 service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION Diff: https://reviews.apache.org/r/53204/diff/ Testing --- Build succeeded. Test results: Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:14 min [INFO] Finished at: 2016-10-26T13:53:15-07:00 [INFO] Final Memory: 36M/1091M [INFO] Thanks, Illya Yalovyy
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
> On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2426
> > <https://reviews.apache.org/r/53204/diff/3/?file=1579539#file1579539line2426>
> >
> > Just curious why we don't just put the constant string
> > "hive.server2.authentication.ldap.userMembershipKey" here like most of
> > other entries?
>
> Illya Yalovyy wrote:
> Because it uses in several places. In particular in documentation.
> Putting a string in documentation is not maintainable, because later someone
> can change the string and forget to update in in all places. Documentation
> would become stale. In such a big project in will be a problem. JavaDoc has
> means to prevent that from happening by using string constants in
> documentation sections.
>
> Aihua Xu wrote:
> I got what you mean. But you can define as
> HIVE_SERVER2_PLAIN_LDAP_USERMEMBERSHIP_KEY("the string", null) and then refer
> the string as
> HiveConf.ConfVars.HIVE_SERVER2_PLAIN_LDAP_USERMEMBERSHIP_KEY.varname. Is that
> what you want?
Technically not a constant and one cannot reference it in that context.
> On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java,
> > line 115
> > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line115>
> >
> > This should be info level which will be consistent with
> > GroupMembershipKeyFilter class.
>
> Illya Yalovyy wrote:
> Ok. I'll generate 2 log entries then: 1. INFO without group information;
> 2. DEBUG with full information.
>
> Does it make sense?
>
> See Naveen's comments for more details.
>
> Illya Yalovyy wrote:
> Actually this is a bit different. I'll change it to INFO.
>
> Aihua Xu wrote:
> Now I'm convinced for having 2 entries, 1. INFO without group
> information; 2. DEBUG with full information.
>
> What do you mean it's different?
This particular record doesn't contain sensitive information.
- Illya
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53204/#review158532
---
On Dec. 9, 2016, 1:03 a.m., Illya Yalovyy wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53204/
> ---
>
> (Updated Dec. 9, 2016, 1:03 a.m.)
>
>
> Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon
> Ho.
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> HIVE-15076 Improve scalability of LDAP authentication provider group filter
>
> https://issues.apache.org/jira/browse/HIVE-15076
>
>
> Diffs
> -
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751
> service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088
> service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java
> 152c4b2
> service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea
> service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938
> service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java
> e9172d3
>
> service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java
> cd62935
>
> service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java
> 4fad755
>
> service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java
> acde8c1
> service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java
> 0cc2ead
> service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java
> 499b624
> service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java
> 3054e33
> service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION
> service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION
>
> Diff: https://reviews.apache.org/r/53204/diff/
>
>
> Testing
> ---
>
> Build succeeded.
>
> Test results:
>
> Tests run: 149, Failures: 0, Errors: 0, Skipped: 0
>
> [INFO]
> ----
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 03:14 min
> [INFO] Finished at: 2016-10-26T13:53:15-07:00
> [INFO] Final Memory: 36M/1091M
> [INFO]
>
>
>
> Thanks,
>
> Illya Yalovyy
>
>
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/ --- (Updated Dec. 9, 2016, 1:03 a.m.) Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Changes --- 1. Updated logging 2. Added exception to error messages 3. Trivial style correction 4. Test methods renamed according to the actual filter implementation names Repository: hive-git Description --- HIVE-15076 Improve scalability of LDAP authentication provider group filter https://issues.apache.org/jira/browse/HIVE-15076 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 152c4b2 service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java e9172d3 service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java cd62935 service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java 4fad755 service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java acde8c1 service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 0cc2ead service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 499b624 service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 3054e33 service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION Diff: https://reviews.apache.org/r/53204/diff/ Testing --- Build succeeded. Test results: Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:14 min [INFO] Finished at: 2016-10-26T13:53:15-07:00 [INFO] Final Memory: 36M/1091M [INFO] Thanks, Illya Yalovyy
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
> On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java, > > line 265 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579546#file1579546line265> > > > > You may need to add some tests for the default configuraiton which is > > null for HIVE_SERVER2_PLAIN_LDAP_USERMEMBERSHIP_KEY. > > Illya Yalovyy wrote: > If HIVE_SERVER2_PLAIN_LDAP_USERMEMBERSHIP_KEY is NULL this filter will > not be used. I think we have enough test for this case. Did I get you > correctly? Could you please provide more details about the test case you have > in mind? I think this use case is tested in #testAuthenticateWhenGroupFilterPasses(). Probably I should rename other tests to make it clear. - Illya --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/#review158532 ------- On Dec. 8, 2016, 12:45 a.m., Illya Yalovyy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53204/ > --- > > (Updated Dec. 8, 2016, 12:45 a.m.) > > > Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon > Ho. > > > Repository: hive-git > > > Description > --- > > HIVE-15076 Improve scalability of LDAP authentication provider group filter > > https://issues.apache.org/jira/browse/HIVE-15076 > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 > service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java > 152c4b2 > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea > service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java > e9172d3 > > service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java > cd62935 > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java > 4fad755 > > service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java > acde8c1 > service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java > 0cc2ead > service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java > 499b624 > service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java > 3054e33 > service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION > service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION > > Diff: https://reviews.apache.org/r/53204/diff/ > > > Testing > --- > > Build succeeded. > > Test results: > > Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > -------- > [INFO] Total time: 03:14 min > [INFO] Finished at: 2016-10-26T13:53:15-07:00 > [INFO] Final Memory: 36M/1091M > [INFO] > > > > Thanks, > > Illya Yalovyy > >
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
> On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, > > line 145 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line145> > > > > You may need to change message since it's expected that the user is not > > in some groups. Probably change to "Cannot match user ... and group ..." > > since "Failed to" seems to be an error. > > Illya Yalovyy wrote: > I will update the message. Usually it should just return true or false. If it fails with exception then something is wrong. That was reflected in the message. I noticed that I'm hiding the exception, which is a very bad practice. Will fix it as well. May be even WARN log message with exception details is required here. What you think? Again it should not happen usually, if it does - something wrong. - Illya --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/#review158532 ----------- On Dec. 8, 2016, 12:45 a.m., Illya Yalovyy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53204/ > --- > > (Updated Dec. 8, 2016, 12:45 a.m.) > > > Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon > Ho. > > > Repository: hive-git > > > Description > --- > > HIVE-15076 Improve scalability of LDAP authentication provider group filter > > https://issues.apache.org/jira/browse/HIVE-15076 > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 > service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java > 152c4b2 > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea > service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java > e9172d3 > > service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java > cd62935 > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java > 4fad755 > > service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java > acde8c1 > service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java > 0cc2ead > service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java > 499b624 > service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java > 3054e33 > service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION > service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION > > Diff: https://reviews.apache.org/r/53204/diff/ > > > Testing > --- > > Build succeeded. > > Test results: > > Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > -------- > [INFO] Total time: 03:14 min > [INFO] Finished at: 2016-10-26T13:53:15-07:00 > [INFO] Final Memory: 36M/1091M > [INFO] > > > > Thanks, > > Illya Yalovyy > >
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
> On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java, line > > 138 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579544#file1579544line138> > > > > Looks like we won't handle NPE so NPE may cause some problems. > > > > If userMembershipAttr is null, will we still check userMememberOfGroup > > or not? If not, maybe we should handle such exception here. How about > > groupMembershipAttr above? Seems we will have such issue as well. > > Illya Yalovyy wrote: > I think it should not happen, but I'll double check. The filter will not be created for a case when 'hive.server2.authentication.ldap.userMembershipKey' is not set (NULL). It means we don't have to handle null in this code. If this NPE happens, it means there is a bug in the code. > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, > > line 115 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line115> > > > > This should be info level which will be consistent with > > GroupMembershipKeyFilter class. > > Illya Yalovyy wrote: > Ok. I'll generate 2 log entries then: 1. INFO without group information; > 2. DEBUG with full information. > > Does it make sense? > > See Naveen's comments for more details. Actually this is a bit different. I'll change it to INFO. - Illya --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/#review158532 --- On Dec. 8, 2016, 12:45 a.m., Illya Yalovyy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53204/ > --- > > (Updated Dec. 8, 2016, 12:45 a.m.) > > > Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon > Ho. > > > Repository: hive-git > > > Description > --- > > HIVE-15076 Improve scalability of LDAP authentication provider group filter > > https://issues.apache.org/jira/browse/HIVE-15076 > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 > service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java > 152c4b2 > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea > service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java > e9172d3 > > service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java > cd62935 > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java > 4fad755 > > service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java > acde8c1 > service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java > 0cc2ead > service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java > 499b624 > service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java > 3054e33 > service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION > service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION > > Diff: https://reviews.apache.org/r/53204/diff/ > > > Testing > --- > > Build succeeded. > > Test results: > > Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 > > [INFO] > -------- > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 03:14 min > [INFO] Finished at: 2016-10-26T13:53:15-07:00 > [INFO] Final Memory: 36M/1091M > [INFO] > > > > Thanks, > > Illya Yalovyy > >
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
> On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2426 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579539#file1579539line2426> > > > > Just curious why we don't just put the constant string > > "hive.server2.authentication.ldap.userMembershipKey" here like most of > > other entries? Because it uses in several places. In particular in documentation. Putting a string in documentation is not maintainable, because later someone can change the string and forget to update in in all places. Documentation would become stale. In such a big project in will be a problem. JavaDoc has means to prevent that from happening by using string constants in documentation sections. > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, > > line 90 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line90> > > > > This seems to be a useful info that will help in diagnostics. Wondering > > why changes from info to debug level? I totally agree, but Naveen doesn't want to expose group names in logs. It is a questionable concern, but moving it to DEBUG may be a good compromise. > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, > > line 115 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line115> > > > > This should be info level which will be consistent with > > GroupMembershipKeyFilter class. Ok. I'll generate 2 log entries then: 1. INFO without group information; 2. DEBUG with full information. Does it make sense? See Naveen's comments for more details. > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, > > line 124 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line124> > > > > Seems 'warn' is not necessary since that could be expected in the for > > loop, right? It means we have a group in configuration that doesn't exist... Would you recommend log it at DEBUG level? > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, > > line 132 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line132> > > > > Since we are throwing the exception, I guess such debug may be > > redundant. We should display such exception in the caller somewhere. Exception message has a different (less descriptive) message. Please see Naveen's comments for more details. > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, > > line 139 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line139> > > > > Seems this could be a info level message. Same here. > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java, > > line 145 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579541#file1579541line145> > > > > You may need to change message since it's expected that the user is not > > in some groups. Probably change to "Cannot match user ... and group ..." > > since "Failed to" seems to be an error. I will update the message. > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java, > > line 265 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579546#file1579546line265> > > > > You may need to add some tests for the default configuraiton which is > > null for HIVE_SERVER2_PLAIN_LDAP_USERMEMBERSHIP_KEY. If HIVE_SERVER2_PLAIN_LDAP_USERMEMBERSHIP_KEY is NULL this filter will not be used. I think we have enough test for this case. Did I get you correctly? Could you please provide more details about the test case you have in mind? > On Dec. 8, 2016, 3:55 p.m., Aihua Xu wrote: > > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java, line > > 138 > > <https://reviews.apache.org/r/53204/diff/3/?file=1579544#file1579544line138> > > > > Looks like we won't handle NPE so NPE may cause some problems. > > > > If userMembershipAttr is null, will we still check userMememberOfGroup > > or not? If not, maybe we should handle such exception here. How about > > groupMembershipAttr above? Seems we will have such issue as w
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/ --- (Updated Dec. 8, 2016, 12:45 a.m.) Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Changes --- Sensitive information was moved to DEBUG log. Repository: hive-git Description --- HIVE-15076 Improve scalability of LDAP authentication provider group filter https://issues.apache.org/jira/browse/HIVE-15076 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 152c4b2 service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java e9172d3 service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java cd62935 service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java 4fad755 service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java acde8c1 service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 0cc2ead service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 499b624 service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 3054e33 service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION Diff: https://reviews.apache.org/r/53204/diff/ Testing --- Build succeeded. Test results: Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:14 min [INFO] Finished at: 2016-10-26T13:53:15-07:00 [INFO] Final Memory: 36M/1091M [INFO] Thanks, Illya Yalovyy
Re: Review Request 54393: HIVE-15361: INSERT dynamic partition on S3 fails with a MoveTask failure
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54393/#review158221 --- Ship it! Ship It! - Illya Yalovyy On Dec. 6, 2016, 8:13 p.m., Sergio Pena wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54393/ > --- > > (Updated Dec. 6, 2016, 8:13 p.m.) > > > Review request for hive. > > > Bugs: HIVE-15361 > https://issues.apache.org/jira/browse/HIVE-15361 > > > Repository: hive-git > > > Description > --- > > Problem: > - DynamicPartitionCtx and ListBucketingCtx objects weren't set on the new > MoveWork created when merging the two MoveWork objects from the > ConditionalTask. > > Solution > - Set the DynamicPartitionCtx and ListBucketingCtx objects to the new > MoveWork created for the S3 optimization. > > Other changes > - Merge the MoveWork objects inside the createCondTask() method for better > error handling. > - Only merge the MoveWork related to the moveOnlyMoveTask. The MoveWork from > the mergeAndMoveMoveTask may cause other issues that are not correctly tested. > - Two new private methods are added to check and merge the conditional > input/output tasks to the linked MoveWork. > > > Diffs > - > > > itests/hive-blobstore/src/test/queries/clientpositive/insert_into_dynamic_partitions.q > PRE-CREATION > itests/hive-blobstore/src/test/queries/clientpositive/insert_into_table.q > 25e2e7007ff539223d9244ca9822aa65d1441eb0 > > itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions.q > PRE-CREATION > > itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_table.q > 846b2b113f09a74a3f05c13ffb56163e81dc1e8e > > itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out > PRE-CREATION > > itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out > fbb52c132a331aefe870264e035c397078f3c82e > > itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out > 9f575a66ecefc3933b16dff554bdcc1c1f6420ee > > itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out > PRE-CREATION > > itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out > c725c96cbb6b0374e67308a54204c7c25e827567 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java > adc1188f09c8019a8aa60403d5813d6fa4509ceb > ql/src/java/org/apache/hadoop/hive/ql/plan/LoadDesc.java > bcd3125ab4ad20c00fec565e5004ee200c0187d5 > ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java > 9a868a04ce93d5c2ee75b5c6e96a1401cea93133 > ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java > 771a919ccd0bd75fe6197299ae057647ece89a7e > ql/src/java/org/apache/hadoop/hive/ql/plan/MoveWork.java > 9f498c7fb88a7a9f77b8c6739c097a2b26b0c617 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java > e6ec44504685bd9e53f158cc359b8a7b79fd0166 > > Diff: https://reviews.apache.org/r/54393/diff/ > > > Testing > --- > > All itests/hive-blobstore tests run. > > Added new blobstore tests: > - insert_into_dynamic_partitions.q > - insert_overwrite_dynamic_partitions.q > > Waiting for HiveQA to run the rest of the q-tests. > > > Thanks, > > Sergio Pena > >
Re: Review Request 54393: HIVE-15361: INSERT dynamic partition on S3 fails with a MoveTask failure
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54393/#review158069 --- itests/hive-blobstore/src/test/queries/clientpositive/insert_into_dynamic_partitions.q (lines 15 - 16) <https://reviews.apache.org/r/54393/#comment228742> Why explicit ADD PARTITION statement is required? I think insert into will create missing partitions. itests/hive-blobstore/src/test/queries/clientpositive/insert_into_dynamic_partitions.q (line 18) <https://reviews.apache.org/r/54393/#comment228741> Style: indentation is not required. itests/hive-blobstore/src/test/queries/clientpositive/insert_into_dynamic_partitions.q (line 24) <https://reviews.apache.org/r/54393/#comment228744> Style: table -> TABLE. It is better to keep code style consistent across all tests. itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions.q (line 25) <https://reviews.apache.org/r/54393/#comment228748> Will "SHOW PARTITIONS" be a good validation in this case? itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions.q (line 27) <https://reviews.apache.org/r/54393/#comment228745> Style: table -> TABLE ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (line 1636) <https://reviews.apache.org/r/54393/#comment228753> Terminology: BlobStorage -> BlobStore ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (line 1640) <https://reviews.apache.org/r/54393/#comment228750> Syntax: tha -> that ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (line 1644) <https://reviews.apache.org/r/54393/#comment228751> Could this method be made protected and covered with unit tests? ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (line 1647) <https://reviews.apache.org/r/54393/#comment228754> Should we take into account HIVE_BLOBSTORE_USE_BLOBSTORE_AS_SCRATCHDIR? ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (line 1668) <https://reviews.apache.org/r/54393/#comment228756> Just note: s3 to s3 copy is *more* efficient than hdfs to s3 copy. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (line 1686) <https://reviews.apache.org/r/54393/#comment228761> It seems to be an impossible outcome. Can we throw an exception instead of returning null? ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (lines 1769 - 1778) <https://reviews.apache.org/r/54393/#comment228766> I feel like this logic should be inside "addDependentMoveTasks" method. ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java (line 45) <https://reviews.apache.org/r/54393/#comment228767> you don't have to call getters here. ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java (line 52) <https://reviews.apache.org/r/54393/#comment228768> Don't have to call getters here. Could you also add a test with a strict dynamic partitioning, like: INSERT OVERWRITE TABLE t2 PARTITION(p1="1", p2) SELECT \*, c1 AS p2 FROM t1; - Illya Yalovyy On Dec. 5, 2016, 9:56 p.m., Sergio Pena wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54393/ > --- > > (Updated Dec. 5, 2016, 9:56 p.m.) > > > Review request for hive. > > > Bugs: HIVE-15361 > https://issues.apache.org/jira/browse/HIVE-15361 > > > Repository: hive-git > > > Description > --- > > Problem: > - DynamicPartitionCtx and ListBucketingCtx objects weren't set on the new > MoveWork created when merging the two MoveWork objects from the > ConditionalTask. > > Solution > - Set the DynamicPartitionCtx and ListBucketingCtx objects to the new > MoveWork created for the S3 optimization. > > Other changes > - Merge the MoveWork objects inside the createCondTask() method for better > error handling. > - Only merge the MoveWork related to the moveOnlyMoveTask. The MoveWork from > the mergeAndMoveMoveTask may cause other issues that are not correctly tested. > - Two new private methods are added to check and merge the conditional > input/output tasks to the linked MoveWork. > > > Diffs > - > > > itests/hive-blobstore/src/test/queries/clientpositive/insert_into_dynamic_partitions.q > PRE-CREATION > itests/hive-blobstore/src/test/queries/clientpositive/insert_into_table.q > 25e2e7007ff539223d9244ca9822aa65d1441eb0 > > itests/hi
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/#review157121 --- service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java (line 90) <https://reviews.apache.org/r/53204/#comment227538> This is a DEBUG level logging. It will not happen in production environment. The purpose of this log message is to help troubleshoot problems. Without name and group it makes little sense. I would prefer to keep it as is. service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java (line 124) <https://reviews.apache.org/r/53204/#comment227548> I will remove sensitive information from all error messages and exceptions. Thank you for suggesting it. I will move them to a debug logs instead. It will keep security and maintainability at high level. service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java (line 138) <https://reviews.apache.org/r/53204/#comment227540> This is a DEBUG level logging. It will not happen in production environment. The purpose of this log message is to help troubleshoot problems. Without name and group it makes little sense. I would prefer to keep it as is. service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java (line 261) <https://reviews.apache.org/r/53204/#comment227542> I don't think it is the best practice to add javadoc to unit test methods. Usually meaningful method names are used instead. If you look at hive code base, it is not very common as well. Only few unit tests have javadoc, some of them already obsolete and misleading. I would prefer keep it as is. service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java (line 32) <https://reviews.apache.org/r/53204/#comment227545> Hamcrest module is a part of JUnit. This module is designed to be used in conjunction with junit/mockito. I see no reason to break this best practice. - Illya Yalovyy On Oct. 28, 2016, 12:59 p.m., Illya Yalovyy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53204/ > --- > > (Updated Oct. 28, 2016, 12:59 p.m.) > > > Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon > Ho. > > > Repository: hive-git > > > Description > --- > > HIVE-15076 Improve scalability of LDAP authentication provider group filter > > https://issues.apache.org/jira/browse/HIVE-15076 > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 > service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java > 152c4b2 > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea > service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java > e9172d3 > > service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java > cd62935 > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java > 4fad755 > > service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java > acde8c1 > service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java > 0cc2ead > service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java > 499b624 > service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java > 3054e33 > service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION > service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION > > Diff: https://reviews.apache.org/r/53204/diff/ > > > Testing > --- > > Build succeeded. > > Test results: > > Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > -------- > [INFO] Total time: 03:14 min > [INFO] Finished at: 2016-10-26T13:53:15-07:00 > [INFO] Final Memory: 36M/1091M > [INFO] > > > > Thanks, > > Illya Yalovyy > >
Re: Review Request 53966: HIVE-15199: INSERT INTO data on S3 is replacing the old rows with the new ones
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53966/#review156738 --- The patch looks good to me. We probably want to add a integration test for it in hive-blobstore package. - Illya Yalovyy On Nov. 22, 2016, 10:35 p.m., Sergio Pena wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53966/ > --- > > (Updated Nov. 22, 2016, 10:35 p.m.) > > > Review request for hive. > > > Bugs: HIVE-15199 > https://issues.apache.org/jira/browse/HIVE-15199 > > > Repository: hive-git > > > Description > --- > > The patch helps execute repeated INSERT INTO statements on S3 tables when the > scratch directory is on S3. > > > Diffs > - > > itests/hive-blobstore/src/test/queries/clientpositive/insert_into.q > 919ff7d9c7cb40062d68b876d6acbc8efb8a8cf1 > itests/hive-blobstore/src/test/results/clientpositive/insert_into.q.out > c25d0c4eec6983b6869e2eba711b39ba91a4c6e0 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > 61b8bd0ac40cffcd6dca0fc874940066bc0aeffe > > Diff: https://reviews.apache.org/r/53966/diff/ > > > Testing > --- > > > Thanks, > > Sergio Pena > >
Re: Review Request 53966: HIVE-15199: INSERT INTO data on S3 is replacing the old rows with the new ones
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53966/#review156629
---
common/src/java/org/apache/hadoop/hive/common/FileUtils.java (line 1007)
<https://reviews.apache.org/r/53966/#comment226819>
Can we use an explicit type as return type? Something like
final class NameAndType {
final String name;
final String type;
}
common/src/java/org/apache/hadoop/hive/common/FileUtils.java (lines 1010 - 1019)
<https://reviews.apache.org/r/53966/#comment226820>
return new NameAndType(FilenameUtils.getBaseName(filename),
FilenameUtils.getExtension(filename));
https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/FilenameUtils.html
common/src/java/org/apache/hadoop/hive/common/FileUtils.java (line 1015)
<https://reviews.apache.org/r/53966/#comment226821>
extra ";" is not needed.
common/src/java/org/apache/hadoop/hive/common/FileUtils.java (line 1019)
<https://reviews.apache.org/r/53966/#comment226835>
This utility method should be covered with unit tests. Please make sure you
have covered cases like:
s3://mybucket.test/foo/bar/0_0
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2786)
<https://reviews.apache.org/r/53966/#comment226822>
Scalability concern:
On some real datasets, it could be millions of elements in that list. If it
happens in HS2 with many cocurrent connection this jvm can easily go down with
OOM Exceptions. I would suggest reconsider that approach.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2921)
<https://reviews.apache.org/r/53966/#comment226828>
is "copy" part of the file name misleading? It is not actually a copy of an
original file.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2923)
<https://reviews.apache.org/r/53966/#comment226825>
Just a note:
filename + "." + filetype is 10x faster than String.format("%s%s",
filename, filetype).
Also it seems like "." is missing.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2928)
<https://reviews.apache.org/r/53966/#comment226826>
"." is missing between name and type.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 2943)
<https://reviews.apache.org/r/53966/#comment226830>
FilenameUtils can do the job:
https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/FilenameUtils.html
- Illya Yalovyy
On Nov. 21, 2016, 11:54 p.m., Sergio Pena wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53966/
> ---
>
> (Updated Nov. 21, 2016, 11:54 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-15199
> https://issues.apache.org/jira/browse/HIVE-15199
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> The patch helps execute repeated INSERT INTO statements on S3 tables when the
> scratch directory is on S3.
>
>
> Diffs
> -
>
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java
> 1d8c04160c35e48781b20f8e6e14760c19df9ca5
> itests/hive-blobstore/src/test/queries/clientpositive/insert_into.q
> 919ff7d9c7cb40062d68b876d6acbc8efb8a8cf1
> itests/hive-blobstore/src/test/results/clientpositive/insert_into.q.out
> c25d0c4eec6983b6869e2eba711b39ba91a4c6e0
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> 61b8bd0ac40cffcd6dca0fc874940066bc0aeffe
>
> Diff: https://reviews.apache.org/r/53966/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Sergio Pena
>
>
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/ --- (Updated Oct. 28, 2016, 12:59 p.m.) Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Changes --- 1. Removed trailing spaces. Repository: hive-git Description --- HIVE-15076 Improve scalability of LDAP authentication provider group filter https://issues.apache.org/jira/browse/HIVE-15076 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 152c4b2 service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java e9172d3 service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java cd62935 service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java 4fad755 service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java acde8c1 service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 0cc2ead service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 499b624 service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 3054e33 service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION Diff: https://reviews.apache.org/r/53204/diff/ Testing --- Build succeeded. Test results: Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:14 min [INFO] Finished at: 2016-10-26T13:53:15-07:00 [INFO] Final Memory: 36M/1091M [INFO] Thanks, Illya Yalovyy
Re: Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/ --- (Updated Oct. 27, 2016, 1:45 p.m.) Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Repository: hive-git Description --- HIVE-15076 Improve scalability of LDAP authentication provider group filter https://issues.apache.org/jira/browse/HIVE-15076 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 152c4b2 service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java e9172d3 service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java cd62935 service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java 4fad755 service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java acde8c1 service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 0cc2ead service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 499b624 service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 3054e33 service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION Diff: https://reviews.apache.org/r/53204/diff/ Testing (updated) --- Build succeeded. Test results: Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:14 min [INFO] Finished at: 2016-10-26T13:53:15-07:00 [INFO] Final Memory: 36M/1091M [INFO] Thanks, Illya Yalovyy
Review Request 53204: HIVE-15076 Improve scalability of LDAP authentication provider group filter
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53204/ --- Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Repository: hive-git Description --- HIVE-15076 Improve scalability of LDAP authentication provider group filter https://issues.apache.org/jira/browse/HIVE-15076 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5ea9751 service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java 33b6088 service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java 152c4b2 service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java 65076ea service/src/java/org/apache/hive/service/auth/ldap/Query.java b8bf938 service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java e9172d3 service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java cd62935 service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java 4fad755 service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java acde8c1 service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java 0cc2ead service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java 499b624 service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java 3054e33 service/src/test/resources/ldap/ad.example.com.ldif PRE-CREATION service/src/test/resources/ldap/microsoft.schema.ldif PRE-CREATION Diff: https://reviews.apache.org/r/53204/diff/ Testing --- Tests run: 149, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:14 min [INFO] Finished at: 2016-10-26T13:53:15-07:00 [INFO] Final Memory: 36M/1091M [INFO] Thanks, Illya Yalovyy
[jira] [Created] (HIVE-15076) Improve scalability of LDAP authentication provider group filter
Illya Yalovyy created HIVE-15076: Summary: Improve scalability of LDAP authentication provider group filter Key: HIVE-15076 URL: https://issues.apache.org/jira/browse/HIVE-15076 Project: Hive Issue Type: Improvement Components: Authentication Affects Versions: 2.1.0 Reporter: Illya Yalovyy Assignee: Illya Yalovyy Current implementation uses following algorithm: # For a given user find all groups that user is a member of. (A list of LDAP groups is constructed as a result of that request) # Match this list of groups with provided group filter. Time/Memory complexity of this approach is O(N) on client side, where N – is a number of groups the user has membership in. On a large directory (800+ groups per user) we can observe up to 2x performance degradation and failures because of size of LDAP response (LDAP: error code 4 - Sizelimit Exceeded). Some Directory Services (Microsoft Active Directory for instance) provide a virtual attribute for User Object that contains a list of groups that user belongs to. This attribute can be used to quickly determine whether this user passes or fails the group filter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 52743: HIVE-14927 Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52743/ --- Review request for hive, Aihua Xu, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Repository: hive-git Description --- jira https://issues.apache.org/jira/browse/HIVE-14927 * Extract inner class User and implement a proper builder for it. * Extract all common code to LdapAuthenticationTestCase class ** setting up the test case ** executing test case ** result validation Diffs - service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java 8c723cf service/src/test/org/apache/hive/service/auth/ldap/LdapAuthenticationTestCase.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/User.java PRE-CREATION Diff: https://reviews.apache.org/r/52743/diff/ Testing --- Tests run: 126, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 04:02 min [INFO] Finished at: 2016-10-06T12:23:16-07:00 [INFO] Final Memory: 42M/1146M [INFO] Thanks, Illya Yalovyy
[jira] [Created] (HIVE-14927) Remove code duplication from tests in TestLdapAtnProviderWithMiniDS
Illya Yalovyy created HIVE-14927: Summary: Remove code duplication from tests in TestLdapAtnProviderWithMiniDS Key: HIVE-14927 URL: https://issues.apache.org/jira/browse/HIVE-14927 Project: Hive Issue Type: Improvement Components: Test Reporter: Illya Yalovyy Assignee: Illya Yalovyy * Extract inner class User and implement a proper builder for it. * Extract all common code to LdapAuthenticationTestCase class * setting up the test case * executing test case * result validation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 52487: HIVE-14875 Enhancement and refactoring of TestLdapAtnProviderWithMiniDS
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52487/ --- (Updated Oct. 3, 2016, 5:40 p.m.) Review request for hive, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Repository: hive-git Description (updated) --- This makes the following enhancements to defining the ldifs to apply to the in-memory LDAP server: * Extract defined ldifs to a resource file * Remove unneeded attributes defined in each ldif entry such as: * sn (Surname) and givenName from group entries * distinguishedName from all entries as this attribute serves more as a parent type of many other attributes. * Remove setting ExtensibleObject as an objectClass for all ldap entries as that is not needed. This objectClass would allow for adding any attribute to an entry. * Add missing uid attribute to group entries whose dn refer to a uid attribute * Add missing uidObject objectClass to entries that have the uid attribute * Explicitly set organizationalPerson objectClass to user entries as they are using inetOrgPerson objectClass which is a subclass of the organizationalPerson objectClass * Create indexes on cn and uid attributes as they are commonly queried. * Removed unused variables and imports. * Fixed givenName for user3. * Other minor code clean up. https://issues.apache.org/jira/browse/HIVE-14875 Diffs - service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java 23a048a service/src/test/resources/ldap/example.com.ldif PRE-CREATION Diff: https://reviews.apache.org/r/52487/diff/ Testing --- Tests run: 126, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 04:00 min [INFO] Finished at: 2016-09-29T21:20:04-07:00 [INFO] Final Memory: 40M/1000M [INFO] Thanks, Illya Yalovyy
Review Request 52487: HIVE-14875 Enhancement and refactoring of TestLdapAtnProviderWithMiniDS
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52487/ --- Review request for hive, Ashutosh Chauhan, Chaoyu Tang, and Szehon Ho. Repository: hive-git Description --- This makes the following enhancements to defining the ldifs to apply to the in-memory LDAP server: * Extract defined ldifs to a resource file * Remove unneeded attributes defined in each ldif entry such as: * sn (Surname) and givenName from group entries * distinguishedName from all entries as this attribute serves more as a parent type of many other attributes. * Remove setting ExtensibleObject as an objectClass for all ldap entries as that is not needed. This objectClass would allow for adding any attribute to an entry. * Add missing uid attribute to group entries whose dn refer to a uid attribute * Add missing uidObject objectClass to entries that have the uid attribute * Explicitly set organizationalPerson objectClass to user entries as they are using inetOrgPerson objectClass which is a subclass of the organizationalPerson objectClass * Create indexes on cn and uid attributes as they are commonly queried. * Removed unused variables and imports. * Fixed givenName for user3. * Other minor code clean up. Diffs - service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java 23a048a service/src/test/resources/ldap/example.com.ldif PRE-CREATION Diff: https://reviews.apache.org/r/52487/diff/ Testing --- Tests run: 126, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 04:00 min [INFO] Finished at: 2016-09-29T21:20:04-07:00 [INFO] Final Memory: 40M/1000M [INFO] Thanks, Illya Yalovyy
[jira] [Created] (HIVE-14875) Enhancement and refactoring of TestLdapAtnProviderWithMiniDS
Illya Yalovyy created HIVE-14875: Summary: Enhancement and refactoring of TestLdapAtnProviderWithMiniDS Key: HIVE-14875 URL: https://issues.apache.org/jira/browse/HIVE-14875 Project: Hive Issue Type: Test Components: Authentication, Tests Reporter: Illya Yalovyy Assignee: Illya Yalovyy This makes the following enhancements to TestLdapAtnProviderWithMiniDS: * Extract defined ldifs to a resource file. * Remove unneeded attributes defined in each ldif entry such as: * sn (Surname) and givenName from group entries * distinguishedName from all entries as this attribute serves more as a parent type of many other attributes. * Remove setting ExtensibleObject as an objectClass for all ldap entries as that is not needed. This objectClass would allow for adding any attribute to an entry. * Add missing uid attribute to group entries whose dn refer to a uid attribute * Add missing uidObject objectClass to entries that have the uid attribute * Explicitly set organizationalPerson objectClass to user entries as they are using inetOrgPerson objectClass which is a subclass of the organizationalPerson objectClass * Create indexes on cn and uid attributes as they are commonly queried. * Removed unused variables and imports. * Fixed givenName for user3. * Other minor code clean up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51694/ --- (Updated Sept. 22, 2016, 5:46 p.m.) Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and Szehon Ho. Changes --- Use "limit 2" for LDAP search queries that are expected to return a single result. Repository: hive-git Description --- Currently LdapAuthenticationProviderImpl class is not covered with unit tests. To make this class testable some minor refactoring will be required. Diffs (updated) - service/pom.xml ecea719 service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java efd5393 service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java PRE-CREATION service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java 089a059 service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java f276906 service/src/test/org/apache/hive/service/auth/ldap/Credentials.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/LdapTestUtils.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestChainFilter.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestCustomQueryFilter.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestLdapUtils.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestQuery.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestSearchResultHandler.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestUserFilter.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestUserSearchFilter.java PRE-CREATION Diff: https://reviews.apache.org/r/51694/diff/ Testing --- ...hive/service> mvn clean test ... Results : Tests run: 123, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 04:18 min [INFO] Finished at: 2016-09-06T08:46:04-07:00 [INFO] Final Memory: 66M/984M [INFO] ---- Thanks, Illya Yalovyy
Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests
> On Sept. 21, 2016, 12:50 a.m., Szehon Ho wrote: > > This looks like a great refactoring to me. This class was always hard to > > understand, and this makes it a little easier. I'll defer to Chaoyu to the > > comments. Thank you! - Illya --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51694/#review149769 --- On Sept. 20, 2016, 7:39 p.m., Illya Yalovyy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51694/ > --- > > (Updated Sept. 20, 2016, 7:39 p.m.) > > > Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and > Szehon Ho. > > > Repository: hive-git > > > Description > --- > > Currently LdapAuthenticationProviderImpl class is not covered with unit > tests. To make this class testable some minor refactoring will be required. > > > Diffs > - > > service/pom.xml ecea719 > > service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java > efd5393 > service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java > PRE-CREATION > > service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java > PRE-CREATION > > service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java > 089a059 > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java > f276906 > service/src/test/org/apache/hive/service/auth/ldap/Credentials.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/LdapTestUtils.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestChainFilter.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestCustomQueryFilter.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestLdapUtils.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestQuery.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestSearchResultHandler.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestUserFilter.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestUserSearchFilter.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/51694/diff/ > > > Testing > --- > > ...hive/service> mvn clean test > > ... > > Results : > > Tests run: 123, Failures: 0, Errors: 0, Skipped: 0 > > [INFO] > -------- > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 04:18 min > [INFO] Finished at: 2016-09-06T08:46:04-07:00 > [INFO] Final Memory: 66M/984M > [INFO] > > > > Thanks, > > Illya Yalovyy > >
Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests
> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote: > > service/src/java/org/apache/hive/service/auth/ldap/Query.java, line 122 > > <https://reviews.apache.org/r/51694/diff/1/?file=1492949#file1492949line122> > > > > Will it improve the performance to set the search limit? I did not see > > it is used. > > Illya Yalovyy wrote: > I will be used for different filters. Do you think we should use it for > existing filters? Which one in particular? Or you would prefer me to remove > this option? > > Please keep in mind that this CR is not about performance. > > Chaoyu Tang wrote: > I thought in the existing implementation, the search limits in some > methods like findGroupDNByName, findUserDNByName are set to 2 to reduce the > returned results in case there are many, is not it? It does make sense. I'll make this change. - Illya --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51694/#review148634 ------- On Sept. 20, 2016, 7:39 p.m., Illya Yalovyy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51694/ > --- > > (Updated Sept. 20, 2016, 7:39 p.m.) > > > Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and > Szehon Ho. > > > Repository: hive-git > > > Description > --- > > Currently LdapAuthenticationProviderImpl class is not covered with unit > tests. To make this class testable some minor refactoring will be required. > > > Diffs > - > > service/pom.xml ecea719 > > service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java > efd5393 > service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java > PRE-CREATION > > service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java > PRE-CREATION > > service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java > 089a059 > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java > f276906 > service/src/test/org/apache/hive/service/auth/ldap/Credentials.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/LdapTestUtils.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestChainFilter.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestCustomQueryFilter.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestLdapUtils.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestQuery.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestSearchResultHandler.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestUserFilter.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestU
Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests
> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java,
> > line 37
> > <https://reviews.apache.org/r/51694/diff/1/?file=1492945#file1492945line37>
> >
> > Do we really need an extra factory layer and have a factory for each
> > filter?
> > In Hive, actaully each session instantiates its own
> > LdapAuthenticationProviderImpl, which now contains different factories with
> > each one generating only one instance of its filter.
I think so. Factories encapsulate logic related to choosing whether particular
filter is required or not based on the provided configuration.
> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java, line 108
> > <https://reviews.apache.org/r/51694/diff/1/?file=1492946#file1492946line108>
> >
> > can getSingleLdapName be used to enforce only one returned entry? that
> > API in SearachResultHandler is never used.
I was trying to copy existing logic. At the moment, I don't want to do any
changes to that. This particular code should be improved in separate CR. Does
it make sense?
> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java, line 105
> > <https://reviews.apache.org/r/51694/diff/1/?file=1492948#file1492948line105>
> >
> > This method might throw out runtime exception such as NPE,
> > IndexOutOfBoundsException, should we check the passed in parameter rdn?
> > We might not run into this situation in old code, but since this line
> > of code is refactored as a separate API, I think we should do the check.
> > Same for the other methods like patternToBaseDn etc.
Agree. The DN parsing in general implemented quite poorly. I have a task
already to re-implement it completely. There are many problems with current
one. Handling incorrect format or invalid input is only one of them. My
intention is to use RDN java class to do a correct parsing.
https://docs.oracle.com/javase/7/docs/api/javax/naming/ldap/Rdn.html
I think we can leave it for now, and I will submit another CR that addresses
this concern sortly.
> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java, line 159
> > <https://reviews.apache.org/r/51694/diff/1/?file=1492948#file1492948line159>
> >
> > I am not sure if there is any precedence for these configurations, but
> > here it seems that the GUIDKEY/BASEDN takes precedence over DNPATTERN,
> > which is different from the existing implementation and cause the behavior
> > change.
Could you please give more details on the case when the behavior will be
different.
The logic seems to be same: It uses GUIDKEY/BASEDN only when DNPATTERN is not
configured:
if (StringUtils.isBlank(patternsString)) {
...
Which means *only* if patternsString is blank, try to use GUIDKEY/BASEDN.
Please let me know if I did not get it correctly.
> On Sept. 17, 2016, 1:36 a.m., Chaoyu Tang wrote:
> > service/src/java/org/apache/hive/service/auth/ldap/Query.java, line 122
> > <https://reviews.apache.org/r/51694/diff/1/?file=1492949#file1492949line122>
> >
> > Will it improve the performance to set the search limit? I did not see
> > it is used.
I will be used for different filters. Do you think we should use it for
existing filters? Which one in particular? Or you would prefer me to remove
this option?
Please keep in mind that this CR is not about performance.
- Illya
-----------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51694/#review148634
---
On Sept. 7, 2016, 2:24 p.m., Illya Yalovyy wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51694/
> ---
>
> (Updated Sept. 7, 2016, 2:24 p.m.)
>
>
> Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and
> Szehon Ho.
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> Currently LdapAuthenticationProviderImpl class is not covered with unit
> tests. To make this class testable some minor refactoring will be required.
>
>
> Diffs
> -
>
> service/pom.xml ecea719
>
> service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java
> efd5393
> service/src/java/org/apache/hive/service/auth/ldap/ChainFilt
Re: Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51694/#review149493 --- Chaoyu, Thank you for a great review. I have also done some minor changes on my side. I'll update this CR in a couple of days. - Illya Yalovyy On Sept. 7, 2016, 2:24 p.m., Illya Yalovyy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51694/ > --- > > (Updated Sept. 7, 2016, 2:24 p.m.) > > > Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and > Szehon Ho. > > > Repository: hive-git > > > Description > --- > > Currently LdapAuthenticationProviderImpl class is not covered with unit > tests. To make this class testable some minor refactoring will be required. > > > Diffs > - > > service/pom.xml ecea719 > > service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java > efd5393 > service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java > PRE-CREATION > > service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java > PRE-CREATION > service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java > PRE-CREATION > > service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java > 089a059 > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java > f276906 > service/src/test/org/apache/hive/service/auth/ldap/Credentials.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/LdapTestUtils.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestChainFilter.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestCustomQueryFilter.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestLdapUtils.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestQuery.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestSearchResultHandler.java > PRE-CREATION > service/src/test/org/apache/hive/service/auth/ldap/TestUserFilter.java > PRE-CREATION > > service/src/test/org/apache/hive/service/auth/ldap/TestUserSearchFilter.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/51694/diff/ > > > Testing > --- > > ...hive/service> mvn clean test > > ... > > Results : > > Tests run: 123, Failures: 0, Errors: 0, Skipped: 0 > > [INFO] > ---- > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 04:18 min > [INFO] Finished at: 2016-09-06T08:46:04-07:00 > [INFO] Final Memory: 66M/984M > [INFO] > > > > Thanks, > > Illya Yalovyy > >
Review Request 51694: HIVE-14713 LDAP Authentication Provider should be covered with unit tests
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51694/ --- Review request for hive, Ashutosh Chauhan, Chaoyu Tang, Naveen Gangam, and Szehon Ho. Repository: hive-git Description --- Currently LdapAuthenticationProviderImpl class is not covered with unit tests. To make this class testable some minor refactoring will be required. Diffs - service/pom.xml ecea719 service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java efd5393 service/src/java/org/apache/hive/service/auth/ldap/ChainFilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/CustomQueryFilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/DirSearch.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/DirSearchFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/Filter.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/FilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/GroupFilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/LdapSearch.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/LdapSearchFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/LdapUtils.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/Query.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/QueryFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/SearchResultHandler.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/UserFilterFactory.java PRE-CREATION service/src/java/org/apache/hive/service/auth/ldap/UserSearchFilterFactory.java PRE-CREATION service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java 089a059 service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java f276906 service/src/test/org/apache/hive/service/auth/ldap/Credentials.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/LdapTestUtils.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestChainFilter.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestCustomQueryFilter.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestGroupFilter.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestLdapSearch.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestLdapUtils.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestQuery.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestQueryFactory.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestSearchResultHandler.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestUserFilter.java PRE-CREATION service/src/test/org/apache/hive/service/auth/ldap/TestUserSearchFilter.java PRE-CREATION Diff: https://reviews.apache.org/r/51694/diff/ Testing --- ...hive/service> mvn clean test ... Results : Tests run: 123, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 04:18 min [INFO] Finished at: 2016-09-06T08:46:04-07:00 [INFO] Final Memory: 66M/984M [INFO] Thanks, Illya Yalovyy
[jira] [Created] (HIVE-14713) LDAP Authentication Provider should be covered with unit tests
Illya Yalovyy created HIVE-14713: Summary: LDAP Authentication Provider should be covered with unit tests Key: HIVE-14713 URL: https://issues.apache.org/jira/browse/HIVE-14713 Project: Hive Issue Type: Test Components: Authentication, Tests Affects Versions: 2.1.0 Reporter: Illya Yalovyy Assignee: Illya Yalovyy Currently LdapAuthenticationProviderImpl class is not covered with unit tests. To make this class testable some minor refactoring will be required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 51037: HIVE-14373 Add integration tests for hive on S3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51037/ --- Review request for hive, Ashutosh Chauhan, Abdullah Yousufi, and Sergio Pena. Repository: hive-git Description --- Alternative implementation of an integration test framework. It can be used to test Hive against S3. Diffs - data/conf/s3/hive-site.xml PRE-CREATION data/conf/s3/log4j2.properties PRE-CREATION itests/pom.xml 426ba04 itests/qtest-s3/README PRE-CREATION itests/qtest-s3/pom.xml PRE-CREATION itests/qtest-s3/src/test/data/scripts/s3_test_cleanup.q PRE-CREATION itests/qtest-s3/src/test/data/scripts/s3_test_init.q PRE-CREATION itests/qtest-s3/src/test/expected-results/s3positive/insert_into_s3.q.out PRE-CREATION itests/qtest-s3/src/test/java/org/apache/hadoop/hive/cli/Masker.java PRE-CREATION itests/qtest-s3/src/test/java/org/apache/hadoop/hive/cli/QTestUtilBuilder.java PRE-CREATION itests/qtest-s3/src/test/java/org/apache/hadoop/hive/cli/S3TestUtil.java PRE-CREATION itests/qtest-s3/src/test/java/org/apache/hadoop/hive/cli/TestMasker.java PRE-CREATION itests/qtest-s3/src/test/queries/s3positive/insert_into_s3.q PRE-CREATION itests/qtest-s3/src/test/resources/testconfiguration.properties PRE-CREATION itests/qtest-s3/src/test/templates/TestCliDriverS3.vm PRE-CREATION Diff: https://reviews.apache.org/r/51037/diff/ Testing --- It can be buld and executed with the latest hive master branch. Thanks, Illya Yalovyy
Review Request 50816: HIVE-7239 Fix bug in HiveIndexedInputFormat implementation that causes incorrect query result when input backed by Sequence/RC files
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50816/ --- Review request for hive, Ashutosh Chauhan and Gopal V. Repository: hive-git Description --- HIVE-7239 Fix bug in HiveIndexedInputFormat implementation that causes incorrect query result when input backed by Sequence/RC files In case of sequence files, it's crucial that splits are calculated around the boundaries enforced by the input sequence file. However by default hadoop creates input splits depending on the configuration parameters which may not match the boundaries for the input sequence file. Hive provides HiveIndexedInputFormat that provides extra logic and recalculates the split boundaries for each split depending on the sequence file's boundaries. However we noticed this behavior of "over" reporting from data backed by sequence file. We've a sample data on which we experimented and fixed this bug, we have verified this fix by comparing the query output for input being sequence file format, rc file and regular format. https://issues.apache.org/jira/browse/HIVE-7239 Diffs - ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java 33cc5c3 ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java 5247ece ql/src/java/org/apache/hadoop/hive/ql/index/IndexResult.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/SplitFilter.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/index/MockHiveInputSplits.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/index/MockIndexResult.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/index/MockInputFile.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/index/SplitFilterTestCase.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/index/TestHiveInputSplitComparator.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/index/TestSplitFilter.java PRE-CREATION Diff: https://reviews.apache.org/r/50816/diff/ Testing --- Manually tested on a cluster. HiveQA: Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/674/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/674/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-674/ Thanks, Illya Yalovyy
[jira] [Created] (HIVE-13510) Dynamic partitioning doesn’t work when remote metastore is used
Illya Yalovyy created HIVE-13510:
Summary: Dynamic partitioning doesn’t work when remote metastore
is used
Key: HIVE-13510
URL: https://issues.apache.org/jira/browse/HIVE-13510
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 2.1.0
Environment: Hadoop 2.7.1
Reporter: Illya Yalovyy
Assignee: Illya Yalovyy
Priority: Critical
*Steps to reproduce:*
# Configure remote metastore (hive.metastore.uris)
# Create table t1 (a string);
# Create table t2 (a string) partitioned by (b string);
# set hive.exec.dynamic.partition.mode=nonstrict;
# Insert overwrite table t2 partition (b) select a,a from t1;
*Result:*
{noformat}
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
16/04/13 15:04:51 [c679e424-2501-4347-8146-cf1b1cae217c main]: ERROR ql.Driver:
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
at
org.apache.hadoop.hive.ql.plan.DynamicPartitionCtx.(DynamicPartitionCtx.java:84)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6550)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9315)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9204)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10071)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9949)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10607)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:358)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10618)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:233)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:476)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:318)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1192)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1287)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1106)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:339)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:748)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getMetaConf(Hive.java:3493)
at
org.apache.hadoop.hive.ql.plan.DynamicPartitionCtx.(DynamicPartitionCtx.java:82)
... 29 more
Caused by: org.apache.thrift.TApplicationException: getMetaConf failed: unknown
result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_getMetaConf(ThriftHiveMetastore.java:666)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.getMetaConf(ThriftHiveMetastore.java:646)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getMetaConf(HiveMetaStoreClient.java:550)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606
[jira] [Created] (HIVE-13185) orc.ReaderImp.ensureOrcFooter() method fails on small text files with IndexOutOfBoundsException
Illya Yalovyy created HIVE-13185:
Summary: orc.ReaderImp.ensureOrcFooter() method fails on small
text files with IndexOutOfBoundsException
Key: HIVE-13185
URL: https://issues.apache.org/jira/browse/HIVE-13185
Project: Hive
Issue Type: Bug
Components: ORC
Affects Versions: 2.1.0
Reporter: Illya Yalovyy
Steps to reproduce:
1. Create a Text source table with one line of data:
{code}
create table src (id int);
insert overwrite table src values (1);
{code}
2. Create a target table:
{code}
create table trg (id int);
{code}
3. Try to load small text file to the target table:
{code}
load data inpath 'user/hive/warehouse/src/00_0' into table trg;
{code}
*Error message:*
{quote}
FAILED: SemanticException Unable to load data to destination table. Error:
java.lang.IndexOutOfBoundsException
{quote}
*Stack trace:*
{noformat}
org.apache.hadoop.hive.ql.parse.SemanticException: Unable to load data to
destination table. Error: java.lang.IndexOutOfBoundsException
at
org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.ensureFileFormatsMatch(LoadSemanticAnalyzer.java:340)
at
org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:224)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:242)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:481)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:317)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1190)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1285)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1104)
...
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12715) Unit test for HIVE-10685 fix
Illya Yalovyy created HIVE-12715: Summary: Unit test for HIVE-10685 fix Key: HIVE-12715 URL: https://issues.apache.org/jira/browse/HIVE-12715 Project: Hive Issue Type: Test Reporter: Illya Yalovyy It seems like bugfix provided for HIVE-10685 is not covered with tests. This tricky scenario could happen not only when table gets concatenated but also in some other use cases. I'm going to implement a unit test for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11882) Fetch optimizer should stop source files traversal once it exceeds the hive.fetch.task.conversion.threshold
Illya Yalovyy created HIVE-11882: Summary: Fetch optimizer should stop source files traversal once it exceeds the hive.fetch.task.conversion.threshold Key: HIVE-11882 URL: https://issues.apache.org/jira/browse/HIVE-11882 Project: Hive Issue Type: Improvement Components: Physical Optimizer Affects Versions: 1.0.0 Reporter: Illya Yalovyy Hive 1.0's fetch optimizer tries to optimize queries of the form "select from where limit " to a fetch task (see the hive.fetch.task.conversion property). This optimization gets the lengths of all the files in the specified partition and does some comparison against a threshold value to determine whether it should use a fetch task or not (see the hive.fetch.task.conversion.threshold property). This process of getting the length of all files. One of the main problems in this optimization is the fetch optimizer doesn't seem to stop once it exceeds the hive.fetch.task.conversion.threshold. It works fine on HDFS, but could cause a significant performance degradation on other supported file systems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 38268: HIVE-10980 Merge of dynamic partitions loads all data to default partition
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38268/ --- Review request for hive and Gopal V. Bugs: HIVE-10980 https://issues.apache.org/jira/browse/HIVE-10980 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-10980 Conditions that lead to the issue: 1. Execution engine set to MapReduce 2. Partition columns have different types 3. Both static and dynamic partitions are used in the query 4. Dynamically generated partitions require merge Result: Final data is loaded to "__HIVE_DEFAULT_PARTITION__". Steps to reproduce: set hive.exec.dynamic.partition=true; set hive.exec.dynamic.partition.mode=strict; set hive.optimize.sort.dynamic.partition=false; set hive.merge.mapfiles=true; set hive.merge.mapredfiles=true; set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; set hive.execution.engine=mr; create external table sdp ( dataint bigint, hour int, req string, cid string, caid string ) row format delimited fields terminated by ','; load data local inpath '../../data/files/dynpartdata1.txt' into table sdp; load data local inpath '../../data/files/dynpartdata2.txt' into table sdp; ... load data local inpath '../../data/files/dynpartdataN.txt' into table sdp; create table tdp (cid string, caid string) partitioned by (dataint bigint, hour int, req string); insert overwrite table tdp partition (dataint=20150316, hour=16, req) select cid, caid, req from sdp where dataint=20150316 and hour=16; select * from tdp order by caid; show partitions tdp; Example of the input file: 20150316,16,reqA,clusterIdA,cacheId1 20150316,16,reqB,clusterIdB,cacheId2 20150316,16,reqA,clusterIdC,cacheId3 20150316,16,reqD,clusterIdD,cacheId4 20150316,16,reqA,clusterIdA,cacheId5 Actual result: clusterIdA cacheId12015031616 __HIVE_DEFAULT_PARTITION__ clusterIdA cacheId12015031616 __HIVE_DEFAULT_PARTITION__ clusterIdB cacheId22015031616 __HIVE_DEFAULT_PARTITION__ clusterIdC cacheId32015031616 __HIVE_DEFAULT_PARTITION__ clusterIdD cacheId42015031616 __HIVE_DEFAULT_PARTITION__ clusterIdA cacheId52015031616 __HIVE_DEFAULT_PARTITION__ clusterIdD cacheId82015031616 __HIVE_DEFAULT_PARTITION__ clusterIdB cacheId92015031616 __HIVE_DEFAULT_PARTITION__ dataint=20150316/hour=16/req=__HIVE_DEFAULT_PARTITION__ Diffs - data/files/dynpartdata1.txt PRE-CREATION data/files/dynpartdata2.txt PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 4a325fb ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsUsePartitionColumnsNegative.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsUsePartitionColumnsPositive.java PRE-CREATION ql/src/test/queries/clientpositive/dynpart_merge.q PRE-CREATION ql/src/test/results/clientpositive/dynpart_merge.q.out PRE-CREATION ql/src/test/results/clientpositive/list_bucket_dml_6.q.java1.7.out d223234 ql/src/test/results/clientpositive/list_bucket_dml_6.q.java1.8.out f884ace ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 541944d Diff: https://reviews.apache.org/r/38268/diff/ Testing --- 1. Added new unit tests 2. Added qtest 3. Updated old qtests Thanks, Illya Yalovyy
[jira] [Created] (HIVE-11791) Add test for HIVE-10122
Illya Yalovyy created HIVE-11791: Summary: Add test for HIVE-10122 Key: HIVE-11791 URL: https://issues.apache.org/jira/browse/HIVE-11791 Project: Hive Issue Type: Test Components: Metastore Affects Versions: 1.1.0 Reporter: Illya Yalovyy Priority: Minor Unit tests for PartitionPruner.compactExpr() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11583) When PTF is used over a large partitions result could be corrupted
Illya Yalovyy created HIVE-11583:
Summary: When PTF is used over a large partitions result could be
corrupted
Key: HIVE-11583
URL: https://issues.apache.org/jira/browse/HIVE-11583
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Affects Versions: 1.2.1, 1.2.0, 1.0.0, 0.13.1, 0.14.0, 0.14.1
Environment: Hadoop 2.6 + Apache hive built from trunk
Reporter: Illya Yalovyy
Priority: Critical
Dataset:
Window has 50001 record (2 blocks on disk and 1 block in memory)
Size of the second block is 32Mb (2 splits)
Result:
When the last block is read from the disk only first split is actually loaded.
The second split gets missed. The total count of the result dataset is correct,
but some records are missing and another are duplicated.
Example:
{code:sql}
CREATE TABLE ptf_big_src (
id INT,
key STRING,
grp STRING,
value STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
LOAD DATA LOCAL INPATH '../../data/files/ptf_3blocks.txt.gz' OVERWRITE INTO
TABLE ptf_big_src;
SELECT grp, COUNT(1) cnt FROM ptf_big_trg GROUP BY grp ORDER BY cnt desc;
---
-- A25000
-- B2
-- C5001
---
CREATE TABLE ptf_big_trg AS SELECT *, row_number() OVER (PARTITION BY key ORDER
BY grp) grp_num FROM ptf_big_src;
SELECT grp, COUNT(1) cnt FROM ptf_big_trg GROUP BY grp ORDER BY cnt desc;
--
-- A34296
-- B15704
-- C1
---
{code}
Counts by 'grp' are incorrect!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10980) Merge of dynamic partitions loads all data to default partition
Illya Yalovyy created HIVE-10980: Summary: Merge of dynamic partitions loads all data to default partition Key: HIVE-10980 URL: https://issues.apache.org/jira/browse/HIVE-10980 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 0.14.0 Environment: HDP 2.2.4 (also reproduced on apache hive built from trunk) Reporter: Illya Yalovyy Conditions that lead to the issue: 1. Partition columns have different types 2. Both static and dynamic partitions are used in the query 3. Dynamically generated partitions require merge Result: Final data is loaded to __HIVE_DEFAULT_PARTITION__. Steps to reproduce: set hive.exec.dynamic.partition=true; set hive.exec.dynamic.partition.mode=strict; set hive.optimize.sort.dynamic.partition=false; set hive.merge.mapfiles=true; set hive.merge.mapredfiles=true; set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; create external table sdp ( dataint bigint, hour int, req string, cid string, caid string ) row format delimited fields terminated by ','; load data local inpath '../../data/files/dynpartdata1.txt' into table sdp; load data local inpath '../../data/files/dynpartdata2.txt' into table sdp; ... load data local inpath '../../data/files/dynpartdataN.txt' into table sdp; create table tdp (cid string, caid string) partitioned by (dataint bigint, hour int, req string); insert overwrite table tdp partition (dataint=20150316, hour=16, req) select cid, caid, req from sdp where dataint=20150316 and hour=16; select * from tdp order by caid; show partitions tdp; Example of the input file: 20150316,16,reqA,clusterIdA,cacheId1 20150316,16,reqB,clusterIdB,cacheId2 20150316,16,reqA,clusterIdC,cacheId3 20150316,16,reqD,clusterIdD,cacheId4 20150316,16,reqA,clusterIdA,cacheId5 Actual result: clusterIdA cacheId12015031616 __HIVE_DEFAULT_PARTITION__ clusterIdA cacheId12015031616 __HIVE_DEFAULT_PARTITION__ clusterIdB cacheId22015031616 __HIVE_DEFAULT_PARTITION__ clusterIdC cacheId32015031616 __HIVE_DEFAULT_PARTITION__ clusterIdD cacheId42015031616 __HIVE_DEFAULT_PARTITION__ clusterIdA cacheId52015031616 __HIVE_DEFAULT_PARTITION__ clusterIdD cacheId82015031616 __HIVE_DEFAULT_PARTITION__ clusterIdB cacheId92015031616 __HIVE_DEFAULT_PARTITION__ dataint=20150316/hour=16/req=__HIVE_DEFAULT_PARTITION__ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9225) Windowing functions are not executing efficiently when the window is identical
Illya Yalovyy created HIVE-9225:
---
Summary: Windowing functions are not executing efficiently when
the window is identical
Key: HIVE-9225
URL: https://issues.apache.org/jira/browse/HIVE-9225
Project: Hive
Issue Type: Improvement
Components: PTF-Windowing
Affects Versions: 0.13.0
Environment: Linux
Reporter: Illya Yalovyy
Hive optimizer and the runtime are not smart enough to recognize if the
windowing is the same. Even when the window is identical, the windowing is
re-executed again and cause the runtime increase proportionally to # of
windows.
Example:
{code:sql}
select code,min(emp) over (partition by code order by emp range between
current row and 3 following)from sample_big limit 10;
{code}
*Time taken: 1h:36m:12s*
{code:sql}
select code,
min(emp) over (partition by code order by emp range between current row and
3 following),
max(emp) over (partition by code order by emp range between current row and
3 following),
min(salary) over (partition by code order by emp range between current row and
3 following),
max(salary) over (partition by code order by emp range between current row and
3 following)
from sample_big limit 10;
{code}
*Time taken: 4h:0m:37s*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
