from:"Vihang Karajgaonkar via Review Board"

Re: Review Request 69834: HIVE-21083: Removed the truststore location property requirement and removed the warnings on the truststore password property

2019-01-25 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69834/#review212345
---


Fix it, then Ship it!




Some minor comments related to logs. Rest looks good.


standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 472 (patched)


Nit, It think it would be useful to say "Defaults to jssecacerts, if it 
exists, otherwise uses cacerts"



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 362 (patched)


May be its useful to specify what is the default. So a message like ".. has 
not been set. Defaulting to jssecacerts, if it exists. Otherwise, cacerts."



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 369 (original), 371 (patched)


nit, instead of defaulting to default .. may be just say Using default Java 
truststore password.


- Vihang Karajgaonkar


On Jan. 25, 2019, 7:22 p.m., Morio Ramdenbourg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69834/
> ---
> 
> (Updated Jan. 25, 2019, 7:22 p.m.)
> 
> 
> Review request for hive, Adam Holley, Karthik Manamcheri, Na Li, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-21083
> https://issues.apache.org/jira/browse/HIVE-21083
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> It was identified that a valid way of configuring TLS is by using the Java 
> default truststore and directly adding the trusted certificates to it. The 
> previous HMS implementation did not support this.
> 
> Modified the TLS properties in the following ways:
>  - Removed the requirement for metastore.dbaccess.ssl.truststore.path. If the 
> user does not specify a custom one, then it will default to the Java 
> truststore.
>  - Removed the logs / warnings on metastore.dbaccess.ssl.truststore.password. 
> This used to generate a lot of noise if the user did not provide one. Also, 
> the contents of the truststore is certificates, which is public information 
> and doesn't require strict security.
>  - Removed the unit test that checks for an empty truststore path.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  75f0c0a356f3b894408aa54b9cce5220d47d7f26 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  9f721243c94d48eef35acdcbd0c2e143ab6d23ec 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  29738ba19b0d5ed9ec224d2288c0c1c922d0674c 
> 
> 
> Diff: https://reviews.apache.org/r/69834/diff/3/
> 
> 
> Testing
> ---
> 
> - Existing unit test coverage
> - Manual testing by verifying that these properties can configure TLS to a 
> MySQL DB
> 
> 
> Thanks,
> 
> Morio Ramdenbourg
> 
>

Re: Review Request 69585: HIVE-20776: Run HMS filterHooks on server-side in addition to client-side

2019-01-18 Thread Vihang Karajgaonkar via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69585/#review212147
---

Fix it, then Ship it!

Overall the patch looks good. I have some minor suggestions below. RLGTM.

standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FilterUtils.java
Lines 36 (patched)

please add documentation for each method

standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FilterUtils.java
Lines 236 (patched)

more descriptive message would be dbName is null. Same for line 251 tblName 
is null.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 628 (patched)

nit, could be simplified as 
filterHook = isServerFilterEnabled ? loadFilterHooks() : null;

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 668-671 (patched)

Isn't this redundant since you already checked for a valid configuration in 
getIfServerFilterenabled() method? A easier way would be to add 

Preconditions.checkState(!isBlank(MetastoreConf.getVar(conf, 
ConfVars.FILTER_HOOK)));

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 4731 (patched)

nit, remove "For improved performance". I am not very convinced that this 
is helping the performance. its okay to say "we'll check if the said db and 
table are to be filtered out, if so, then we won't proceed with querying the 
partitions."

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 4873 (patched)

nit, remove "For improved performance". I am not very convinced that this 
is helping the performance. its okay to say "we'll check if the said db and 
table are to be filtered out, if so, then we won't proceed with querying the 
partitions."

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 5682 (patched)

nit, remove "For improved performance". I am not very convinced that this 
is helping the performance. its okay to say "we'll check if the said db and 
table are to be filtered out, if so, then we won't proceed with querying the 
partitions."

Same comment for other places with this comment.

standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetastoreFilterHook.java
Lines 52 (patched)

If this test class added more test coverage to TestFilterHooks test, I 
would suggest to move new tests in TestFilterHooks instead of removing 
TestFilterHooks and adding a new test class which copies all the code from 
TestFilterHooks. That way you don't lose the git history of TestFilterHooks 
unnecessarily.

- Vihang Karajgaonkar

On Jan. 18, 2019, 3:45 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69585/
> ---
> 
> (Updated Jan. 18, 2019, 3:45 p.m.)
> 
> 
> Review request for hive, Adam Holley, Karthick Sankarachary, Morio 
> Ramdenbourg, Peter Vary, Sergio Pena, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20776
> https://issues.apache.org/jira/browse/HIVE-20776
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> add filtering to read result at HMS server, so user cannot see metadata 
> he/she has no privileges. Filtering is enabled/disabled based on 
> configuration.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  19bd9bac84c20f94ac819a80e3cc89e0dc66396d 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  be1f8c78497fe3d0816ad3935ba07cd5ad379b08 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FilterUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestFilterHooks.java
>  7dc69bc4e92875c8962dcd313b16f0f90ea8b057 
>   
>

Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

2019-01-14 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/
---

(Updated Jan. 15, 2019, midnight)


Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.


Changes
---

added suggested change by Bharath


Bugs: HIVE-21077
https://issues.apache.org/jira/browse/HIVE-21077


Repository: hive-git


Description
---

HIVE-21077 : Database and Catalogs should have creation time


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
 3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 994797698a379e0b08604d73d2d6728a2fcee4df 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 13e287e352bdbfe5263b058e1b430af8613fe815 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 8f149d1d6e2a5b9571eeef3c05d68834e4035172 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 9e5f0860f2b0e8caa9abf213e2a2c91b8e16d985 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
9576f8775a4a8a314e09462cbaaaeaebd3b4921f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d43c0c1e70cffbebd39b05f89ec396227c58ac77 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
 f3d2182a04ab81417a4ba58d9340721513e8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
 e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
 815b39c483b2233660310983d58194fb1ab2d107 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
caaec457194332a99d5cd57bef746e969dd38161 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 a3c4196dbff7e53be5317631b314983d16a99020 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 bcaebd18accf86846ae44a6498046514575fc069 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 edde08db9ef7ee01800c7cc3a04c813014abdd18 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 701acb00984c61f7511dcc48053890b154575d1f 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 b1980c5b83f16614845063516495188ebdd8c2a3 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 b9f63313251ab1fa6278b862ed9e07e62b234c04 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 0c36069d071d4b60cc338ba729da5d22e08ca8ca 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
 bb20d9f42a855100397140f9e018c04c5f61dde7 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestCatalogs.java
 28eb1fadca80dfd3c962e4163120b83f00410c4a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 d323ac6c90ed20f092b4e179fdb1bed8602ecf63 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java
 c2eb6c9e22a22f09cc1d2cc6394aa4e0e339b63a 


Diff: https://reviews.apache.org/r/69664/diff/7/

Changes: https://reviews.apache.org/r/69664/diff/6-7/


Testing
---

Ran ITests to check the db install and upgrade scripts are working for mysql, 
postgres, oracle and derby databases. The ITests for mssql is timing out for 
some reason due to container provisioning issues.


Thanks,

Vihang Karajgaonkar

Re: Review Request 69585: HIVE-20776: Run HMS filterHooks on server-side in addition to client-side

2019-01-10 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69585/#review211843
---


Fix it, then Ship it!





standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 652-655 (patched)


I think this is misleading since this assumes that 
DefaultMetaStoreFilterHook will not add anything in the future too. We should 
either depend only on METASTORE_SERVER_FILTER_ENABLED and run whichever 
filterHook is configured or throw an error when METASTORE_SERVER_FILTER_ENABLED 
is true but FILTER_HOOK is empty or DefaultMetaStoreFilterHook instead of 
silently disabling the filtering logic.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 661 (patched)


I didn't realize that FILTER_HOOK takes only one value not a comma 
separated list of classnames. In that case, this code can be made simpler using 
existing util methods

Class clazz = 
JavaUtils.getClass(MetastoreConf.getVar(conf, ConfVars.FILTER_HOOK), 
MetaStoreFilterHook.class);

return JavaUtils.getInstance(clazz, new Class[] {Configuration.class}, 
new Object[] {conf});



standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestFilterHooks.java
Lines 254 (patched)


Its not clear why this should throw NoSuchObjectException? Can you please 
add a comment. Are we changing the behavior of this API?



standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetastoreFilterHook.java
Lines 49 (patched)


This test looks very familiar to TestFilterHooks. What is the difference? 
If it is different, can you please add some javadoc on the top to explain what 
the test is doing. If there is not much difference can we refactor (or add 
these tests to TestFilterHooks) to re-use the code instead of duplicating it?


- Vihang Karajgaonkar


On Jan. 10, 2019, 9:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69585/
> ---
> 
> (Updated Jan. 10, 2019, 9:10 p.m.)
> 
> 
> Review request for hive, Adam Holley, Karthick Sankarachary, Morio 
> Ramdenbourg, Peter Vary, Sergio Pena, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20776
> https://issues.apache.org/jira/browse/HIVE-20776
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> add filtering to read result at HMS server, so user cannot see metadata 
> he/she has no privileges. Filtering is enabled/disabled based on 
> configuration.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  748b56b0a268c1ec7dea022722478ec50889c016 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  be1f8c78497fe3d0816ad3935ba07cd5ad379b08 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FilterUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestFilterHooks.java
>  7dc69bc4e92875c8962dcd313b16f0f90ea8b057 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetastoreFilterHook.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69585/diff/8/
> 
> 
> Testing
> ---
> 
> Existing unit tests passed. 
> add new unit tests for filtering at HMS server and HMS client
> add code to enabled/disable filtering at HMS client based on configuration
> add code to enabled/disable filtering at HMS server based on configuration
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

2019-01-09 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/
---

(Updated Jan. 9, 2019, 6:50 p.m.)


Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.


Changes
---

Fixed TestCacheStore test failure


Bugs: HIVE-21077
https://issues.apache.org/jira/browse/HIVE-21077


Repository: hive-git


Description
---

HIVE-21077 : Database and Catalogs should have creation time


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
 3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 994797698a379e0b08604d73d2d6728a2fcee4df 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 2953a8f327eabdee42dbc66e0c65f89d17add59a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d43c0c1e70cffbebd39b05f89ec396227c58ac77 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
 f3d2182a04ab81417a4ba58d9340721513e8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
 e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
 815b39c483b2233660310983d58194fb1ab2d107 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
caaec457194332a99d5cd57bef746e969dd38161 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 a3c4196dbff7e53be5317631b314983d16a99020 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 bcaebd18accf86846ae44a6498046514575fc069 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 edde08db9ef7ee01800c7cc3a04c813014abdd18 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 701acb00984c61f7511dcc48053890b154575d1f 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 b1980c5b83f16614845063516495188ebdd8c2a3 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 b9f63313251ab1fa6278b862ed9e07e62b234c04 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 0c36069d071d4b60cc338ba729da5d22e08ca8ca 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
 bb20d9f42a855100397140f9e018c04c5f61dde7 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestCatalogs.java
 28eb1fadca80dfd3c962e4163120b83f00410c4a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 d323ac6c90ed20f092b4e179fdb1bed8602ecf63 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java
 c2eb6c9e22a22f09cc1d2cc6394aa4e0e339b63a 


Diff: https://reviews.apache.org/r/69664/diff/6/

Changes: https://reviews.apache.org/r/69664/diff/5-6/


Testing
---

Ran ITests to check the db install and upgrade scripts are working for mysql, 
postgres, oracle and derby databases. The ITests for mssql is timing out for 
some reason due to container provisioning issues.


Thanks,

Vihang Karajgaonkar

Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

2019-01-08 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/
---

(Updated Jan. 9, 2019, 1:01 a.m.)


Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.


Changes
---

Fixed test failures.


Bugs: HIVE-21077
https://issues.apache.org/jira/browse/HIVE-21077


Repository: hive-git


Description
---

HIVE-21077 : Database and Catalogs should have creation time


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
 3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 994797698a379e0b08604d73d2d6728a2fcee4df 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 2953a8f327eabdee42dbc66e0c65f89d17add59a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d43c0c1e70cffbebd39b05f89ec396227c58ac77 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
 f3d2182a04ab81417a4ba58d9340721513e8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
 e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
 815b39c483b2233660310983d58194fb1ab2d107 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
caaec457194332a99d5cd57bef746e969dd38161 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 a3c4196dbff7e53be5317631b314983d16a99020 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 bcaebd18accf86846ae44a6498046514575fc069 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 edde08db9ef7ee01800c7cc3a04c813014abdd18 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 701acb00984c61f7511dcc48053890b154575d1f 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 b1980c5b83f16614845063516495188ebdd8c2a3 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 b9f63313251ab1fa6278b862ed9e07e62b234c04 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 0c36069d071d4b60cc338ba729da5d22e08ca8ca 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestCatalogs.java
 28eb1fadca80dfd3c962e4163120b83f00410c4a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 d323ac6c90ed20f092b4e179fdb1bed8602ecf63 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java
 c2eb6c9e22a22f09cc1d2cc6394aa4e0e339b63a 


Diff: https://reviews.apache.org/r/69664/diff/4/

Changes: https://reviews.apache.org/r/69664/diff/3-4/


Testing
---

Ran ITests to check the db install and upgrade scripts are working for mysql, 
postgres, oracle and derby databases. The ITests for mssql is timing out for 
some reason due to container provisioning issues.


Thanks,

Vihang Karajgaonkar

Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

2019-01-08 Thread Vihang Karajgaonkar via Review Board



> On Jan. 7, 2019, 8:28 p.m., Adam Holley wrote:
> > standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
> > Lines 137 (patched)
> > 
> >
> > nit: extraneous space.

The api/Catalog.java and api/Database.java are auto-generated files which 
cannot not be edited.


> On Jan. 7, 2019, 8:28 p.m., Adam Holley wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
> > Lines 1063 (patched)
> > 
> >
> > Need parenthesis around (System.currentTimeMillis() / 1000).

Thanks for catching this.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/#review211734
---


On Jan. 7, 2019, 7:16 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69664/
> ---
> 
> (Updated Jan. 7, 2019, 7:16 p.m.)
> 
> 
> Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.
> 
> 
> Bugs: HIVE-21077
> https://issues.apache.org/jira/browse/HIVE-21077
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21077 : Database and Catalogs should have creation time
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
>  3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
>  994797698a379e0b08604d73d2d6728a2fcee4df 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  2953a8f327eabdee42dbc66e0c65f89d17add59a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d43c0c1e70cffbebd39b05f89ec396227c58ac77 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
>  f3d2182a04ab81417a4ba58d9340721513e8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
>  e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
>  815b39c483b2233660310983d58194fb1ab2d107 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> caaec457194332a99d5cd57bef746e969dd38161 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  a3c4196dbff7e53be5317631b314983d16a99020 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  bcaebd18accf86846ae44a6498046514575fc069 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  edde08db9ef7ee01800c7cc3a04c813014abdd18 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  701acb00984c61f7511dcc48053890b154575d1f 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  b1980c5b83f16614845063516495188ebdd8c2a3 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  b9f63313251ab1fa6278b862ed9e07e62b234c04 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
>  9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
>   
> standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
>

Re: Review Request 69585: HIVE-20776: Run HMS filterHooks on server-side in addition to client-side

2019-01-08 Thread Vihang Karajgaonkar via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69585/#review211768
---

Thanks for the patch. Do we really need to introduce
authorizeTableForPartitionMetadata in these API calls. For the common case, it
can potentially degrade API performance. For instance, for fetching a single
partition, we are now doing a get_table and then get_partition for the common
case. I think if it is not related to the functionality of this patch, we
should do it in a separate patch with more investigation on its impact.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 84 (patched)

redundant import?

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 217 (patched)

If you want to initialize this member using init() it shouldn't be static
since it relies on the conf object which is not static. Technically, there is a
race-condition in this variable since it is being overwritten every time init()
method is called with the instance specific conf object.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 649 (patched)

nit, formatting. The curly brace convention we follow is if () {
blah;
}

Easiest way to fix these errors is to import the code-style formatter xml
file from the dev-support/eclipse-styles.xml (works for intellij too) and let
IDE to reformat the newly added code.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 663-676 (patched)

You can reuse a existing method which does this with some minor renaming of
the method and variables. The implementation of
MetaStoreServerUtils.getMetaStoreListeners is generic enough to be used to any
class. We probably can just rename it to more generic like
getInstancesFromClass for example.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 667-674 (patched)

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 682 (patched)

Is there a better way to do this? This method is introducing a additional
db call for all the methods for the common case of users having the required
permissions.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 3143-3145 (patched)

Shouldn't the FilterUtils.filterTables be used here for consistency?

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 4610 (patched)

The original API is fetching only one partition, this method is not
improving performance but rather degrading it since this would do a fetch table
and fetch partition for the most common case. I think we should do this check
only in case of fetching lots of partitions where the cost of doing one
get_table call is relatively low compared to fetching lots of partitions.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 4655 (patched)

same comment as above. not sure if this method is helping much with the
performance here.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 4683 (patched)

same comment as above. not sure if this method is helping much with the
performance here.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 4711 (patched)

move this method call below checkLimitNumberOfPartitionsByFilter.

standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetastoreFilterHook.java
Lines 318 (patched)

Don't we need a boolean argument here too to confirm that only server side
filter logic is tested?

- Vihang Karajgaonkar

On Jan. 8, 2019, 8:03 p.m., Na Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69585/
>

Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

2019-01-07 Thread Vihang Karajgaonkar via Review Board



> On Jan. 4, 2019, 5:33 p.m., Karthik Manamcheri wrote:
> >
> 
> Karthik Manamcheri wrote:
> 1. While we are at it, can we make sure that the "create time" exists for 
> everything else (if required)? Such as catalog, partitions..
> 2. Out of curiosity.. are there any tests to ensure that the upgrade path 
> works? What if we miss adding this to some sql file? Would we catch it?
> 
> Vihang Karajgaonkar wrote:
> createTime exists for tables and partitions. It doesn't exist for 
> catalog. I will see if it doesn't add too much to this patch to add 
> createTime to catalogs. Otherwise, I will prefer to do it in a separate 
> patch. For upgrade testing the precommit job does not run such tests. It is 
> expected that schema patches are tested either manually or using the ITests 
> (steps are provided in DEV-README). I have tested this patch on mysql 
> manually and I am setting up my machine to give a try to the newly introduced 
> ITests currently. Will update the RB once I complete the testing.
> 
> Karthik Manamcheri wrote:
> Don't you think its better to make the schema changes all together 
> instead of splitting it up. If we think we might need creation time for 
> catalogs, we should add it to this changeset because we can knock off the 
> schema change as well.

It was not too hard to add createTime to catalog. Added that in the latest 
patch.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/#review211677
---


On Jan. 7, 2019, 7:16 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69664/
> ---
> 
> (Updated Jan. 7, 2019, 7:16 p.m.)
> 
> 
> Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.
> 
> 
> Bugs: HIVE-21077
> https://issues.apache.org/jira/browse/HIVE-21077
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21077 : Database and Catalogs should have creation time
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
>  3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
>  994797698a379e0b08604d73d2d6728a2fcee4df 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  2953a8f327eabdee42dbc66e0c65f89d17add59a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d43c0c1e70cffbebd39b05f89ec396227c58ac77 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
>  f3d2182a04ab81417a4ba58d9340721513e8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
>  e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
>  815b39c483b2233660310983d58194fb1ab2d107 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> caaec457194332a99d5cd57bef746e969dd38161 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  a3c4196dbff7e53be5317631b314983d16a99020 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  bcaebd18accf86846ae44a6498046514575fc069 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  edde08db9ef7ee01800c7cc3a04c813014abdd18 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
>   
>

Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

2019-01-07 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/
---

(Updated Jan. 7, 2019, 7:16 p.m.)


Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.


Changes
---

added create time to catalogs as well as suggested.


Bugs: HIVE-21077
https://issues.apache.org/jira/browse/HIVE-21077


Repository: hive-git


Description
---

HIVE-21077 : Database and Catalogs should have creation time


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
 3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 994797698a379e0b08604d73d2d6728a2fcee4df 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 2953a8f327eabdee42dbc66e0c65f89d17add59a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d43c0c1e70cffbebd39b05f89ec396227c58ac77 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
 f3d2182a04ab81417a4ba58d9340721513e8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
 e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
 815b39c483b2233660310983d58194fb1ab2d107 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
caaec457194332a99d5cd57bef746e969dd38161 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 a3c4196dbff7e53be5317631b314983d16a99020 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 bcaebd18accf86846ae44a6498046514575fc069 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 edde08db9ef7ee01800c7cc3a04c813014abdd18 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 701acb00984c61f7511dcc48053890b154575d1f 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 b1980c5b83f16614845063516495188ebdd8c2a3 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 b9f63313251ab1fa6278b862ed9e07e62b234c04 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 0c36069d071d4b60cc338ba729da5d22e08ca8ca 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestCatalogs.java
 28eb1fadca80dfd3c962e4163120b83f00410c4a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 d323ac6c90ed20f092b4e179fdb1bed8602ecf63 


Diff: https://reviews.apache.org/r/69664/diff/3/

Changes: https://reviews.apache.org/r/69664/diff/2-3/


Testing
---

Ran ITests to check the db install and upgrade scripts are working for mysql, 
postgres, oracle and derby databases. The ITests for mssql is timing out for 
some reason due to container provisioning issues.


Thanks,

Vihang Karajgaonkar

Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

2019-01-04 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/
---

(Updated Jan. 5, 2019, 2:54 a.m.)


Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.


Changes
---

Added createTime in catalog as suggested.


Summary (updated)
-

HIVE-21077 : Database and Catalogs should have creation time


Bugs: HIVE-21077
https://issues.apache.org/jira/browse/HIVE-21077


Repository: hive-git


Description (updated)
---

HIVE-21077 : Database and Catalogs should have creation time


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
 3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 994797698a379e0b08604d73d2d6728a2fcee4df 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 2953a8f327eabdee42dbc66e0c65f89d17add59a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d43c0c1e70cffbebd39b05f89ec396227c58ac77 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
 f3d2182a04ab81417a4ba58d9340721513e8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
 e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
 815b39c483b2233660310983d58194fb1ab2d107 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
caaec457194332a99d5cd57bef746e969dd38161 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 a3c4196dbff7e53be5317631b314983d16a99020 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 bcaebd18accf86846ae44a6498046514575fc069 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 edde08db9ef7ee01800c7cc3a04c813014abdd18 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 701acb00984c61f7511dcc48053890b154575d1f 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 b1980c5b83f16614845063516495188ebdd8c2a3 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 b9f63313251ab1fa6278b862ed9e07e62b234c04 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 0c36069d071d4b60cc338ba729da5d22e08ca8ca 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestCatalogs.java
 28eb1fadca80dfd3c962e4163120b83f00410c4a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 d323ac6c90ed20f092b4e179fdb1bed8602ecf63 


Diff: https://reviews.apache.org/r/69664/diff/2/

Changes: https://reviews.apache.org/r/69664/diff/1-2/


Testing (updated)
---

Ran ITests to check the db install and upgrade scripts are working for mysql, 
postgres, oracle and derby databases. The ITests for mssql is timing out for 
some reason due to container provisioning issues.


Thanks,

Vihang Karajgaonkar

Re: Review Request 69664: HIVE-21077 : Database should have creation time

2019-01-04 Thread Vihang Karajgaonkar via Review Board



> On Jan. 4, 2019, 5:33 p.m., Karthik Manamcheri wrote:
> >
> 
> Karthik Manamcheri wrote:
> 1. While we are at it, can we make sure that the "create time" exists for 
> everything else (if required)? Such as catalog, partitions..
> 2. Out of curiosity.. are there any tests to ensure that the upgrade path 
> works? What if we miss adding this to some sql file? Would we catch it?

createTime exists for tables and partitions. It doesn't exist for catalog. I 
will see if it doesn't add too much to this patch to add createTime to 
catalogs. Otherwise, I will prefer to do it in a separate patch. For upgrade 
testing the precommit job does not run such tests. It is expected that schema 
patches are tested either manually or using the ITests (steps are provided in 
DEV-README). I have tested this patch on mysql manually and I am setting up my 
machine to give a try to the newly introduced ITests currently. Will update the 
RB once I complete the testing.


> On Jan. 4, 2019, 5:33 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
> > Lines 785 (patched)
> > 
> >
> > Why not just store it in milliseconds?

This is to be consitent with the createTime stored in table and partitions


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/#review211677
---


On Jan. 3, 2019, 11:44 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69664/
> ---
> 
> (Updated Jan. 3, 2019, 11:44 p.m.)
> 
> 
> Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.
> 
> 
> Bugs: HIVE-21077
> https://issues.apache.org/jira/browse/HIVE-21077
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21077 : Database should have creation time
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
>  994797698a379e0b08604d73d2d6728a2fcee4df 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  2953a8f327eabdee42dbc66e0c65f89d17add59a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a9398ae1e79404a15894aa42f451df5d18ed3e4c 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d43c0c1e70cffbebd39b05f89ec396227c58ac77 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
>  f3d2182a04ab81417a4ba58d9340721513e8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
>  815b39c483b2233660310983d58194fb1ab2d107 
>   standalone-metastore/metastore-server/src/main/resources/package.jdo 
> caaec457194332a99d5cd57bef746e969dd38161 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
>  a3c4196dbff7e53be5317631b314983d16a99020 
>   
> standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
>  bcaebd18accf86846ae44a6498046514575fc069 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
>  5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
>   
> standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
>  edde08db9ef7ee01800c7cc3a04c813014abdd18 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
>  a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
>   
> standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
>  701acb00984c61f7511dcc48053890b154575d1f 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
>  b1980c5b83f16614845063516495188ebdd8c2a3 
>   
> standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
>  b9f63313251ab1fa6278b862ed9e07e62b234c04 
>   
>

Review Request 69664: HIVE-21077 : Database should have creation time

2019-01-03 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/
---

Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.


Bugs: HIVE-21077
https://issues.apache.org/jira/browse/HIVE-21077


Repository: hive-git


Description
---

HIVE-21077 : Database should have creation time


Diffs
-

  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 994797698a379e0b08604d73d2d6728a2fcee4df 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 2953a8f327eabdee42dbc66e0c65f89d17add59a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d43c0c1e70cffbebd39b05f89ec396227c58ac77 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
 f3d2182a04ab81417a4ba58d9340721513e8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
 815b39c483b2233660310983d58194fb1ab2d107 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
caaec457194332a99d5cd57bef746e969dd38161 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 a3c4196dbff7e53be5317631b314983d16a99020 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 bcaebd18accf86846ae44a6498046514575fc069 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 edde08db9ef7ee01800c7cc3a04c813014abdd18 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 701acb00984c61f7511dcc48053890b154575d1f 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 b1980c5b83f16614845063516495188ebdd8c2a3 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 b9f63313251ab1fa6278b862ed9e07e62b234c04 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 0c36069d071d4b60cc338ba729da5d22e08ca8ca 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 d323ac6c90ed20f092b4e179fdb1bed8602ecf63 


Diff: https://reviews.apache.org/r/69664/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 69534: HIVE-20992: Split the property "hive.metastore.dbaccess.ssl.properties" into more coherent and user-friendly properties.

2018-12-17 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69534/#review211367
---


Fix it, then Ship it!




Thanks for the changes. Couple of minor comments. Rest looks good to me.


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 133-135 (patched)


Nit, constants are recommended to be in upper-case.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 379 (patched)


you are catching IOException and changing it to IllegalArgumentException 
which may not be the right thing to do all the times. I would suggest changing 
this to RuntimeException since setConf() doesn't throw checked exceptions. 
Also, would be good to have e thrown as a cause as well, so would suggest using 
throw new RuntimeException("Failed to set .., e); constructor instead.



standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
Lines 1082 (patched)


If you decide to change the thrown exception to RuntimeException, this 
might need a change too.


- Vihang Karajgaonkar


On Dec. 16, 2018, 3:11 a.m., Morio Ramdenbourg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69534/
> ---
> 
> (Updated Dec. 16, 2018, 3:11 a.m.)
> 
> 
> Review request for hive, Adam Holley, Karthik Manamcheri, Peter Vary, and 
> Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20992
> https://issues.apache.org/jira/browse/HIVE-20992
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following new properties were added:
> 
> 1. metastore.dbaccess.use.SSL (hive.metastore.dbaccess.use.SSL)
> 2. javax.net.ssl.trustStore
> 3. javax.net.ssl.trustStorePassword
> 4. javax.net.ssl.trustStoreType
> 
> This was in an effort to guide the user towards an easier SSL
> configuration experience. This is the minimum requirement to set up SSL
> encryption to the HMS backend store.
> 
> This also solves the issue of the truststore password being stored in
> plain text. It can now be encrypted by default and loaded through the
> MetastoreConf.getPassword() method which handles secure password access
> 
> The property "hive.metastore.dbaccess.ssl.properties" is now
> deprecated, but it will still be kept for backwards-compatibility purposes.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  e25a8cf9a19d78c0cc00bb2e5e0abee4d851ad98 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  3fa21b768cd120cd89343c9ccd142d5e2ccdef2e 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  0cf113c927f2274d085e07cd72921fb35227e1f3 
> 
> 
> Diff: https://reviews.apache.org/r/69534/diff/5/
> 
> 
> Testing
> ---
> 
> Tests:
> 1. Unit tests were added to cover the functionality of configuring the Java 
> system properties.
> 2. Performed some manual and sanity tests to ensure that SSL was still 
> configurable to a remote DB. I performed these on MySQL, PostgreSQL, Oracle, 
> and Derby DB by creating generic DB hosts and setting them up with SSL. Once 
> SSL was set up, I triggered the metastore to perform database calls, and 
> captured packets using tcpdump. I then uploaded my packet captures to 
> Wireshark, and ensured that none of the data was human-readable.
> 
> I plan to upload a document to our Wiki explaining the process of enabling 
> TLS to these databases.
> 
> 
> Thanks,
> 
> Morio Ramdenbourg
> 
>

Re: Review Request 69534: HIVE-20992: Split the property "hive.metastore.dbaccess.ssl.properties" into more coherent and user-friendly properties.

2018-12-14 Thread Vihang Karajgaonkar via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69534/#review211343
---

standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 462 (patched)

s/System/Java system

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 328 (original), 342 (patched)

This patch may introduce performance regression in the setConf method which 
is called for every new connection. MetastoreConf.getPassword is expensive 
since it needs to decrypt the truststore password. See HIVE-20740 which has 
more details. In most common cases, these configuration properties almost never 
change once they are set. But they are being read again and again at every new 
connection initialization time. I think we can improve this by caching the db 
and truststore password and reading it once when HMS starts. But I guess this 
could be separate from this JIRA.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 343 (patched)

may be rename this to configureSSLDeprecated

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 346 (patched)

This is a redundant log. If ssl is configured we have other logs below to 
tell us that. Note that logs printed in this method are going to be very 
frequent for each new HMS connection.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 354 (patched)

The exception message suggests taht it Disables SSL and continues, but 
actually, the exception is uncaught and it will terminate the connection 
request (and infact HMS coming up). I think you can remove "Disabling SSL and 
continuing." if thats not needed.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 356 (patched)

This comment is unclear. The getPassword method will get the encrypted 
password if configured right?

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 367-369 (patched)

Can you define constants for javax.net.ssl* property keys at the class 
level?

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 371 (patched)

Redundant log, please remove.

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 373 (patched)

Don't see the code where we disable and continue. Am I missing something?

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 332 (original), 398 (patched)

I would suggest change this log to say "Configuring SSL using a deprecated 
key " + ConfVars.DBACCESS_SSL_PROPS.toString() + ". This may be removed in the 
future. See HIVE-20992 for more details."

standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
Lines 1069 (patched)

suggest rename to testDeprecatedConfigIsOverriden()

- Vihang Karajgaonkar

On Dec. 14, 2018, 1:26 a.m., Morio Ramdenbourg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69534/
> ---
> 
> (Updated Dec. 14, 2018, 1:26 a.m.)
> 
> 
> Review request for hive, Adam Holley, Karthik Manamcheri, Peter Vary, and 
> Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20992
> https://issues.apache.org/jira/browse/HIVE-20992
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following new properties were added:
> 
> 1. metastore.dbaccess.use.SSL (hive.metastore.dbaccess.use.SSL)
> 2. javax.net.ssl.trustStore
> 3. javax.net.ssl.trustStorePassword
> 4. javax.net.ssl.trustStoreType
> 
> This was in an effort to guide the user towards an easier SSL
> configuration experience. This is the minimum requirement to set up SSL
> encryption to the HMS backend store.
> 
> This also solves the issue of the truststore password being stored in
> plain text. It can now

Re: Review Request 69550: Add credential store env properties redaction in JobConf

2018-12-11 Thread Vihang Karajgaonkar via Review Board



> On Dec. 11, 2018, 4:30 p.m., Vihang Karajgaonkar wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java
> > Lines 217 (patched)
> > 
> >
> > Do we need to do redact for the Spark config as well?
> 
> Denys Kuzmenko wrote:
> no, it's already handled in Spark:
> 
> https://github.com/apache/spark/commit/66636ef0b046e5d1f340c3b8153d7213fa9d19c7

Okay. Thanks for checking.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/#review211199
---


On Dec. 11, 2018, 12:43 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69550/
> ---
> 
> (Updated Dec. 11, 2018, 12:43 p.m.)
> 
> 
> Review request for hive, Peter Vary and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-21030
> https://issues.apache.org/jira/browse/HIVE-21030
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Credstore decryption password should be redacted in JobConf
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
> 2ad5f9ee39f376d3466994a24cc9f7850be902ae 
> 
> 
> Diff: https://reviews.apache.org/r/69550/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 69550: Add credential store env properties redaction in JobConf

2018-12-11 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/#review211205
---


Ship it!




Ship It!

- Vihang Karajgaonkar


On Dec. 11, 2018, 12:43 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69550/
> ---
> 
> (Updated Dec. 11, 2018, 12:43 p.m.)
> 
> 
> Review request for hive, Peter Vary and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-21030
> https://issues.apache.org/jira/browse/HIVE-21030
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Credstore decryption password should be redacted in JobConf
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
> 2ad5f9ee39f376d3466994a24cc9f7850be902ae 
> 
> 
> Diff: https://reviews.apache.org/r/69550/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 69550: Add credential store env properties redaction in JobConf

2018-12-11 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/#review211199
---




common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java
Lines 217 (patched)


Do we need to do redact for the Spark config as well?


- Vihang Karajgaonkar


On Dec. 11, 2018, 12:43 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69550/
> ---
> 
> (Updated Dec. 11, 2018, 12:43 p.m.)
> 
> 
> Review request for hive, Peter Vary and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-21030
> https://issues.apache.org/jira/browse/HIVE-21030
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Credstore decryption password should be redacted in JobConf
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
> 2ad5f9ee39f376d3466994a24cc9f7850be902ae 
> 
> 
> Diff: https://reviews.apache.org/r/69550/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-11-27 Thread Vihang Karajgaonkar via Review Board



> On Nov. 27, 2018, 10:55 p.m., Naveen Gangam wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
> > Lines 162 (patched)
> > 
> >
> > Should we acquire the writeLock before releasing the readLock to 
> > prevent another thread from entering this code? This way we dont have to 
> > re-check these as well
> >   if (prop == null || pmf == null || !propsFromConf.equals(prop))

AFAIK we cannot upgrade from readLock to a writeLock when using 
ReentrantReadWriteLock, only downgrade is possible (a thread which holds write 
lock can acquire read lock) 
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/#review210911
---


On Nov. 27, 2018, 7:18 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69054/
> ---
> 
> (Updated Nov. 27, 2018, 7:18 a.m.)
> 
> 
> Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.
> 
> 
> Bugs: HIVE-20740
> https://issues.apache.org/jira/browse/HIVE-20740
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20740 : Remove global lock in ObjectStore.setConf method
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
>  5a88550f0625a7ec1890df7f54e7fa579f58fff4 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 5cb0a887e672f49739f5b648e608fba66de06326 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 455ffc3887e62fa503cc3fa28255702ea9da3cc0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  570281b54fa236d5bb568b4ded9b166ef367f613 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  af9efd98ea210335c6ac1d3da8624e02aadc2706 
> 
> 
> Diff: https://reviews.apache.org/r/69054/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-11-26 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/
---

(Updated Nov. 27, 2018, 7:18 a.m.)


Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.


Changes
---

Rebased to the latest code on master.


Bugs: HIVE-20740
https://issues.apache.org/jira/browse/HIVE-20740


Repository: hive-git


Description
---

HIVE-20740 : Remove global lock in ObjectStore.setConf method


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
 5a88550f0625a7ec1890df7f54e7fa579f58fff4 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
5cb0a887e672f49739f5b648e608fba66de06326 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
455ffc3887e62fa503cc3fa28255702ea9da3cc0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 570281b54fa236d5bb568b4ded9b166ef367f613 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 af9efd98ea210335c6ac1d3da8624e02aadc2706 


Diff: https://reviews.apache.org/r/69054/diff/6/

Changes: https://reviews.apache.org/r/69054/diff/5-6/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-11-01 Thread Vihang Karajgaonkar via Review Board



> On Oct. 18, 2018, 2:33 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Line 244 (original), 341 (patched)
> > 
> >
> > You are ignoring the return value? Should you have 
> > pmf=getUpdatedPmfIfNeeded(..)?
> 
> Vihang Karajgaonkar wrote:
> The pmf is updated by the method if needed, so we don't need to use the 
> return value. Will rename the method to updatePmfIfNeeded to make it more 
> readable.
> 
> Karthik Manamcheri wrote:
> And you can have it has void return.

comment does not apply to latest version of the patch. Dropping it.


> On Oct. 18, 2018, 2:33 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Line 287 (original), 374 (patched)
> > 
> >
> > What is dss.log?

Actually, I don't know what it is. Looking at the git history, this line has 
been there forever. Looks like its about time to update it :)


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/#review209742
---


On Nov. 1, 2018, 8:48 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69054/
> ---
> 
> (Updated Nov. 1, 2018, 8:48 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.
> 
> 
> Bugs: HIVE-20740
> https://issues.apache.org/jira/browse/HIVE-20740
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20740 : Remove global lock in ObjectStore.setConf method
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
>  75cd68a9d6be6c5804e458d19a0023f0d7f5beae 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 5cb0a887e672f49739f5b648e608fba66de06326 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> ee7c940d2b7fbd66af2d006da0585c6b42b9b0bb 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  03e3a2d2573b54651833867b906821650f4fb9c1 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/69054/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-11-01 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/
---

(Updated Nov. 1, 2018, 8:48 p.m.)


Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.


Changes
---

Fixed failing tests


Bugs: HIVE-20740
https://issues.apache.org/jira/browse/HIVE-20740


Repository: hive-git


Description
---

HIVE-20740 : Remove global lock in ObjectStore.setConf method


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
 75cd68a9d6be6c5804e458d19a0023f0d7f5beae 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
5cb0a887e672f49739f5b648e608fba66de06326 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
ee7c940d2b7fbd66af2d006da0585c6b42b9b0bb 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 03e3a2d2573b54651833867b906821650f4fb9c1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 b74c3048fa2e18adc7f0d7cc813a180d4466fa36 


Diff: https://reviews.apache.org/r/69054/diff/4/

Changes: https://reviews.apache.org/r/69054/diff/3-4/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-10-19 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/
---

(Updated Oct. 19, 2018, 8:54 p.m.)


Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.


Changes
---

Fixed couple of bugs in the patch to fix the test failures.


Bugs: HIVE-20740
https://issues.apache.org/jira/browse/HIVE-20740


Repository: hive-git


Description
---

HIVE-20740 : Remove global lock in ObjectStore.setConf method


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 b74c3048fa2e18adc7f0d7cc813a180d4466fa36 


Diff: https://reviews.apache.org/r/69054/diff/3/

Changes: https://reviews.apache.org/r/69054/diff/2-3/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-10-18 Thread Vihang Karajgaonkar via Review Board



> On Oct. 18, 2018, 2:33 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Line 244 (original), 341 (patched)
> > 
> >
> > You are ignoring the return value? Should you have 
> > pmf=getUpdatedPmfIfNeeded(..)?

The pmf is updated by the method if needed, so we don't need to use the return 
value. Will rename the method to updatePmfIfNeeded to make it more readable.


> On Oct. 18, 2018, 2:33 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
> > Lines 927 (patched)
> > 
> >
> > Is this testing the new read/write lock behavior? As in, would this 
> > test have failed if you had the previous behavior of a global lock?

This is performance fix. There was nothing broken in the previous 
implementation so its hard to come up with a test which break on the previous 
code. The purpose of this test to cover the newly added code so that if there 
are any obvious bugs they will be seen. If you have more ideas on how can we 
add more tests please suggest and I would be happy to add them.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/#review209742
---


On Oct. 16, 2018, 10:47 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69054/
> ---
> 
> (Updated Oct. 16, 2018, 10:47 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.
> 
> 
> Bugs: HIVE-20740
> https://issues.apache.org/jira/browse/HIVE-20740
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20740 : Remove global lock in ObjectStore.setConf method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/69054/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-10-16 Thread Vihang Karajgaonkar via Review Board



> On Oct. 16, 2018, 8:59 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 291 (patched)
> > 
> >
> > What is someone has set  ConfVars.MANAGER_FACTORY_CLASS to some 
> > non-default value? Is this still correct?

Yes, looks like it will fail in that case although I am not sure the use-cases 
where you will use a different PersistenceManagerFactory. This code has been 
there since before the patch and has not been changed in this patch. Perhaps we 
can look at this as a separate JIRA.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/#review209662
---


On Oct. 16, 2018, 10:47 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69054/
> ---
> 
> (Updated Oct. 16, 2018, 10:47 p.m.)
> 
> 
> Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.
> 
> 
> Bugs: HIVE-20740
> https://issues.apache.org/jira/browse/HIVE-20740
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20740 : Remove global lock in ObjectStore.setConf method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  b74c3048fa2e18adc7f0d7cc813a180d4466fa36 
> 
> 
> Diff: https://reviews.apache.org/r/69054/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-10-16 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/
---

(Updated Oct. 16, 2018, 10:47 p.m.)


Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.


Changes
---

Added suggested changes from Andrew


Bugs: HIVE-20740
https://issues.apache.org/jira/browse/HIVE-20740


Repository: hive-git


Description
---

HIVE-20740 : Remove global lock in ObjectStore.setConf method


Diffs (updated)
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 b74c3048fa2e18adc7f0d7cc813a180d4466fa36 


Diff: https://reviews.apache.org/r/69054/diff/2/

Changes: https://reviews.apache.org/r/69054/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Review Request 69054: HIVE-20740 : Remove global lock in ObjectStore.setConf method

2018-10-16 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69054/
---

Review request for hive, Andrew Sherman, Alan Gates, and Peter Vary.


Bugs: HIVE-20740
https://issues.apache.org/jira/browse/HIVE-20740


Repository: hive-git


Description
---

HIVE-20740 : Remove global lock in ObjectStore.setConf method


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 b74c3048fa2e18adc7f0d7cc813a180d4466fa36 


Diff: https://reviews.apache.org/r/69054/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 68969: HIVE-20307 : Add support for filterspec to the getPartitions with projection API

2018-10-09 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68969/
---

(Updated Oct. 9, 2018, 9:23 p.m.)


Review request for hive, Alexander Kolbasov and Andrew Sherman.


Changes
---

Added Andrew's suggested changes.


Bugs: HIVE-20307
https://issues.apache.org/jira/browse/HIVE-20307


Repository: hive-git


Description
---

HIVE-20307 : Add support for filterspec to the getPartitions with projection API


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 d59d5d807a26378a430e683533e53d0831cf9514 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2b57fb646899c54b63be14a8cde9b8644a973aa 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50d69f9120d565f61d028b060d7776689fc 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 08614749b7aba54f9eb9b54ac46f79dbac6bc5cd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 af757932a191675bc8fb9236209a2efba9f3d335 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 a6d9583364be20758444ebe25c8cf636f0ea740f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 70490f09e765d4e42391c67eb5cf018e93ad04aa 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 4dd4edccd66f8ea8ea189a2d27f970c8113e3a0f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 06f4cbce58c16f98257e7f529ffe31c983f2919f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
 dcff606937157e63694657b42392875d50b17be6 


Diff: https://reviews.apache.org/r/68969/diff/2/

Changes: https://reviews.apache.org/r/68969/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Review Request 68969: HIVE-20307 : Add support for filterspec to the getPartitions with projection API

2018-10-09 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68969/
---

Review request for hive, Alexander Kolbasov and Andrew Sherman.


Bugs: HIVE-20307
https://issues.apache.org/jira/browse/HIVE-20307


Repository: hive-git


Description
---

HIVE-20307 : Add support for filterspec to the getPartitions with projection API


Diffs
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 d59d5d807a26378a430e683533e53d0831cf9514 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
a2b57fb646899c54b63be14a8cde9b8644a973aa 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16f4a50d69f9120d565f61d028b060d7776689fc 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 08614749b7aba54f9eb9b54ac46f79dbac6bc5cd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 af757932a191675bc8fb9236209a2efba9f3d335 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 66977d79c946f1ac57aacfbe8704d37bfbac3ea3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 a6d9583364be20758444ebe25c8cf636f0ea740f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 70490f09e765d4e42391c67eb5cf018e93ad04aa 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 4dd4edccd66f8ea8ea189a2d27f970c8113e3a0f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 06f4cbce58c16f98257e7f529ffe31c983f2919f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
 dcff606937157e63694657b42392875d50b17be6 


Diff: https://reviews.apache.org/r/68969/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 68664: HIVE-20306: Implement projection spec for fetching only requested fields from partitions

2018-10-02 Thread Vihang Karajgaonkar via Review Board



> On Oct. 3, 2018, 12:33 a.m., Andrew Sherman wrote:
> > All looks good, just a few nits which I noticed

Unfortunately, I cannot update this review board URL since its owned by 
Alexander. Latest patch on the JIRA has the suggested changes. Will see if I 
can update the diffs here somehow.


> On Oct. 3, 2018, 12:33 a.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Line 1667 (original), 1667 (patched)
> > 
> >
> > I think he previous text 'E.g.' is better

done


> On Oct. 3, 2018, 12:33 a.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Line 1673 (original), 1673 (patched)
> > 
> >
> > Not sure why this changed, should be 'compliant'

done


> On Oct. 3, 2018, 12:33 a.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Line 1676 (original), 1676 (patched)
> > 
> >
> > Not sure why this changed, should be 'compliant'

done


> On Oct. 3, 2018, 12:33 a.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
> > Lines 3765 (patched)
> > 
> >
> > Arguably clearer to say "providing different ways of filtering and 
> > controlling output"

changed


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68664/#review209171
---


On Oct. 3, 2018, 12:07 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68664/
> ---
> 
> (Updated Oct. 3, 2018, 12:07 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Peter Vary, Todd Lipcon, 
> and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306: Implement projection spec for fetching only requested fields from 
> partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  ae0956870a7d01c24f5fdaa07094c3dc6604ab9a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  4574c6a4925ae3df9dd1ee7b8786976ae6fc8397 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  22deffe1d31a64f95c49d7f017dfeb2994233e71 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  38074ce79b8a06b3795d00431025240778abb569 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  38fac465d73c264f85fc512548ebe1919ee35c17 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  0192c6da314694c1253b49949bbe749902f49b4b 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e6a72762bb7b0d36fdf6d20d02cb1da3337a98a0 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  d226db50a520707abee11fe7b81e2d95c39a6679 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>

Re: Review Request 68664: HIVE-20306: Implement projection spec for fetching only requested fields from partitions

2018-10-02 Thread Vihang Karajgaonkar via Review Board



> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
> > Lines 38 (patched)
> > 
> >
> > Why is this new interface marked Stable? I know this is generated code, 
> > but where does ths come from?
> 
> Alexander Kolbasov wrote:
> There is probably some magic code that marks everything here as stable 
> interface

I think this is done in the "process-sources" phase during maven execution. See 
"process-thrift-sources-interface-annotations" in metastore-common/pom.xml


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Lines 1101 (patched)
> > 
> >
> > add method description

done


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
> > Lines 65 (patched)
> > 
> >
> > Add a brief comment explaining what the test does

done


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
> > Lines 84 (patched)
> > 
> >
> > This seems weird
> 
> Alexander Kolbasov wrote:
> This is copied from TestHiveMetastore. I am not sure why it is needed.

removed this line. It was not needed.


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
> > Lines 278 (patched)
> > 
> >
> > Shoudl we have the fields here that are not supported in direct sqL?  
> > serdeType, serializerClass and deserializerClass

added them in the test.


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
> > Lines 300 (patched)
> > 
> >
> > I couldn't work out what "i" was so pls document

added a description


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
> > Lines 309 (patched)
> > 
> >
> > is 3 here just the length of "sd." ?

yes


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
> > Lines 400 (patched)
> > 
> >
> > weird that you set an exclude prefix with a method called include?

the test is actually using setIncludeParamKeyPattern to test the include 
parameters. I just reused existing parameter keys instead of creating new ones. 
Added a comment to make it slightly better :)


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
> > Lines 442 (patched)
> > 
> >
> > assertion message should be that key is found in the list?

fixed.


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestGetPartitionsUsingProjection.java
> > Lines 568 (patched)
> > 
> >
> > maybe this should be an old fashioned for loop?

done


> On Sept. 21, 2018, 8:50 p.m., Andrew Sherman wrote:
> > standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreServerUtils.java
> > Lines 396 (patched)
> > 
> >
> > objects *with* null

done


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68664/#review208866
---


On Oct. 3, 2018, 12:07 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 27, 2018, 10:33 p.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

Added support for JDO result for projected fields.


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 47f96f323ade4e15753d6fd98709b9a882de624d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 3c0d0a55b1dcca96dca77676873b68e52703715d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 e54a7321e2e1e6069a3e598627bc6f6eaed93449 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
feb44d5159f131fae932739923b1a41f5e72e74b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 33b22a9fc3e60cb6e11bec63d397f1fa712a41db 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 a0ff79cc5ca39f8af8bac672393e82d365c9fd4a 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java
 4c6ce008f89469353bfee3175168a518534a42b1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 10ff9dfbb6d8f61fa75f731f4cd0f006c98e0067 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 c681a87a1c6b10a4f9494e49a42282cf90027ad7

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Vihang Karajgaonkar via Review Board



> On Aug. 27, 2018, 4:42 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
> > Line 200 (original), 200 (patched)
> > 
> >
> > Seems it can be static since it doesn't refer to any variables outside. 
> > Maybe I miss anything?

changed this method to static as suggested


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207981
---


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  b61ee81533930c889f23d2551041055cbdd1a6b2 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
>  7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
>   
>

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 25, 2018, 12:23 a.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

Made changes suggested by Aihua


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 47f96f323ade4e15753d6fd98709b9a882de624d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 3c0d0a55b1dcca96dca77676873b68e52703715d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 e54a7321e2e1e6069a3e598627bc6f6eaed93449 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
feb44d5159f131fae932739923b1a41f5e72e74b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 6b660f55aca5b99517447f164339ed92261c9370 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 faf6810420ccf6088a3398711c0c216af722f062 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 506d9ab5590a245049f3c4e71324823465a91586 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Vihang Karajgaonkar via Review Board



> On Aug. 24, 2018, 11:19 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Line 376 (original), 416 (patched)
> > 
> >
> > Why some functions like extractSqlString() not moving into 
> > MetastoreDirectSqlUtils class, but extractSqlLong() gets moved? Can we make 
> > it consistent?

I moved only the functions which were needed in PartitionProjectionEvaluator 
class. Makes sense to move these as well to keep it consistent.


> On Aug. 24, 2018, 11:19 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 547 (patched)
> > 
> >
> > Better add comments for this public function.

Added a javadoc


> On Aug. 24, 2018, 11:19 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 886 (patched)
> > 
> >
> > This function is a little confusing actually. Since we are not used 
> > anywhere else, probably we don't need to create such function seperately.

this was a left over from some of my code changes while development. Reverted 
this part.


> On Aug. 24, 2018, 11:19 p.m., Aihua Xu wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
> > Lines 200 (patched)
> > 
> >
> > This function can be static and probably gives the name find() as well? 
> > findUtil is a little confusing to me.

This method cannot be changed to static since it operates on the roots field 
which is non-static. renamed the findUtil to find as suggested.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207920
---


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Vihang Karajgaonkar via Review Board



> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 852 (patched)
> > 
> >
> > Does this and other new methods belong here or in MetastoreServerUtils?
> 
> Vihang Karajgaonkar wrote:
> Its unclear what is the difference between MetaStoreUtils and 
> MetastoreServerUtils? Is MetaStoreUtils going to move to common in the near 
> future? If yes, I can move it to MetastoreServerUtils.

Moved them to MetastoreServerUtils.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207802
---


On Aug. 24, 2018, 6:15 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 24, 2018, 6:15 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 24, 2018, 6:15 p.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

Added some of the suggested changes by Alexander


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 7ab64eadac7948a7f5077260694926cc5b6e4e4b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 cc19f2389e7b595722dcc1f3296877a02b20e0a4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 d098dba100bdfe56aa6ecafb31880098a2d7c6cb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 92424a43feefc8c0db7c91302045437f3afbf274 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
8a4bdd8ed827572f5fd9d291c5454630d84284bd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 324035a8096a001d3fb170b8640805258d5e2cdd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 5ae00af564b05ec2720a03072f94c8f8579378a7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 91405b9a334a4b031a5dc7f4a1757a3895bfb386 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 70a17f51b9b5a9fb0b5640988318fd39a82b895d

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-23 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 23, 2018, 8:06 p.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

added catName in the thrift definition for filterspec as suggested.


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 7ab64eadac7948a7f5077260694926cc5b6e4e4b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 cc19f2389e7b595722dcc1f3296877a02b20e0a4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 d098dba100bdfe56aa6ecafb31880098a2d7c6cb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 92424a43feefc8c0db7c91302045437f3afbf274 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
8a4bdd8ed827572f5fd9d291c5454630d84284bd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 324035a8096a001d3fb170b8640805258d5e2cdd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 5ae00af564b05ec2720a03072f94c8f8579378a7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 91405b9a334a4b031a5dc7f4a1757a3895bfb386 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 70a17f51b9b5a9fb0b5640988318fd39a82b895d

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-23 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 23, 2018, 6:44 p.m.)


Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and Sergey 
Shelukhin.


Changes
---

Addressed some of the review comments. More to come.


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 7ab64eadac7948a7f5077260694926cc5b6e4e4b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 cc19f2389e7b595722dcc1f3296877a02b20e0a4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 d098dba100bdfe56aa6ecafb31880098a2d7c6cb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 92424a43feefc8c0db7c91302045437f3afbf274 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
8a4bdd8ed827572f5fd9d291c5454630d84284bd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 324035a8096a001d3fb170b8640805258d5e2cdd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 5ae00af564b05ec2720a03072f94c8f8579378a7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 91405b9a334a4b031a5dc7f4a1757a3895bfb386 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 70a17f51b9b5a9fb0b5640988318fd39a82b895d

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-23 Thread Vihang Karajgaonkar via Review Board

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1646 (patched)
> > 
> >
> > does this allow for x.y.z?
> > Does this work for sd columns and other list fields?

yes, this works for multi-valued fields, so something like sd.cols.name will 
work. Only name will be set for all the sd.cols for each partition. I will add 
more documentation here to make it clearer.

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1647 (patched)
> > 
> >
> > Should this be optional?
> > Is this a regexp or something else? What is the syntax here? What are _ 
> > or % wildcards?

This does not need to be optional, the default requiredness of the field makes 
sure that thrift will always check if this field is set and on the writers will 
serialize the field only when its possible (null value is not serialized). Will 
update the documentation

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1648 (patched)
> > 
> >
> > Why not just have optional include and exclude patterns instead of a 
> > pattern and a boolean?

Thats an interesting thought. Do you think it would be useful for clients to 
have something like PARAM_KEY LIKE ('includepattern') AND NOT LIKE 
('excludepattern')? It should not be too difficult to add to have two separate 
include and exclude paramKeyPattern.

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1658 (patched)
> > 
> >
> > Do we need catalog name?

Good point. I will add it.

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 56 (patched)
> > 
> >
> > This shows as an unused import

removed it

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 65 (patched)
> > 
> >
> > Unused import?

removed it.

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 852 (patched)
> > 
> >
> > Does this and other new methods belong here or in MetastoreServerUtils?

Its unclear what is the difference between MetaStoreUtils and 
MetastoreServerUtils? Is MetaStoreUtils going to move to common in the near 
future? If yes, I can move it to MetastoreServerUtils.

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 853 (patched)
> > 
> >
> > It would be good to explain what this method actually does converts 
> > list of objects to a comma-separated string of stringified objects. Then 
> > you can explain the intended use case.

This method is used only in MetastoreDirectSQL class. Moved it back to that 
class. It was probably a left over from the many iterations of my code changes. 
Updated the documentations as suggested.

> On Aug. 23, 2018, 6:26 a.m., Alexander Kolbasov wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
> > Lines 950 (patched)
> > 
> >
> > What is this class?

This class was originally in HMSHandler and used to grouping Partitions based 
on a common StorageDescriptor. In order to do that, this class is used to 
generate the hashKey of partitions so that all the partitions having the same 
hashcode can be grouped together. The hashCode is overridden such that if the 
sd.cols, input/outputformat, serializationLib and base location is same, it 
returns the same hashCode so that all such partitions can be grouped together.

I had to move this class from HMSHandler to MetaStoreUtils so that this code 
could be reused by the new API as well as the previously existing APIs using 
this grouping logic. I think I can move

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-21 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

(Updated Aug. 21, 2018, 6:50 p.m.)


Review request for hive, Alexander Kolbasov and Peter Vary.


Changes
---

Reduced thrift changes only to related files. Fixed test failures.


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 7ab64eadac7948a7f5077260694926cc5b6e4e4b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 cc19f2389e7b595722dcc1f3296877a02b20e0a4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 a595732f04af4304974186178377192227bb80fb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 d098dba100bdfe56aa6ecafb31880098a2d7c6cb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 7fc1e43de03eac1cfe802439ba38f83988299169 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 92424a43feefc8c0db7c91302045437f3afbf274 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
8a4bdd8ed827572f5fd9d291c5454630d84284bd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 324035a8096a001d3fb170b8640805258d5e2cdd 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 5ae00af564b05ec2720a03072f94c8f8579378a7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 91405b9a334a4b031a5dc7f4a1757a3895bfb386 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d27224b23580b4662a85c874b657847ed068c9a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
 PRE-CREATION 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 b61ee81533930c889f23d2551041055cbdd1a6b2 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
 7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
 68f07e2569b6531cf3e18919209aed1a17e88bf7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 5233bee59220244e89f05b6c4dbf86a2cc6dc9fe 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 0934aeb3a7d5413cacde500a5575e4f676306bd0 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 70a17f51b9b5a9fb0b5640988318fd39a82b895d

Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-16 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/
---

Review request for hive, Alexander Kolbasov and Peter Vary.


Bugs: HIVE-20306
https://issues.apache.org/jira/browse/HIVE-20306


Repository: hive-git


Description
---

HIVE-20306 : Implement projection spec for fetching only requested fields from 
partitions


Diffs
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 0ad2a2469e0330e050fdb8983078b80617afbbf1 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AbortTxnsRequest.java
 7dcfc170cc86f237398aad1536c3fbd79561d5a9 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddCheckConstraintRequest.java
 8ece410555619626797770206e7d959cd6c31e6f 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDefaultConstraintRequest.java
 8a125d854edea766610eca8fe7a8a8f1ed3f5594 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java
 45618e781cea1a8090a75dfadb56262041e3a0cd 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddForeignKeyRequest.java
 0f443d4f60ae6b2f55ea5d1902708d005b53cac0 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddNotNullConstraintRequest.java
 0266bba2b0b9c347bc555ee7b40b4c2794e3ad41 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java
 469a9a804171321e55ace6764c2cf3355f0778a9 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java
 a00af609d45e3f4eb055e3664a376510aec926ae 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPrimaryKeyRequest.java
 9069a419ec6f6d0fba3428e556f9cd815aa714cb 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddUniqueConstraintRequest.java
 c47db4a91082df50169388b3d58c067069b790b4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AggrStats.java
 67d9b1061477e85cc71a10c4b5cc1559ac4ee1dc 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java
 5fcb98fb1b48bbff79515a984724040c5e22e602 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsResponse.java
 2a13eba708f177b26bbbd7455ce71101a33ea0b5 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java
 4d4595a429b8437e187c05cd52795f9e4885f6c4 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CheckConstraintsResponse.java
 68582058ab159126dab71e68a665a81ba9781231 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java
 3fdd295fb12240d7a94315a97419cc9a570c1536 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java
 f5c9582fa908e2d428cb897e7cc1cdd31bf45286 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java
 fd4619f0e5a9078e6e527e8f452a4d4e5da6a0b7 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CommitTxnRequest.java
 db47f9db8b094a8f34e27241e8ce33943ac90617 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionRequest.java
 dd6df744cb60f35cff5cb58e7679e4a2b8d19ffc 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CreationMetadata.java
 d631e21cec66d519e376625e6c3f41d071716834 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DefaultConstraintsResponse.java
 e29932c5abe88217414c31c81d431dfc519e2969 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsResult.java
 0a7d3b5bca893f7c673bc60fd16b655f362a33e2 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
 a128dacd898ea548b535373ffdd87d8b5398056c

Re: Review Request 68351: HIVE-20388: Move common classes out of metastore-server

2018-08-16 Thread Vihang Karajgaonkar via Review Board



> On Aug. 15, 2018, 1:12 a.m., Vihang Karajgaonkar wrote:
> > I think the DelegationToken* classes should be in the server module. For 
> > the method in SecurityUtils to get the name of TokenStore classes, we can 
> > keep it in MetastoreServerUtils class since I don't think it will be needed 
> > by MetastoreClient. Not sure if the repl*manager classes belong to common 
> > either. Can you specify why we need to move these classes in common?
> 
> Alexander Kolbasov wrote:
> There is a bunch of tangled dependencies coming from HiveMetastoreClient. 
> I suggest to keep it like this for now and then try to antangle the security 
> part and move some of it to the metastore-server server module as a follow-up 
> work.

Sounds good. I think it would be good to break up the Server and Client 
portions of the HiveAuthFactory if possible otherwise we would be leaking 
server-specific classes to client module.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68351/#review207286
---


On Aug. 15, 2018, 12:31 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68351/
> ---
> 
> (Updated Aug. 15, 2018, 12:31 a.m.)
> 
> 
> Review request for hive, Alan Gates, Peter Vary, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20388
> https://issues.apache.org/jira/browse/HIVE-20388
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20388: Move common classes out of metastore-server
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
> f746d2dff32573470ee104311d014893205894d5 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
> a50ec18b8aee0e24b632147f02c43526e0fe3071 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 
> d23e4389a770d11008d309d41780a8b692fc551a 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnCommonUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ColumnType.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  0441a33cd751e3668c43a1086adf932b0d67f169 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  324035a8096a001d3fb170b8640805258d5e2cdd 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  5ae00af564b05ec2720a03072f94c8f8579378a7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MaterializationsRebuildLockCleanerTask.java
>  9ce7d6db497079ef874eef387d86d11d893d4470 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFS.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/TableIterable.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/TimeValidator.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionListComposingSpecProxy.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecProxy.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenIdentifier.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenSecretManager.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenSelector.java
>   
>   
>

Re: Review Request 68351: HIVE-20388: Move common classes out of metastore-server

2018-08-14 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68351/#review207286
---



I think the DelegationToken* classes should be in the server module. For the 
method in SecurityUtils to get the name of TokenStore classes, we can keep it 
in MetastoreServerUtils class since I don't think it will be needed by 
MetastoreClient. Not sure if the repl*manager classes belong to common either. 
Can you specify why we need to move these classes in common?

- Vihang Karajgaonkar


On Aug. 15, 2018, 12:31 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68351/
> ---
> 
> (Updated Aug. 15, 2018, 12:31 a.m.)
> 
> 
> Review request for hive, Alan Gates, Peter Vary, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20388
> https://issues.apache.org/jira/browse/HIVE-20388
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20388: Move common classes out of metastore-server
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 
> f746d2dff32573470ee104311d014893205894d5 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
> a50ec18b8aee0e24b632147f02c43526e0fe3071 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 
> d23e4389a770d11008d309d41780a8b692fc551a 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnCommonUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ColumnType.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  0441a33cd751e3668c43a1086adf932b0d67f169 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  324035a8096a001d3fb170b8640805258d5e2cdd 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  5ae00af564b05ec2720a03072f94c8f8579378a7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MaterializationsRebuildLockCleanerTask.java
>  9ce7d6db497079ef874eef387d86d11d893d4470 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFS.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/TableIterable.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/conf/TimeValidator.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/CompositePartitionSpecProxy.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionListComposingSpecProxy.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecProxy.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/partition/spec/PartitionSpecWithSharedSDProxy.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenIdentifier.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenSecretManager.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenSelector.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/HadoopThriftAuthBridge.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/HadoopThriftAuthBridge23.java
>   
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/MetastoreDelegationTokenManager.java
>  8d84fd7f323af1de4ac994863d81e5b06be5bcce 
>   
>

Re: Review Request 67954: HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded HMS instance

2018-08-10 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67954/#review207090
---


Ship it!




Ship It!

- Vihang Karajgaonkar


On Aug. 9, 2018, 10:35 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67954/
> ---
> 
> (Updated Aug. 9, 2018, 10:35 p.m.)
> 
> 
> Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-20194
> https://issues.apache.org/jira/browse/HIVE-20194
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
> HMS instance
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  a53d4be03d695bf2176436967026757391531bc9 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  91c86a749c7afb06737c850e57f60820710c51f5 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  34055d2d4d39dc63d505a5ef95d190aa80a49d14 
> 
> 
> Diff: https://reviews.apache.org/r/67954/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 67954: HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded HMS instance

2018-07-18 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67954/#review206230
---




standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
Lines 111-116 (patched)


Do you think we can get these values from a config instead of hard coding?


- Vihang Karajgaonkar


On July 18, 2018, 12:10 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67954/
> ---
> 
> (Updated July 18, 2018, 12:10 a.m.)
> 
> 
> Review request for hive, Alan Gates, Peter Vary, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-20194
> https://issues.apache.org/jira/browse/HIVE-20194
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20194: HiveMetastoreClient should use reflection to instantiate embedded 
> HMS instance
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  47f819b60a6c7eb2acac9217724e67a976341093 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  92e280570f44cd56fe87c3123001b9263d28186f 
>   
> standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
>  d91f737450efa9de6db3806401072645c96bbb7d 
> 
> 
> Diff: https://reviews.apache.org/r/67954/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 67485: HIVE-19783 Retrieve only locations in HiveMetaStore.dropPartitionsAndGetLocations

2018-06-20 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67485/#review205084
---


Fix it, then Ship it!




Thanks a lot for the changes. Mostly looks good to me. Couple of minor 
suggestions.


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 2854 (patched)


this line should be moved to finally



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java
Line 170 (original), 170 (patched)


any particular reason to change this 2? I think it will be usefull to have 
a batch size which is not same as number of partitions so that batching logic 
is exercised.


- Vihang Karajgaonkar


On June 19, 2018, 12:23 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67485/
> ---
> 
> (Updated June 19, 2018, 12:23 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19783
> https://issues.apache.org/jira/browse/HIVE-19783
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added a new getPartitionLocations method to the RawStore interface.
> 
> Implemented getPartitionLocations in ObjectStore using JDQL.
> Question: In CachedObjectStore: Shall I call rawStore.getPartitionLocations 
> or reimplement it using getPartitions?
> 
> Modified dropPartitionsAndGetLocations:
> - Instead of querying every partition data. Query only the locations using 
> the new interface method
> - Removed partKeys parameter which become unneccessary
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  8f9a03fcd1 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  e88f9a5fee 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  e99f888eef 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  bbbdf21d4b 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  7c3588d104 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java
>  ec9e9e2b95 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  7c7429db15 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  e4f2a17d64 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/MetaStoreFactoryForTests.java
>  1a57df2680 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java
>  e1c3dcb47f 
> 
> 
> Diff: https://reviews.apache.org/r/67485/diff/4/
> 
> 
> Testing
> ---
> 
> Run the TestTablesCreateDropAlterTruncate test (partitioned table creation 
> and drop)
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 67485: HIVE-19783 Retrieve only locations in HiveMetaStore.dropPartitionsAndGetLocations

2018-06-13 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67485/#review204700
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Line 2545 (original), 2538 (patched)


My concern here is that we are removing the batch processing from this 
method. While the memory footprint of this method has reduced since we are not 
retrieving the fully loaded partition objects, I am worried that it may still 
cause OOMs for very large tables. Do you have any testing results which shows 
that this implementation is not any worse than what we already have in terms of 
the memory footprint?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 2575 (patched)


Can we avoid creating a new List here by changing the method signature to 
get a Collection instead of List?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 2784-2785 (patched)


may be a future task for improvement. We should think of ways to reduce the 
duplicate strings here. Most of the partitions locations will have the same 
prefix in the their path location. Same with the partition keys part from the 
partname. Right now, may be if we just limit the number of rows in smaller 
batches somehow would do the trick.


- Vihang Karajgaonkar


On June 7, 2018, 10:31 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67485/
> ---
> 
> (Updated June 7, 2018, 10:31 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19783
> https://issues.apache.org/jira/browse/HIVE-19783
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added a new getPartitionLocations method to the RawStore interface.
> 
> Implemented getPartitionLocations in ObjectStore using JDQL.
> Question: In CachedObjectStore: Shall I call rawStore.getPartitionLocations 
> or reimplement it using getPartitions?
> 
> Modified dropPartitionsAndGetLocations:
> - Instead of querying every partition data. Query only the locations using 
> the new interface method
> - Removed partKeys parameter which become unneccessary
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  ff97522 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  b9f5fb8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  b3a8dd0 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  f350aa9 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  d9356b8 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  8c3ada3 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  f98e8de 
> 
> 
> Diff: https://reviews.apache.org/r/67485/diff/2/
> 
> 
> Testing
> ---
> 
> Run the TestTablesCreateDropAlterTruncate test (partitioned table creation 
> and drop)
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 67351: HIVE-19718 Adding partitions in bulk also fetches table for each partition

2018-06-13 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67351/#review204697
---


Ship it!




Ship It!

- Vihang Karajgaonkar


On June 6, 2018, 11:05 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67351/
> ---
> 
> (Updated June 6, 2018, 11:05 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19718
> https://issues.apache.org/jira/browse/HIVE-19718
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Various optimization for addPartitions call:
> - Push down table object to convertToMPart
> - Push down partitionKeys to startAddPartition -> doesPartitionExist -> 
> getMPartition, so it does not have to query the table object for every time 
> if we add multiple partitions for the same table
> - The original getMPartition used to query the table every time. Created a 
> new version of getMPartition, which can use the provided partitionKeys 
> instead of querying it again.
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0cc0ae5 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  77ed2b4 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  b15d89d 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  283798c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  9da8d72 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  0461c4e 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  b71eda4 
> 
> 
> Diff: https://reviews.apache.org/r/67351/diff/3/
> 
> 
> Testing
> ---
> 
> Run several performance tests with Sasha's performance tool. These 
> optimisations shave of ~10% of the runtime
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 67351: HIVE-19718 Adding partitions in bulk also fetches table for each partition

2018-06-04 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67351/#review204262
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 2411-2416 (original)


Why do we need to remove these lines?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 2422 (original)


Why do we need to remove this line?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 2426 (patched)


Adding a javadoc would be great. esp. mentioning that the advantage of 
using this method and when its better to use it.


- Vihang Karajgaonkar


On May 29, 2018, 10:53 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67351/
> ---
> 
> (Updated May 29, 2018, 10:53 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19718
> https://issues.apache.org/jira/browse/HIVE-19718
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Various optimization for addPartitions call:
> - Push down table object to convertToMPart
> - Push down partitionKeys to startAddPartition -> doesPartitionExist -> 
> getMPartition, so it does not have to query the table object for every time 
> if we add multiple partitions for the same table
> - The original getMPartition used to query the table every time. Created a 
> new version of getMPartition, which can use the provided partitionKeys 
> instead of querying it again.
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  3d6fda6 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  c1d25db 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  13ccdb1 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  ce7d286 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  b223920 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  f6899be 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  98a85cc 
> 
> 
> Diff: https://reviews.apache.org/r/67351/diff/1/
> 
> 
> Testing
> ---
> 
> Run several performance tests with Sasha's performance tool. These 
> optimisations shave of ~10% of the runtime
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 67399: HIVE-19503 Create a test that checks for dropPartitions with directSql

2018-06-01 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/#review204195
---


Ship it!




Thanks for the changes. LGTM

- Vihang Karajgaonkar


On June 1, 2018, 10:23 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67399/
> ---
> 
> (Updated June 1, 2018, 10:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19503
> https://issues.apache.org/jira/browse/HIVE-19503
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains 2 tests:
> 
> Test which checks if the JDO cache is able to handle directSql partition drops
> Test which checks if the directSQL partition drop removes every connected 
> data from the RDBMS tables.
> To create these tests we have 2 helper methods:
> 
> Method to create the partitioned table
> Method to check the number of rows in a given RDBMS table
> Added a new ObjectStore.dropPartitionsInternal method which only visible for 
> testing so we can make sure that the dropPartition is using directSql and 
> does not fall back to JDO.
> 
> Fixed a problem where some of the tables are not created automatically by the 
> tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  5bb1985 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  b15d89d 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  9912213 
> 
> 
> Diff: https://reviews.apache.org/r/67399/diff/2/
> 
> 
> Testing
> ---
> 
> Run the new tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 67399: HIVE-19503 Create a test that checks for dropPartitions with directSql

2018-05-31 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/#review204139
---



Thanks for the tests. Can we add a one more test which repeats the same test 
accross multiple sessions (can be simulated by using objectstore instances)? So 
for example: 

1. Session 1 : list partitions with jdo, Session 2 : list partitions with jdo
2. Session 1 : drop partitions with directSQL
3. Session 2 : list partitions with jdo


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 2562 (patched)


why do we need it to be protected? Can we use package private instead?


- Vihang Karajgaonkar


On May 31, 2018, 1:53 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67399/
> ---
> 
> (Updated May 31, 2018, 1:53 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19503
> https://issues.apache.org/jira/browse/HIVE-19503
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains 2 tests:
> 
> Test which checks if the JDO cache is able to handle directSql partition drops
> Test which checks if the directSQL partition drop removes every connected 
> data from the RDBMS tables.
> To create these tests we have 2 helper methods:
> 
> Method to create the partitioned table
> Method to check the number of rows in a given RDBMS table
> Added a new ObjectStore.dropPartitionsInternal method which only visible for 
> testing so we can make sure that the dropPartition is using directSql and 
> does not fall back to JDO.
> 
> Fixed a problem where some of the tables are not created automatically by the 
> tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  5bb1985 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  b15d89d 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  9912213 
> 
> 
> Diff: https://reviews.apache.org/r/67399/diff/1/
> 
> 
> Testing
> ---
> 
> Run the new tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 66800: HIVE-6980 Drop table by using direct sql

2018-05-24 Thread Vihang Karajgaonkar via Review Board



> On May 14, 2018, 7:07 p.m., Vihang Karajgaonkar wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 2545 (patched)
> > 
> >
> > Why not LOG.error?
> 
> Peter Vary wrote:
> My original thought was, that we fall back to JDO solution this case, so 
> the problem is not fatal.
> I think this is why the other cases are using LOG.warn too.
> Shall I change here, and other places as well?
> 
> Thanks for the review!

I see, that sounds good to me then. Thanks


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66800/#review203055
---


On May 11, 2018, 2:13 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66800/
> ---
> 
> (Updated May 11, 2018, 2:13 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Marta Kuczora, Adam 
> Szita, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-6980
> https://issues.apache.org/jira/browse/HIVE-6980
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> First version of the patch.
> 
> Splits getPartitionsViaSqlFilterInternal to:
> 
> getPartitionIdsViaSqlFilter - which returns the partition ids
> getPartitionsFromPartitionIds - which returns the partition data for the 
> partitions
> Creates dropPartitionsByPartitionIds which drops the partitions by directSQL 
> commands
> 
> Creates a dropPartitionsViaSqlFilter using getPartitionIdsViaSqlFilter and 
> dropPartitionsByPartitionIds.
> 
> Modifies the ObjectStore to drop partitions with directsql if possible.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  56fbfed 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  b0a805f 
> 
> 
> Diff: https://reviews.apache.org/r/66800/diff/4/
> 
> 
> Testing
> ---
> 
> Run the TestDropPartition tests, also checked the database manually, that no 
> object left in the database
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 66800: HIVE-6980 Drop table by using direct sql

2018-05-14 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66800/#review203055
---


Fix it, then Ship it!




LGTM. Thanks for making this change. This should help a lot for dropping big 
tables!


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 2545 (patched)


Why not LOG.error?


- Vihang Karajgaonkar


On May 11, 2018, 2:13 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66800/
> ---
> 
> (Updated May 11, 2018, 2:13 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Marta Kuczora, Adam 
> Szita, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-6980
> https://issues.apache.org/jira/browse/HIVE-6980
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> First version of the patch.
> 
> Splits getPartitionsViaSqlFilterInternal to:
> 
> getPartitionIdsViaSqlFilter - which returns the partition ids
> getPartitionsFromPartitionIds - which returns the partition data for the 
> partitions
> Creates dropPartitionsByPartitionIds which drops the partitions by directSQL 
> commands
> 
> Creates a dropPartitionsViaSqlFilter using getPartitionIdsViaSqlFilter and 
> dropPartitionsByPartitionIds.
> 
> Modifies the ObjectStore to drop partitions with directsql if possible.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  56fbfed 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  b0a805f 
> 
> 
> Diff: https://reviews.apache.org/r/66800/diff/4/
> 
> 
> Testing
> ---
> 
> Run the TestDropPartition tests, also checked the database manually, that no 
> object left in the database
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 67074: HIVE-19493 : VectorUDFDateDiffColCol copySelected does not handle nulls correctly

2018-05-10 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67074/
---

(Updated May 11, 2018, 12:05 a.m.)


Review request for hive and Matt McCline.


Changes
---

added review comment from Matt


Bugs: HIVE-19493
https://issues.apache.org/jira/browse/HIVE-19493


Repository: hive-git


Description
---

HIVE-19493 : VectorUDFDateDiffColCol copySelected does not handle nulls 
correctly


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateDiffColCol.java
 84ee94432b48b4e9fdec39e72b90809cfe67a950 


Diff: https://reviews.apache.org/r/67074/diff/2/

Changes: https://reviews.apache.org/r/67074/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Review Request 67074: HIVE-19493 : VectorUDFDateDiffColCol copySelected does not handle nulls correctly

2018-05-10 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67074/
---

Review request for hive and Matt McCline.


Bugs: HIVE-19493
https://issues.apache.org/jira/browse/HIVE-19493


Repository: hive-git


Description
---

HIVE-19493 : VectorUDFDateDiffColCol copySelected does not handle nulls 
correctly


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorUDFDateDiffColCol.java
 84ee94432b48b4e9fdec39e72b90809cfe67a950 


Diff: https://reviews.apache.org/r/67074/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 66979: HIVE-19374: Parse and process ALTER TABLE SET OWNER command syntax

2018-05-09 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66979/#review202771
---


Fix it, then Ship it!




LGTM. Thanks for the changes.


itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/metadata/TestAlterTableMetadata.java
Lines 1 (patched)


Apache header missing


- Vihang Karajgaonkar


On May 8, 2018, 11:57 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66979/
> ---
> 
> (Updated May 8, 2018, 11:57 p.m.)
> 
> 
> Review request for hive and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19374
> https://issues.apache.org/jira/browse/HIVE-19374
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch implements the new ALTER SET ... SET OWNER command and calls the 
> HMS api calls to change the owner of the table.
> 
> The command syntax is:
> > ALTER TABLE  SET OWNER { USER |GROUP |ROLE 
> >  }
> 
> Currently, Hive sets the owner of a table to the user who created that table. 
> With this command, we will be able to change it to another user, group of 
> role (as ALTER DATABASE does).
> 
> The changes are:
> - HiveParser.g which adds the new syntax
> - HiveOperation.java which adds the new ALTERTABLE_OWNER operation
> - Table.java which gets/sets the owner type
> - SemanticAnalyzer.java which returns the DDLSemanticAnalyzer if an 
> ALTERTABLE_OWNER operation is detected
> - DDLSemanticAnalyzer.java which analyzes the ALTERTABLE_OWNER Operation
> - AlterTableDesc.java uses by the DDL semantic analyzer to change the new 
> owner information
> - MetaDataFormatUtils which displays the owner type when the DESCRIBE command 
> is called
> - JsonMetaDataFormatted which is another implementation to display the owner 
> type in Json format
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/metadata/TestAlterTableMetadata.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 3141a7e981eb35a9fbc7f367f38f8ad420f1f928 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 
> 879b4224494c3a9adb0713f319e586db4865fb17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  cd70eee26c06ee6476964508c54c2bb10b167530 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
>  af283e693b5a0fc68e35221b2005fcf1910bdb8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> defb8becdb5d767ae71d5c962afac43f0c068c3c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
> a837d67b9615ca1ee359c7aa26f79b6f2504dd99 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
> 820046388adbc65664ae36b08aaba72943ccb6af 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 
> a767796a949da3c23ebe6d8c78b995c8638ebfef 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 
> cd4c206a89f1bc1a6195b0f1f39d3c4b462dc027 
> 
> 
> Diff: https://reviews.apache.org/r/66979/diff/2/
> 
> 
> Testing
> ---
> 
> Waiting for HiveQA
> 
> - alter_table_set_owner.q which verifies that the new command works. Describe 
> is not tested because the .q tests files mask the owner information.
> - the describe command verified manually in my local hive environment
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 66979: HIVE-19374: Parse and process ALTER TABLE SET OWNER command syntax

2018-05-08 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66979/#review202679
---



Overall looks good. Would be great if we could have a junit test to confirm the 
owner value is same as the one being set from the alter table command to catch 
regressions.


ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
Lines 217 (patched)


do we need to check for null for tbl.getOwnerType()?



ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
Lines 428 (patched)


can ownerType be null?


- Vihang Karajgaonkar


On May 7, 2018, 2:36 a.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66979/
> ---
> 
> (Updated May 7, 2018, 2:36 a.m.)
> 
> 
> Review request for hive and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19374
> https://issues.apache.org/jira/browse/HIVE-19374
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch implements the new ALTER SET ... SET OWNER command and calls the 
> HMS api calls to change the owner of the table.
> 
> The command syntax is:
> > ALTER TABLE  SET OWNER { USER |GROUP |ROLE 
> >  }
> 
> Currently, Hive sets the owner of a table to the user who created that table. 
> With this command, we will be able to change it to another user, group of 
> role (as ALTER DATABASE does).
> 
> The changes are:
> - HiveParser.g which adds the new syntax
> - HiveOperation.java which adds the new ALTERTABLE_OWNER operation
> - Table.java which gets/sets the owner type
> - SemanticAnalyzer.java which returns the DDLSemanticAnalyzer if an 
> ALTERTABLE_OWNER operation is detected
> - DDLSemanticAnalyzer.java which analyzes the ALTERTABLE_OWNER Operation
> - AlterTableDesc.java uses by the DDL semantic analyzer to change the new 
> owner information
> - MetaDataFormatUtils which displays the owner type when the DESCRIBE command 
> is called
> - JsonMetaDataFormatted which is another implementation to display the owner 
> type in Json format
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 3141a7e981eb35a9fbc7f367f38f8ad420f1f928 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 
> 879b4224494c3a9adb0713f319e586db4865fb17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
>  cd70eee26c06ee6476964508c54c2bb10b167530 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
>  af283e693b5a0fc68e35221b2005fcf1910bdb8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> defb8becdb5d767ae71d5c962afac43f0c068c3c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
> a837d67b9615ca1ee359c7aa26f79b6f2504dd99 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
> 820046388adbc65664ae36b08aaba72943ccb6af 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 
> a767796a949da3c23ebe6d8c78b995c8638ebfef 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 
> cd4c206a89f1bc1a6195b0f1f39d3c4b462dc027 
>   ql/src/test/queries/clientpositive/alter_table_set_owner.q PRE-CREATION 
>   ql/src/test/results/clientpositive/alter_table_set_owner.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66979/diff/1/
> 
> 
> Testing
> ---
> 
> Waiting for HiveQA
> 
> - alter_table_set_owner.q which verifies that the new command works. Describe 
> is not tested because the .q tests files mask the owner information.
> - the describe command verified manually in my local hive environment
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 66890: HIVE-19371: Add table ownerType to HMS thrift API

2018-05-02 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66890/#review202309
---


Ship it!




- Vihang Karajgaonkar


On May 1, 2018, 8:30 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66890/
> ---
> 
> (Updated May 1, 2018, 8:30 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-19371
> https://issues.apache.org/jira/browse/HIVE-19371
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a subtask part of 'HIVE-18762 Support ALTER TABLE SET OWNER command' 
> which adds a new ownerType field on the Table object of the Thrift API of HMS.
> 
> The only file updated (before generting the thrift code) is:
> - hive_metastore.thrift
> 
> The new field is added as 'optional' at the end of the 'struct Table' in 
> order to be backward compatible with older clients of HMS.
> The ownerType filed will be set to a USER as the default value as current 
> tables created are using the owner name as the user.
> 
> 
> Diffs
> -
> 
>   standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 
> 47877034746a1cf81cc6731bedb8a6da9cee5b6d 
>   standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h 
> 629889389e29dcbf9802c348f3b73f6c65695f6f 
>   standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 
> c6fadf8a24a32bb208ae6819f159b9b4d5301a2b 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
>  8dfec980d9e14ad01fda27ad97e2e6a91846f782 
>   standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php 
> 9f6cc0e871754afc4fcdbdfb0907d78a0298634c 
>   standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 
> 69579e2f593d35c4d52ae7c81d5186d0ae3d379a 
>   standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb 
> d7ebaaf914997371201a123f3fc28fac63464ce3 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 
> ccc3c93bcec88b96e524eeb6d47b7c77ccf49d34 
> 
> 
> Diff: https://reviews.apache.org/r/66890/diff/1/
> 
> 
> Testing
> ---
> 
> Waiting for Hive QA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 66890: HIVE-19371: Add table ownerType to HMS thrift API

2018-05-02 Thread Vihang Karajgaonkar via Review Board



> On May 2, 2018, 5:49 p.m., Vihang Karajgaonkar wrote:
> > standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
> > Lines 1071 (patched)
> > 
> >
> > Is it possible to write a test to make sure this works as expected?
> 
> Sergio Pena wrote:
> This is Ruby on Rails code. I don't know how to create a test to verify 
> Ruby.
> Should we verify it?
> 
> Btw, there are other tests in a follow-up patch which implements this 
> Thrift API on the ObjectStore.
> 
> See https://reviews.apache.org/r/66909/ (needs review too)

Oh, this is in the auto-generated code. Sorry, I missed that. Dropping this 
comment.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66890/#review202291
---


On May 1, 2018, 8:30 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66890/
> ---
> 
> (Updated May 1, 2018, 8:30 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-19371
> https://issues.apache.org/jira/browse/HIVE-19371
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a subtask part of 'HIVE-18762 Support ALTER TABLE SET OWNER command' 
> which adds a new ownerType field on the Table object of the Thrift API of HMS.
> 
> The only file updated (before generting the thrift code) is:
> - hive_metastore.thrift
> 
> The new field is added as 'optional' at the end of the 'struct Table' in 
> order to be backward compatible with older clients of HMS.
> The ownerType filed will be set to a USER as the default value as current 
> tables created are using the owner name as the user.
> 
> 
> Diffs
> -
> 
>   standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 
> 47877034746a1cf81cc6731bedb8a6da9cee5b6d 
>   standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h 
> 629889389e29dcbf9802c348f3b73f6c65695f6f 
>   standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 
> c6fadf8a24a32bb208ae6819f159b9b4d5301a2b 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
>  8dfec980d9e14ad01fda27ad97e2e6a91846f782 
>   standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php 
> 9f6cc0e871754afc4fcdbdfb0907d78a0298634c 
>   standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 
> 69579e2f593d35c4d52ae7c81d5186d0ae3d379a 
>   standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb 
> d7ebaaf914997371201a123f3fc28fac63464ce3 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 
> ccc3c93bcec88b96e524eeb6d47b7c77ccf49d34 
> 
> 
> Diff: https://reviews.apache.org/r/66890/diff/1/
> 
> 
> Testing
> ---
> 
> Waiting for Hive QA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 66890: HIVE-19371: Add table ownerType to HMS thrift API

2018-05-02 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66890/#review202291
---




standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
Lines 1071 (patched)


Is it possible to write a test to make sure this works as expected?


- Vihang Karajgaonkar


On May 1, 2018, 8:30 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66890/
> ---
> 
> (Updated May 1, 2018, 8:30 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-19371
> https://issues.apache.org/jira/browse/HIVE-19371
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a subtask part of 'HIVE-18762 Support ALTER TABLE SET OWNER command' 
> which adds a new ownerType field on the Table object of the Thrift API of HMS.
> 
> The only file updated (before generting the thrift code) is:
> - hive_metastore.thrift
> 
> The new field is added as 'optional' at the end of the 'struct Table' in 
> order to be backward compatible with older clients of HMS.
> The ownerType filed will be set to a USER as the default value as current 
> tables created are using the owner name as the user.
> 
> 
> Diffs
> -
> 
>   standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 
> 47877034746a1cf81cc6731bedb8a6da9cee5b6d 
>   standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h 
> 629889389e29dcbf9802c348f3b73f6c65695f6f 
>   standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 
> c6fadf8a24a32bb208ae6819f159b9b4d5301a2b 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
>  8dfec980d9e14ad01fda27ad97e2e6a91846f782 
>   standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php 
> 9f6cc0e871754afc4fcdbdfb0907d78a0298634c 
>   standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 
> 69579e2f593d35c4d52ae7c81d5186d0ae3d379a 
>   standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb 
> d7ebaaf914997371201a123f3fc28fac63464ce3 
>   standalone-metastore/src/main/thrift/hive_metastore.thrift 
> ccc3c93bcec88b96e524eeb6d47b7c77ccf49d34 
> 
> 
> Diff: https://reviews.apache.org/r/66890/diff/1/
> 
> 
> Testing
> ---
> 
> Waiting for Hive QA.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 65985: HIVE-18783: ALTER TABLE post-commit listener does not include the transactional listener responses

2018-04-04 Thread Vihang Karajgaonkar via Review Board



> On April 4, 2018, 12:34 a.m., Vihang Karajgaonkar wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
> > Lines 408 (patched)
> > 
> >
> > why do we need !success check here?
> 
> Sergio Pena wrote:
> Ah? Thanks, I missed this. I removed the first condition and leave this 
> one instead.

I thought we need the first condition and not the second condition for 
!success. Are we not generating the events only when the transaction is 
successful? Am I missing something?


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65985/#review200405
---


On April 4, 2018, 4:06 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65985/
> ---
> 
> (Updated April 4, 2018, 4:06 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-18783
> https://issues.apache.org/jira/browse/HIVE-18783
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16164 introduced a mechanism to pass HMS notification events ID to the 
> post-commit listeners for all DDL operations, but it didn't add it to the 
> ALTER TABLE event. This patch in review adds the same behavior for ALTER 
> TABLE events.
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  5459554fec911b253583df1f528e5abf134f055b 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  ed1b8c5cc2a4cb974dd37419c6b5f0601a035232 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  8539fea42fa2743381833ab3137579caeac64672 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IHMSHandler.java
>  f59f40bc33367cff7a8d0d24d0e200b16c2c30e5 
> 
> 
> Diff: https://reviews.apache.org/r/65985/diff/3/
> 
> 
> Testing
> ---
> 
> All tests passed.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 65985: HIVE-18783: ALTER TABLE post-commit listener does not include the transactional listener responses

2018-04-03 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65985/#review200405
---


Fix it, then Ship it!




LGTM. Just one minor comment below.


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
Lines 408 (patched)


why do we need !success check here?


- Vihang Karajgaonkar


On March 15, 2018, 5:45 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65985/
> ---
> 
> (Updated March 15, 2018, 5:45 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-18783
> https://issues.apache.org/jira/browse/HIVE-18783
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16164 introduced a mechanism to pass HMS notification events ID to the 
> post-commit listeners for all DDL operations, but it didn't add it to the 
> ALTER TABLE event. This patch in review adds the same behavior for ALTER 
> TABLE events.
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  e0e29652da94bbdaca515a17955d1409824c1742 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  0dd3eb101709969a77998e1488e1c97214426cd3 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  66353e769b2d633ad9dfad2bcae25e8ad90f61d1 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IHMSHandler.java
>  e6de0013bc1be12b2772e2e97102ed476cf5 
> 
> 
> Diff: https://reviews.apache.org/r/65985/diff/2/
> 
> 
> Testing
> ---
> 
> All tests passed.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>

Re: Review Request 66185: JDBC: Provide an option to simplify beeline usage by supporting default and named URL for beeline

2018-03-30 Thread Vihang Karajgaonkar via Review Board

> On March 29, 2018, 11:32 p.m., Vihang Karajgaonkar wrote:
> > beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParser.java
> > Lines 79 (patched)
> > 
> >
> > Most of the code in this file is duplicated from 
> > UserHS2ConnectionFileParser. This logic would work for what you are trying 
> > to achieve but I think we are doing this in much more complex way than 
> > necessary. Here is my suggestion:
> > 
> > Based on my understanding you have following requirements:
> > 1. User will provide a named url like eg blue using beeline -c blue
> > 2. If the user doesn't provide a named url a default url is used.
> > 
> > You can do this with a small change in the existing code and without 
> > the need to merge the two connection files. In the example above, if the 
> > user doesn't specify -c option above but the beeline-hs2-connection.xml 
> > exists the code is same as what we have currently. (We can obviously 
> > support beeline-site.xml name as well with a trivial change)
> > 
> > if the user specifies beeline -c blue instead, in 
> > UserHS2ConnectionFileParser.getConnectionProperties() instead of looking 
> > for keys starting for beeline.hs2.connection. you can modify the code to 
> > look for keys starting from beeline.hs2.connection.blue. In that case you 
> > reuse most of the existing code and don't need the additional merging 
> > logic. The result is you have only one user connection file and one 
> > hive-site.xml which can be overriden by user connection file.
> > 
> > The only compromise in the above design is that you cannot provide one 
> > long url string like you want to do in beeline-site.xml. If this is not a 
> > hard requirement, I think we will have a much cleaner implementation with 
> > less confusion between having two connection xml files which behave almost 
> > the same way. If its a hard requirement, I guess you can still do by 
> > defining a new key like you are doing currently for beeline-site.xml
> > 
> > Is there a use-case which will not work in my suggestion above? May be 
> > I am missing something very obvious. Would appreciate if you could help me 
> > undestand. Thanks!
> 
> Thejas Nair wrote:
> Vihang, here are the requirements -
> 
> * Beeline can be used in hosts that are not part of the hadoop cluster. 
> hive-site.xml might not be available there. Also, editing the large 
> hive-site.xml (if user is not in the cluster) is going to be complicated for 
> users. So we don't want to rely on hive-site.xml.
> * Having the complete url as a config param in beeline-site.xml makes 
> easy to manage. You don't have to edit multiple parameters to add a new named 
> URL to it. Ambari provides full HS2 jdbc URL that is easy to copy and then 
> add to this file (when not managed by Ambari).
> * If the host is part of a hadoop cluster, then beeline-site.xml can be 
> added by Ambari (or equivalent). However, there are client environment 
> related parameters that user might want to set in addition to that (SSL 
> truststore for example). In that case you don't want the user to modify 
> beeline-site.xml directly as it would get overwritten. So having a second 
> user override .xml file helps users to manage those settings seperately.

Thanks for the clarifying the details. Would appreciate if can you also clarify 
the below questions too.
How do you handle the case where there are multiple HS2 instances on the 
cluster? Eg. one with Hive 2.3.x and one with Hive 3.0.0?

Regarding the third point:
Seems like the order of precendence is beeline-site.xml (first of 
$HOME/.beeline/beeline-site.xml, $HIVE_CONF_DIR/beeline-site.xml and 
/etc/hive/conf/beeline-site.xml) which gets overridden by 
beeline-hs2-connection.xml (first of $HOME/.beeline/beeline-site.xml, 
$HIVE_CONF_DIR/beeline-site.xml, /etc/hive/conf/beeline-site.xml) which gets 
merged into hive-site.xml if available.
I agree with the usecase where user might want to override auto-generated file. 
Do you think it makes sense to have two different xml file names to do that? 
Isn't it confusing? Why not make it such that user-specific file in 
$HOME/.beeline/beeline-site.xml overrides the autogenerated 
$HIVE_CONF_DIR/beeline-site.xml or /etc/hive/conf/beeline-site.xml? if we do 
that we can get rid of one additional filename altogether. I am okay if you 
want to get rid of beeline-hs2-connection.xml if you follow the above approach.

Please consider this a non-blocking suggestion so feel free to go ahead with 
the commit if you don't think the above is a good idea. I can create a 
follow-up JIRA separately and try to reconcile all these requirements by 
removing some of the redundant stuff. Thanks!

- Vihang

---
This is an automatically generated e-mail. To reply, visit:

Re: Review Request 66185: JDBC: Provide an option to simplify beeline usage by supporting default and named URL for beeline

2018-03-29 Thread Vihang Karajgaonkar via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66185/#review200215
---

beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParser.java
Lines 34-37 (patched)

Can you update the this javadoc?

beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParser.java
Lines 79 (patched)

Most of the code in this file is duplicated from 
UserHS2ConnectionFileParser. This logic would work for what you are trying to 
achieve but I think we are doing this in much more complex way than necessary. 
Here is my suggestion:

Based on my understanding you have following requirements:
1. User will provide a named url like eg blue using beeline -c blue
2. If the user doesn't provide a named url a default url is used.

You can do this with a small change in the existing code and without the 
need to merge the two connection files. In the example above, if the user 
doesn't specify -c option above but the beeline-hs2-connection.xml exists the 
code is same as what we have currently. (We can obviously support 
beeline-site.xml name as well with a trivial change)

if the user specifies beeline -c blue instead, in 
UserHS2ConnectionFileParser.getConnectionProperties() instead of looking for 
keys starting for beeline.hs2.connection. you can modify the code to look for 
keys starting from beeline.hs2.connection.blue. In that case you reuse most of 
the existing code and don't need the additional merging logic. The result is 
you have only one user connection file and one hive-site.xml which can be 
overriden by user connection file.

The only compromise in the above design is that you cannot provide one long 
url string like you want to do in beeline-site.xml. If this is not a hard 
requirement, I think we will have a much cleaner implementation with less 
confusion between having two connection xml files which behave almost the same 
way. If its a hard requirement, I guess you can still do by defining a new key 
like you are doing currently for beeline-site.xml

Is there a use-case which will not work in my suggestion above? May be I am 
missing something very obvious. Would appreciate if you could help me 
undestand. Thanks!

- Vihang Karajgaonkar

On March 28, 2018, 8:27 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66185/
> ---
> 
> (Updated March 28, 2018, 8:27 p.m.)
> 
> 
> Review request for hive, Thejas Nair and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18963
> https://issues.apache.org/jira/browse/HIVE-18963
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18963
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 402fae 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineConfFileParseException.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineHS2ConnectionFileParseException.java
>  acddf82a67 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParseException.java
>  PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/BeelineSiteParser.java 
> PRE-CREATION 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
>  b769e8581f 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileUtils.java
>  f635b40633 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
>  2801ebee09 
>   beeline/src/main/resources/BeeLine.properties 6fca953836 
>   
> beeline/src/test/org/apache/hive/beeline/hs2connection/TestUserHS2ConnectionFileParser.java
>  1d17887417 
>   beeline/src/test/resources/test-hs2-named-connection-config.xml 
> PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/BeelineWithHS2ConnectionFileTestBase.java
>  3da31ad8a9 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 6d7787da7d 
> 
> 
> Diff: https://reviews.apache.org/r/66185/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 66288: HIVE-18885 : DbNotificationListener has a deadlock between Java and DB locks (2.x line)

2018-03-26 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66288/
---

(Updated March 26, 2018, 10:14 p.m.)


Review request for hive and Alexander Kolbasov.


Changes
---

added Alexander's suggestions.


Bugs: HIVE-18885
https://issues.apache.org/jira/browse/HIVE-18885


Repository: hive-git


Description
---

HIVE-18885 : DbNotificationListener has a deadlock between Java and DB locks 
(2.x line)


Diffs (updated)
-

  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 41347c22df21a678241edeb766264e6d19c7885a 


Diff: https://reviews.apache.org/r/66288/diff/2/

Changes: https://reviews.apache.org/r/66288/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Review Request 66288: HIVE-18885 : DbNotificationListener has a deadlock between Java and DB locks (2.x line)

2018-03-26 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66288/
---

Review request for hive and Alexander Kolbasov.


Repository: hive-git


Description
---

HIVE-18885 : DbNotificationListener has a deadlock between Java and DB locks 
(2.x line)


Diffs
-

  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 41347c22df21a678241edeb766264e6d19c7885a 


Diff: https://reviews.apache.org/r/66288/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 66185: JDBC: Provide an option to simplify beeline usage by supporting default and named URL for beeline

2018-03-22 Thread Vihang Karajgaonkar via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66185/#review199770
---

I am a bit confused here. If the full url can be provided in the config file by 
the user, how is it better than just creating a environment variable like 
BEELINE_URL_ and use it instead of adding it in the config file? I think 
the objective of this config file was to automatically figure out the 
connection url based on hive-site.xml and the additional 
beeline-hs2-connection.xml to override/augment the information from 
hive-site.xml

The current code is structured such that all keys start with 
beeline.hs2.connection. and components of the url are parsed automatically 
using the values of those keys. If we want to add full support of named urls 
which can have completely different url components like session vars etc, what 
do you think of adding a new prefix key of the form 
beeline.hs2.connection. and then the existing code will work exactly like 
it does currently but instead will parse the keys starting with 
beeline.hs2.connection.. For example, a named url called "blue" will be 
constructed using all the keys from beeline.hs2.connection.blue. That way we 
reuse existing logic. The beeline will be invoked like beeline -c blue. Do you 
see any problems with this approach? This way the user doesn't have to provide 
all the url components which can be reused from hive-site.xml (like the nasty 
ssl, kerberos settings)

- Vihang Karajgaonkar

On March 20, 2018, 10:54 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66185/
> ---
> 
> (Updated March 20, 2018, 10:54 p.m.)
> 
> 
> Review request for hive, Thejas Nair and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18963
> https://issues.apache.org/jira/browse/HIVE-18963
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18963
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 402fae 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
>  b769e8581f 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileUtils.java
>  f635b40633 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
>  2801ebee09 
>   beeline/src/main/resources/BeeLine.properties 6fca953836 
>   
> beeline/src/test/org/apache/hive/beeline/hs2connection/TestUserHS2ConnectionFileParser.java
>  1d17887417 
>   beeline/src/test/resources/test-hs2-named-connection-config.xml 
> PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/BeelineWithHS2ConnectionFileTestBase.java
>  3da31ad8a9 
> 
> 
> Diff: https://reviews.apache.org/r/66185/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Review Request 66068: [HIVE-16882] Improvements For Avro SerDe Package

2018-03-14 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66068/#review199185
---


Fix it, then Ship it!




LGTM


serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java
Line 29 (original), 29 (patched)


Didn't the author really mean memoizes 
https://en.wikipedia.org/wiki/Memoization


- Vihang Karajgaonkar


On March 14, 2018, 4:10 p.m., David Mollitor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66068/
> ---
> 
> (Updated March 14, 2018, 4:10 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> [HIVE-16882] Improvements For Avro SerDe Package
> 
> 
> Diffs
> -
> 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 
> b7b3d12 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroLazyObjectInspector.java
>  e1e4d8b 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerDe.java 1746a0f 
>   serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java 
> 2d52020 
> 
> 
> Diff: https://reviews.apache.org/r/66068/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> David Mollitor
> 
>

Re: Review Request 65746: HIVE-18768: Use Datanucleus to serialize notification updates

2018-02-22 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65746/#review198134
---


Ship it!




Patch LGTM

- Vihang Karajgaonkar


On Feb. 22, 2018, 4:22 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65746/
> ---
> 
> (Updated Feb. 22, 2018, 4:22 a.m.)
> 
> 
> Review request for hive, Aihua Xu, anishek, Andrew Sherman, Janaki Lahorani, 
> Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18768
> https://issues.apache.org/jira/browse/HIVE-18768
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18768: Use Datanucleus to serialize notification updates
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  7b44df41286d0356fda97914ce7460fc38518b9f 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  f9bd49574a75c813346d787767126902c192d48b 
> 
> 
> Diff: https://reviews.apache.org/r/65746/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

2018-02-13 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review197464
---


Ship it!




Thanks for the changes Ferdinand. Patch looks good to me.

- Vihang Karajgaonkar


On Feb. 14, 2018, 1:51 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 14, 2018, 1:51 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java
>  c36640d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java
>  39689f1 
>   ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/package-info.java 
> PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedDictionaryEncodingColumnReader.java
>  3e5d831 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   
> ql/src/test/queries/clientpositive/schema_evol_par_vec_table_dictionary_encoding.q
>  PRE-CREATION 
>   
> ql/src/test/queries/clientpositive/schema_evol_par_vec_table_non_dictionary_encoding.q
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/schema_evol_par_vec_table_dictionary_encoding.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/schema_evol_par_vec_table_non_dictionary_encoding.q.out
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/8/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 65500: HIVE-18421 : Vectorized execution handles overflows in a different manner than non-vectorized execution

2018-02-12 Thread Vihang Karajgaonkar via Review Board



> On Feb. 8, 2018, 3:51 a.m., Aihua Xu wrote:
> > ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt
> > Lines 131 (patched)
> > 
> >
> > Can we remove #ELSE from here and some other places below? Seems not 
> > needed.

The #ELSE is needed since it is used for the generating the unchecked variants 
of the vector expressions.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/#review197073
---


On Feb. 7, 2018, 7:13 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65500/
> ---
> 
> (Updated Feb. 7, 2018, 7:13 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.
> 
> 
> Bugs: HIVE-18421
> https://issues.apache.org/jira/browse/HIVE-18421
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 99e8457c7b2d506cfc7c71ca18bc678fe5cdf049 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
> b5011c3adcedf8974d3241994733e0021a851cbd 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
> cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
> 3e955578933dd7990939865527c3bd11023b3a90 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
> f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
>   ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
> e95baa6199e138a4e0c009e62ce495b626e5909c 
>   ql/src/gen/vectorization/TestTemplates/TestClass.txt 
> 62c58fb293fbe2d4d948c6a3409ee31466424a02 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
>  bbe78c8720e16163b642f54d27fdf6b65ba9850b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 8264e8ad285deac29424bd1cb0bf626436d47c75 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
>  75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
>  6b4d714c9a79a55593c4a4d254267a3035abb10f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
>  710165033627b33d9b238cc847dbac36c07ee5f6 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
> af8552caa02f2896f393a5099abdb1ae5abd4c16 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
> e2a638da518a2071ff15b8da6899646ec45c832a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
> 99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
> 4e45788936559bbb7cfe65e9ffd083747b37dcc2 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
> b1200e673e6b470b5fd1cc856270a6da615f16cb 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
>  ab6f6b79316818cac458390dc2d087091057c63b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
>  02dec659ce421eef06f924bb6973070878d57be3 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorMathFunctions.java
>  1950e92bd6d5e4f818588e691db30cd28193c716 
>   ql/src/test/queries/clientpositive/vectorization_numeric_overflows.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/vectorization_numeric_overflows.q.out 
> PRE-CREATION 
>   vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
> 657ea34e11f7465e6c77d45128b298e7326a057b 
>   vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorTestCode.java 
> d97646f8b1c4a074da59b4685939fc4359c9c30d 
> 
> 
> Diff: https://reviews.apache.org/r/65500/diff/6/
> 
> 
> Testing
> ---
> 
> HiveQA
> 
> 
> Thanks,
> 
> Vihang Karajgaonkar
> 
>

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

2018-02-12 Thread Vihang Karajgaonkar via Review Board



> On Feb. 13, 2018, 12:50 a.m., Vihang Karajgaonkar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java
> > Lines 253-254 (original), 255-256 (patched)
> > 
> >
> > before the patch, the code assumes that the ColumnReader's type matches 
> > with what the hive's type is. So these two lines made sense. But after the 
> > patch, it is possible that the underlying parquet type does not have 
> > decimalMetadata so it these lines will throw a NPE. Instead of throwing 
> > NPE, we should check if decimalMetadata is set and throw IOException here 
> > saying that underlying type cannot be converted to decimal. What do you 
> > think?
> > 
> > Also, may be we should add a TODO here to use default precision and 
> > scale if its not available for the future.
> 
> cheng xu wrote:
> Why do we need the default precision and scale here? We can hardly read 
> data as decimal if those values are missing.

hmm .. I am not a 100% sure but I thought HiveDecimal defines something like 
USER_DEFAULT_PRECISION and USER_DEFAULT_SCALE which is used when user doesn't 
provide scale and precision. This is not important for this change so you can 
ignore the comment related to default precision and scale. But I think we 
should protect against a NPE which will be thrown if the user tries to change 
say an int column to a decimal(10,2) (I know it doesn't work anyways, but I 
think it would be better to throw an exception rather than failing with a NPE)


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review197313
---


On Feb. 12, 2018, 8:25 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 12, 2018, 8:25 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java
>  c36640d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java
>  39689f1 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedDictionaryEncodingColumnReader.java
>  3e5d831 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   
> ql/src/test/queries/clientpositive/schema_evol_par_vec_table_dictionary_encoding.q
>  PRE-CREATION 
>   
> ql/src/test/queries/clientpositive/schema_evol_par_vec_table_non_dictionary_encoding.q
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/schema_evol_par_vec_table_dictionary_encoding.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/schema_evol_par_vec_table_non_dictionary_encoding.q.out
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/4/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

2018-02-12 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review197313
---


Fix it, then Ship it!




Overall the patch looks good to me. Some comments below.


ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
Line 65 (original), 69 (patched)


can we rename this member to something more easily understandable like 
dictionaryBasedReader? The name of variable suggests its Dictionary but in fact 
its a DataColumnReader



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
Lines 57 (patched)


nit, some of the methods are missing the @Override



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java
Line 156 (original), 159 (patched)


why do we need to remove this check and return 1 or 0?



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java
Line 175 (original), 179 (patched)


this method has duplicate code, from 
VectorizedPrimitiveColumnReader#decodeDictionaryIds. Can we refactor it to 
remove duplicate code. I think its okay if we do it in a separate change.



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java
Lines 89 (patched)


previously, this reader didn't support INTERVAL_DAY_TIME. Does this mean 
that it supports this hive type? If yes, shouldn't this change in 
decodeDictionaryIds method too?



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java
Lines 253-254 (original), 255-256 (patched)


before the patch, the code assumes that the ColumnReader's type matches 
with what the hive's type is. So these two lines made sense. But after the 
patch, it is possible that the underlying parquet type does not have 
decimalMetadata so it these lines will throw a NPE. Instead of throwing NPE, we 
should check if decimalMetadata is set and throw IOException here saying that 
underlying type cannot be converted to decimal. What do you think?

Also, may be we should add a TODO here to use default precision and scale 
if its not available for the future.


- Vihang Karajgaonkar


On Feb. 12, 2018, 8:25 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 12, 2018, 8:25 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java
>  c36640d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java
>  39689f1 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedDictionaryEncodingColumnReader.java
>  3e5d831 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   
> ql/src/test/queries/clientpositive/schema_evol_par_vec_table_dictionary_encoding.q
>  PRE-CREATION 
>   
> ql/src/test/queries/clientpositive/schema_evol_par_vec_table_non_dictionary_encoding.q
>  PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/schema_evol_par_vec_table_dictionary_encoding.q.out
>  PRE-CREATION 
>   
> ql/src/test/results/clientpositive/schema_evol_par_vec_table_non_dictionary_encoding.q.out
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/4/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 65500: HIVE-18421 : Vectorized execution handles overflows in a different manner than non-vectorized execution

2018-02-06 Thread Vihang Karajgaonkar via Review Board



> On Feb. 7, 2018, 2:40 a.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
> > Lines 46-48 (patched)
> > 
> >
> > is there a way to do this check just once rather than for each value?

Made it slightly better by removing the need for null check and adding method 
call in getTypeName(). You would still need a switch on the typename though. 
Having a couple of booleans can get rid of switch but in terms of number of 
comparisons it would still be at the same cost (in fact switch is better in 
this case I believe)


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/#review196967
---


On Feb. 7, 2018, 7:13 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65500/
> ---
> 
> (Updated Feb. 7, 2018, 7:13 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.
> 
> 
> Bugs: HIVE-18421
> https://issues.apache.org/jira/browse/HIVE-18421
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 99e8457c7b2d506cfc7c71ca18bc678fe5cdf049 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
> b5011c3adcedf8974d3241994733e0021a851cbd 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
> cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
> 3e955578933dd7990939865527c3bd11023b3a90 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
> f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
>   ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
> e95baa6199e138a4e0c009e62ce495b626e5909c 
>   ql/src/gen/vectorization/TestTemplates/TestClass.txt 
> 62c58fb293fbe2d4d948c6a3409ee31466424a02 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
>  bbe78c8720e16163b642f54d27fdf6b65ba9850b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 8264e8ad285deac29424bd1cb0bf626436d47c75 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
>  75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
>  6b4d714c9a79a55593c4a4d254267a3035abb10f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
>  710165033627b33d9b238cc847dbac36c07ee5f6 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
> af8552caa02f2896f393a5099abdb1ae5abd4c16 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
> e2a638da518a2071ff15b8da6899646ec45c832a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
> 99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
> 4e45788936559bbb7cfe65e9ffd083747b37dcc2 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
> b1200e673e6b470b5fd1cc856270a6da615f16cb 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
>  ab6f6b79316818cac458390dc2d087091057c63b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
>  02dec659ce421eef06f924bb6973070878d57be3 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorMathFunctions.java
>  1950e92bd6d5e4f818588e691db30cd28193c716 
>   ql/src/test/queries/clientpositive/vectorization_numeric_overflows.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/vectorization_numeric_overflows.q.out 
> PRE-CREATION 
>   vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
> 657ea34e11f7465e6c77d45128b298e7326a057b 
>   vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorTestCode.java 
> d97646f8b1c4a074da59b4685939fc4359c9c30d 
> 
> 
> Diff: https://reviews.apache.org/r/65500/diff/6/
> 
> 
> Testing
> ---
> 
>

Re: Review Request 65500: HIVE-18421 : Vectorized execution handles overflows in a different manner than non-vectorized execution

2018-02-06 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/
---

(Updated Feb. 7, 2018, 7:13 a.m.)


Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.


Changes
---

added Sahil's suggested changes.


Bugs: HIVE-18421
https://issues.apache.org/jira/browse/HIVE-18421


Repository: hive-git


Description
---

See JIRA.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
99e8457c7b2d506cfc7c71ca18bc678fe5cdf049 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
b5011c3adcedf8974d3241994733e0021a851cbd 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
3e955578933dd7990939865527c3bd11023b3a90 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
  ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
e95baa6199e138a4e0c009e62ce495b626e5909c 
  ql/src/gen/vectorization/TestTemplates/TestClass.txt 
62c58fb293fbe2d4d948c6a3409ee31466424a02 
  
ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
 PRE-CREATION 
  
ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
 bbe78c8720e16163b642f54d27fdf6b65ba9850b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
8264e8ad285deac29424bd1cb0bf626436d47c75 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
 75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
 6b4d714c9a79a55593c4a4d254267a3035abb10f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
 710165033627b33d9b238cc847dbac36c07ee5f6 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
af8552caa02f2896f393a5099abdb1ae5abd4c16 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
e2a638da518a2071ff15b8da6899646ec45c832a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
4e45788936559bbb7cfe65e9ffd083747b37dcc2 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
b1200e673e6b470b5fd1cc856270a6da615f16cb 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
 ab6f6b79316818cac458390dc2d087091057c63b 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
 02dec659ce421eef06f924bb6973070878d57be3 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorMathFunctions.java
 1950e92bd6d5e4f818588e691db30cd28193c716 
  ql/src/test/queries/clientpositive/vectorization_numeric_overflows.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/vectorization_numeric_overflows.q.out 
PRE-CREATION 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
657ea34e11f7465e6c77d45128b298e7326a057b 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorTestCode.java 
d97646f8b1c4a074da59b4685939fc4359c9c30d 


Diff: https://reviews.apache.org/r/65500/diff/6/

Changes: https://reviews.apache.org/r/65500/diff/5-6/


Testing
---

HiveQA


Thanks,

Vihang Karajgaonkar

Re: Review Request 65500: HIVE-18421 : Vectorized execution handles overflows in a different manner than non-vectorized execution

2018-02-06 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/
---

(Updated Feb. 7, 2018, 12:49 a.m.)


Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.


Changes
---

added changes suggested by Sahil.


Bugs: HIVE-18421
https://issues.apache.org/jira/browse/HIVE-18421


Repository: hive-git


Description
---

See JIRA.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
99e8457c7b2d506cfc7c71ca18bc678fe5cdf049 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
b5011c3adcedf8974d3241994733e0021a851cbd 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
3e955578933dd7990939865527c3bd11023b3a90 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
  ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
e95baa6199e138a4e0c009e62ce495b626e5909c 
  ql/src/gen/vectorization/TestTemplates/TestClass.txt 
62c58fb293fbe2d4d948c6a3409ee31466424a02 
  
ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
 PRE-CREATION 
  
ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
 bbe78c8720e16163b642f54d27fdf6b65ba9850b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
8264e8ad285deac29424bd1cb0bf626436d47c75 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
 75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
 6b4d714c9a79a55593c4a4d254267a3035abb10f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
 710165033627b33d9b238cc847dbac36c07ee5f6 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
af8552caa02f2896f393a5099abdb1ae5abd4c16 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
e2a638da518a2071ff15b8da6899646ec45c832a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
4e45788936559bbb7cfe65e9ffd083747b37dcc2 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
b1200e673e6b470b5fd1cc856270a6da615f16cb 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
 ab6f6b79316818cac458390dc2d087091057c63b 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
 02dec659ce421eef06f924bb6973070878d57be3 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorMathFunctions.java
 1950e92bd6d5e4f818588e691db30cd28193c716 
  ql/src/test/queries/clientpositive/vectorization_numeric_overflows.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/vectorization_numeric_overflows.q.out 
PRE-CREATION 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
657ea34e11f7465e6c77d45128b298e7326a057b 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorTestCode.java 
d97646f8b1c4a074da59b4685939fc4359c9c30d 


Diff: https://reviews.apache.org/r/65500/diff/4/

Changes: https://reviews.apache.org/r/65500/diff/3-4/


Testing
---

HiveQA


Thanks,

Vihang Karajgaonkar

Re: Review Request 65500: HIVE-18421 : Vectorized execution handles overflows in a different manner than non-vectorized execution

2018-02-06 Thread Vihang Karajgaonkar via Review Board



> On Feb. 7, 2018, 12:17 a.m., Sahil Takiar wrote:
> > ql/src/gen/vectorization/TestTemplates/TestClass.txt
> > Lines 27 (patched)
> > 
> >
> > is this necessary?

yes, this is needed since this template is used to auto-generate the TestClass 
for vector expressions. The templates for test methods (eg 
TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt) uses 
TypeInfoFactory, so we need the import


> On Feb. 7, 2018, 12:17 a.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
> > Lines 41 (patched)
> > 
> >
> > So would this be printed for each column vector batch? Wouldn't that 
> > flood the logs if there are a lot of rows being read?

fair point. I will remove this


> On Feb. 7, 2018, 12:17 a.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
> > Lines 45-46 (patched)
> > 
> >
> > is the `func` method invoked for each row? does the content of the if 
> > statement change with each value of `v` passed into the method? or can the 
> > check be moved out and just evaluated once?
> > 
> > is it a valid case when the `outputTypeInfo` is `null`? shouldn't there 
> > always be an output type set?

based on my understanding outputTypeInfo should never be null. But I could be 
wrong and I thought its safer to do a null check. Also, many of the junit test 
cases don't set the outputTypeInfo. I will move if check out of this method so 
that it is evaluated only once.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/#review196948
---


On Feb. 7, 2018, 12:49 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65500/
> ---
> 
> (Updated Feb. 7, 2018, 12:49 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.
> 
> 
> Bugs: HIVE-18421
> https://issues.apache.org/jira/browse/HIVE-18421
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 99e8457c7b2d506cfc7c71ca18bc678fe5cdf049 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
> b5011c3adcedf8974d3241994733e0021a851cbd 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
> cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
> 3e955578933dd7990939865527c3bd11023b3a90 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
> f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
>   ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
> e95baa6199e138a4e0c009e62ce495b626e5909c 
>   ql/src/gen/vectorization/TestTemplates/TestClass.txt 
> 62c58fb293fbe2d4d948c6a3409ee31466424a02 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
>  bbe78c8720e16163b642f54d27fdf6b65ba9850b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 8264e8ad285deac29424bd1cb0bf626436d47c75 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
>  75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
>  6b4d714c9a79a55593c4a4d254267a3035abb10f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
>  710165033627b33d9b238cc847dbac36c07ee5f6 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
> af8552caa02f2896f393a5099abdb1ae5abd4c16 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
> e2a638da518a2071ff15b8da6899646ec45c832a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
> 99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
>

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

2018-02-05 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review196867
---




ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
Lines 114 (patched)


nit, It should probably say "Implementation is consistent with .." Same 
with TypesFromInt64PageReader and TypesFromFloatPageReader below. Thanks



ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
Lines 9 (patched)


Does this test work when dictionary encoding is both enabled/disabled? You 
can change parquet dictionary encoding by setting table property 
"parquet.enable.dictionary"="true" or "false". Can you please modify the test 
to include that as well?



ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
Lines 49 (patched)


I think this comment can be removed now so that there is no confusion in 
the future.



ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
Lines 71 (patched)


Can you please test using timestamps as well. Specifically, the following 
should work.

drop table test_alter2;
create table test_alter2 (ts timestamp) stored as parquet;
insert into test_alter2 values ('2018-01-01 13:14:15.123456'), ('2018-01-02 
14:15:16.123456'), ('2018-01-03 16:17:18.123456');
select * from test_alter2;
alter table test_alter2 replace columns (ts string);
select * from test_alter2;

drop table test_alter2;
create table test_alter2 (ts timestamp) stored as parquet;
insert into test_alter2 values ('2018-01-01 13:14:15.123456'), ('2018-01-02 
14:15:16.123456'), ('2018-01-03 16:17:18.123456');
select * from test_alter2;
alter table test_alter2 replace columns (ts varchar(19));
-- this should truncate the microseconds
select * from test_alter2;

drop table test_alter2;
create table test_alter2 (ts timestamp) stored as parquet;
insert into test_alter2 values ('2018-01-01 13:14:15.123456'), ('2018-01-02 
14:15:16.123456'), ('2018-01-03 16:17:18.123456');
select * from test_alter2;
alter table test_alter2 replace columns (ts char(25);
select * from test_alter2;


- Vihang Karajgaonkar


On Feb. 5, 2018, 8:46 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 5, 2018, 8:46 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/2/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 65500: Vectorized execution handles overflows in a different manner than non-vectorized execution

2018-02-05 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/
---

(Updated Feb. 5, 2018, 8:54 p.m.)


Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.


Changes
---

patch version 2 with fix for the tests. Changed default value of 
hive.vectorized.use.checked.expressions. Modified test templates for checked 
Expressions.


Bugs: HIVE-18449
https://issues.apache.org/jira/browse/HIVE-18449


Repository: hive-git


Description
---

See JIRA.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
4f2e6d31af85cf1d9866ad1419f5c06a18eea347 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
b5011c3adcedf8974d3241994733e0021a851cbd 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
3e955578933dd7990939865527c3bd11023b3a90 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
  ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
e95baa6199e138a4e0c009e62ce495b626e5909c 
  ql/src/gen/vectorization/TestTemplates/TestClass.txt 
62c58fb293fbe2d4d948c6a3409ee31466424a02 
  
ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
 PRE-CREATION 
  
ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
 bbe78c8720e16163b642f54d27fdf6b65ba9850b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
d46eb8d737cf2885b7f34004a9bd9eadbcf6af7a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
 75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
 6b4d714c9a79a55593c4a4d254267a3035abb10f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
 710165033627b33d9b238cc847dbac36c07ee5f6 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
af8552caa02f2896f393a5099abdb1ae5abd4c16 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
e2a638da518a2071ff15b8da6899646ec45c832a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
4e45788936559bbb7cfe65e9ffd083747b37dcc2 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
b1200e673e6b470b5fd1cc856270a6da615f16cb 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
 ab6f6b79316818cac458390dc2d087091057c63b 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
 02dec659ce421eef06f924bb6973070878d57be3 
  ql/src/test/queries/clientpositive/vectorization_numeric_overflows.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/vectorization_numeric_overflows.q.out 
PRE-CREATION 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
657ea34e11f7465e6c77d45128b298e7326a057b 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorTestCode.java 
d97646f8b1c4a074da59b4685939fc4359c9c30d 


Diff: https://reviews.apache.org/r/65500/diff/2/

Changes: https://reviews.apache.org/r/65500/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 65478: HIVE-18553 VectorizedParquetReader fails after adding a new column to table

2018-02-05 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65478/#review196718
---




ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
Lines 24 (patched)


Do we need to override the methods for other reads as well? What is the 
criteria to identify the methods which need to be overriden for this and 
TypesFromInt64PageReader?



ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q
Lines 9 (patched)


can you please confirm that the result of this q file matches with 
non-vectorized execution?



ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out
Lines 140 (patched)


this is interesting. Do you know why this row is returned first?


- Vihang Karajgaonkar


On Feb. 5, 2018, 8:46 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65478/
> ---
> 
> (Updated Feb. 5, 2018, 8:46 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> VectorizedParquetReader throws an exception when trying to reading from a 
> parquet table on which new columns are added.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/BaseVectorizedColumnReader.java
>  907a9b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedDummyColumnReader.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
>  08ac57b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java
>  9e414dc 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java
>  5d3ebd6 
>   ql/src/test/queries/clientpositive/schema_evol_par_vec_table.q PRE-CREATION 
>   ql/src/test/results/clientpositive/schema_evol_par_vec_table.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65478/diff/2/
> 
> 
> Testing
> ---
> 
> Newly added UT passed and qtest passed locally.
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Review Request 65500: Vectorized execution handles overflows in a different manner than non-vectorized execution

2018-02-02 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/
---

Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.


Bugs: HIVE-18449
https://issues.apache.org/jira/browse/HIVE-18449


Repository: hive-git


Description
---

See JIRA.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
4f2e6d31af85cf1d9866ad1419f5c06a18eea347 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
b5011c3adcedf8974d3241994733e0021a851cbd 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
3e955578933dd7990939865527c3bd11023b3a90 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
  ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
e95baa6199e138a4e0c009e62ce495b626e5909c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
 bbe78c8720e16163b642f54d27fdf6b65ba9850b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
d46eb8d737cf2885b7f34004a9bd9eadbcf6af7a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
 75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
 6b4d714c9a79a55593c4a4d254267a3035abb10f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
 710165033627b33d9b238cc847dbac36c07ee5f6 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
af8552caa02f2896f393a5099abdb1ae5abd4c16 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
e2a638da518a2071ff15b8da6899646ec45c832a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
4e45788936559bbb7cfe65e9ffd083747b37dcc2 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
b1200e673e6b470b5fd1cc856270a6da615f16cb 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
 ab6f6b79316818cac458390dc2d087091057c63b 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
 02dec659ce421eef06f924bb6973070878d57be3 
  ql/src/test/queries/clientpositive/vectorization_numeric_overflows.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/vectorization_numeric_overflows.q.out 
PRE-CREATION 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
657ea34e11f7465e6c77d45128b298e7326a057b 


Diff: https://reviews.apache.org/r/65500/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 65330: HIVE-18526 Backport HIVE-16886 to Hive 2

2018-01-26 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65330/#review196365
---




metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 8236 (patched)


Does this work when there are multiple HMS instances?



metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java
Lines 442 (patched)


does this need to say "It assumes that no other thread does any 
modifications..."



metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java
Lines 471 (patched)


is this needed?



metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java
Lines 534-535 (patched)


do we need this since we asserting oldVal is always null above?


- Vihang Karajgaonkar


On Jan. 25, 2018, 1:38 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65330/
> ---
> 
> (Updated Jan. 25, 2018, 1:38 a.m.)
> 
> 
> Review request for hive, anishek, Andrew Sherman, Janaki Lahorani, Sergio 
> Pena, Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18526
> https://issues.apache.org/jira/browse/HIVE-18526
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18526 Backport HIVE-16886 to Hive 2
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> d16d2a3cd6a08217bb5b5b7bd9226f5e19f68994 
>   metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
> 69e8826f53c9278a7307817c8bfa4151f36631b9 
> 
> 
> Diff: https://reviews.apache.org/r/65330/diff/1/
> 
> 
> Testing
> ---
> 
> I wrote unit test which runs against stand-alone DB and verified that 
> everything is as expected.
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Review Request 65132: HIVE-18422 : Vectorized input format should not be used when input format is excluded and row.serde is enabled

2018-01-12 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65132/
---

Review request for hive and Matt McCline.


Repository: hive-git


Description
---

HIVE-18422 : Vectorized input format should not be used when input format is 
excluded and row.serde is enabled


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
6f289703a70b2fc2c9cf6fc38707ae0fc4a68809 
  ql/src/test/queries/clientpositive/vectorization_input_format_excludes.q 
f8e82455e059a4ed25b1e1d2dce7f2fe0a6bffa3 
  
ql/src/test/results/clientpositive/llap/vectorization_input_format_excludes.q.out
 ab8752a8d22fb6288396896368124ff7c6561762 
  
ql/src/test/results/clientpositive/spark/vectorization_input_format_excludes.q.out
 887f44bb51665547530430a3ff2f3469ad68bebc 


Diff: https://reviews.apache.org/r/65132/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 65018: HIVE-18372 Create testing infra to test different HMS instances

2018-01-10 Thread Vihang Karajgaonkar via Review Board



> On Jan. 10, 2018, 3:35 a.m., Vihang Karajgaonkar wrote:
> > standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
> > Lines 52 (patched)
> > 
> >
> > Parameterized tests don't work well in Ptest since it is cannot be 
> > parallelized. I would suggest creating a Abstract base class with all the 
> > tests and creating subclasses for remote and embedded like we have for 
> > TestHiveMetaStore test currently.
> 
> Peter Vary wrote:
> TestHiveMetaStore currently has 5 subclasses (TestEmbeddedHiveMetaStore, 
> TestRemoteHiveMetaStore, TestSetUGIOnBothClientServer, 
> TestSetUGIOnOnlyClient, TestSetUGIOnOnlyServer) running the same tests 
> against different configurations. This class is a huge one which should be 
> split, but spliting would mean to multiple the test classes 5 times. That is 
> why I think it is better to do this way.
> If we will have timing concerns then we still can split the test cases 
> more.
> 
> What do you think?

I think thats fine. If we find this test is taking a lot of time may be we 
should split it later. I was concerned because overtime folks would add more 
tests to these classes and they become huge. One of the optimizations which was 
done sometime back to reduce the Ptest execution time was to change some 
long-running parameterized classes to sub-classes so that they can be run in 
parallel. Having *more* number of test classes should not be a concern as long 
as we don't have duplicate code.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65018/#review195114
---


On Jan. 10, 2018, 3:44 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65018/
> ---
> 
> (Updated Jan. 10, 2018, 3:44 p.m.)
> 
> 
> Review request for hive, Alan Gates, Marta Kuczora, Adam Szita, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-18372
> https://issues.apache.org/jira/browse/HIVE-18372
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Created:
> - AbstractMetastore class - to privide an interface for different metastore 
> implementation (start/stop/warehouse path methods)
> -- Implementation for Embedded/Remote/Cluster metastores
> - MiniHMS with builder - to create hms instances for test
> - MetaStoreFactory - to create the parameter list for parametrized test
> - TestDatabases - test for database related metastore functions to showcase 
> the infrastructure
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/MetaStoreFactoryForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/AbstractMetaStoreService.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/ClusterMetaStoreForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/EmbeddedMetaStoreForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/MiniHMS.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/RemoteMetaStoreForTests.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65018/diff/3/
> 
> 
> Testing
> ---
> 
> Run the new tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 65018: HIVE-18372 Create testing infra to test different HMS instances

2018-01-10 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65018/#review195158
---


Fix it, then Ship it!




Overall looks good. Thanks for taking this up. I suggested some small 
improvements to the tests below.


standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 83 (patched)


may be we should catch the exception (and do nothing) here so that it 
atleast attempts to stop other metastores in the list.



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 146 (patched)


should we check if the directory is removed after dropping the db here?



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 185 (patched)


looks like this throws InvalidObjectException according to the annotation, 
so may be this comment is not needed? Also, curious to understand why the name 
is invalid :)



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 350 (patched)


I think we should catch the exception and assert the exception type is 
InvalidOperationException so that we catch errors like if someone changes the 
thrown exception in the future.



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 353 (patched)


may be we should confirm if the directory is removed after drop cascade is 
done.



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 381 (patched)


same as above. Catch the exception and assert the exception type is 
InvalidOperationException



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 384 (patched)


assert directory is cleaned up after this line.



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 433 (patched)


shouldn't we check if default database is also being returned?


- Vihang Karajgaonkar


On Jan. 10, 2018, 3:44 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65018/
> ---
> 
> (Updated Jan. 10, 2018, 3:44 p.m.)
> 
> 
> Review request for hive, Alan Gates, Marta Kuczora, Adam Szita, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-18372
> https://issues.apache.org/jira/browse/HIVE-18372
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Created:
> - AbstractMetastore class - to privide an interface for different metastore 
> implementation (start/stop/warehouse path methods)
> -- Implementation for Embedded/Remote/Cluster metastores
> - MiniHMS with builder - to create hms instances for test
> - MetaStoreFactory - to create the parameter list for parametrized test
> - TestDatabases - test for database related metastore functions to showcase 
> the infrastructure
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/MetaStoreFactoryForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/AbstractMetaStoreService.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/ClusterMetaStoreForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/EmbeddedMetaStoreForTests.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/MiniHMS.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/RemoteMetaStoreForTests.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65018/diff/3/
> 
> 
> Testing
> ---
> 
> Run the new tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 65018: HIVE-18372 Create testing infra to test different HMS instances

2018-01-09 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65018/#review195114
---




standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/MetaStoreFactory.java
Lines 32 (patched)


If this is used for testing only I would suggest renaming it to something 
like MetaStoreFactoryForTests so that it is clear.

Same for the other new classes introduced namely, 
[Cluster|Embedded|Remote]Metastore.java



standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
Lines 52 (patched)


Parameterized tests don't work well in Ptest since it is cannot be 
parallelized. I would suggest creating a Abstract base class with all the tests 
and creating subclasses for remote and embedded like we have for 
TestHiveMetaStore test currently.


- Vihang Karajgaonkar


On Jan. 9, 2018, 2:50 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65018/
> ---
> 
> (Updated Jan. 9, 2018, 2:50 p.m.)
> 
> 
> Review request for hive, Alan Gates, Marta Kuczora, Adam Szita, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-18372
> https://issues.apache.org/jira/browse/HIVE-18372
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Created:
> - AbstractMetastore class - to privide an interface for different metastore 
> implementation (start/stop/warehouse path methods)
> -- Implementation for Embedded/Remote/Cluster metastores
> - MiniHMS with builder - to create hms instances for test
> - MetaStoreFactory - to create the parameter list for parametrized test
> - TestDatabases - test for database related metastore functions to showcase 
> the infrastructure
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/MetaStoreFactory.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/AbstractMetaStoreService.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/ClusterMetaStore.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/EmbeddedMetaStore.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/MiniHMS.java
>  PRE-CREATION 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/minihms/RemoteMetaStore.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/65018/diff/2/
> 
> 
> Testing
> ---
> 
> Run the new tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 63711: HIVE-17528 Add more q-tests for Hive-on-Spark with Parquet vectorized reader

2017-11-16 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63711/#review191195
---


Ship it!




Ship It!

- Vihang Karajgaonkar


On Nov. 15, 2017, 1:34 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63711/
> ---
> 
> (Updated Nov. 15, 2017, 1:34 a.m.)
> 
> 
> Review request for hive and Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Most of the vectorization related q-tests operate on ORC tables using Tez. It 
> would be good to add more coverage on a different combination of engine and 
> file-format. We can model existing q-tests using parquet tables and run it 
> using TestSparkCliDriver
> 
> 
> Diffs
> -
> 
>   data/scripts/q_test_cleanup.sql 4620dcd 
>   data/scripts/q_test_init.sql f763c12 
>   itests/src/test/resources/testconfiguration.properties 1d16b65 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java f1d90ff 
>   pom.xml dfb29ce 
>   ql/src/test/queries/clientpositive/parquet_read_backward_compatible_files.q 
> 0abbc2f 
>   ql/src/test/queries/clientpositive/parquet_vectorization_0.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_10.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_11.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_12.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_13.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_14.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_15.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_16.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_17.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_3.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_7.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_8.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_9.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_decimal_date.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_div0.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_limit.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_nested_udf.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_not.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_offset_limit.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part_project.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part_varchar.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_pushdown.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/add_part_exist.q.out f8d50ca 
>   ql/src/test/results/clientpositive/alter1.q.out c2efbe5 
>   ql/src/test/results/clientpositive/alter2.q.out 18032ac 
>   ql/src/test/results/clientpositive/alter3.q.out 3bd7288 
>   ql/src/test/results/clientpositive/alter4.q.out ddcb0ed 
>   ql/src/test/results/clientpositive/alter5.q.out 1eb24c2 
>   ql/src/test/results/clientpositive/alter_index.q.out bca4e12 
>   ql/src/test/results/clientpositive/alter_rename_partition.q.out 5702d39 
>   ql/src/test/results/clientpositive/authorization_9.q.out 39e0a56 
>   ql/src/test/results/clientpositive/authorization_show_grant.q.out d0fed81 
>   ql/src/test/results/clientpositive/cte_5.q.out ac0dac0 
>   ql/src/test/results/clientpositive/cte_mat_4.q.out bc92f14 
>   ql/src/test/results/clientpositive/cte_mat_5.q.out 3747cec 
>   ql/src/test/results/clientpositive/describe_table_json.q.out ebf15ac 
>   ql/src/test/results/clientpositive/drop_table_with_index.q.out 3175cdb 
>   ql/src/test/results/clientpositive/index_creation.q.out ca113ef 
>   ql/src/test/results/clientpositive/input2.q.out 4dac88a 
>   ql/src/test/results/clientpositive/input3.q.out 8316a46 
>   ql/src/test/results/clientpositive/llap/cte_5.q.out dad6cd8 
>   ql/src/test/results/clientpositive/llap/cte_mat_4.q.out 67910c5 
>

Re: Review Request 63711: HIVE-17528 Add more q-tests for Hive-on-Spark with Parquet vectorized reader

2017-11-16 Thread Vihang Karajgaonkar via Review Board



> On Nov. 15, 2017, 8:44 p.m., Vihang Karajgaonkar wrote:
> > ql/src/test/results/clientpositive/llap/sysdb.q.out
> > Lines 2236 (patched)
> > 
> >
> > not sure why this file is changing. Do you know?
> 
> cheng xu wrote:
> Actually this test case is failing in pre-commit due to difference in 
> output. Locally I am not able to reproduce the output as the golden files. So 
> I prefer to upload the local version instead and file other jira about this 
> failing test case. Any thoughts on that?

Yes, I think we should exclude the changes to this file since it is not related 
to the patch. Lets fix it a separate JIRA.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63711/#review191092
---


On Nov. 15, 2017, 1:34 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63711/
> ---
> 
> (Updated Nov. 15, 2017, 1:34 a.m.)
> 
> 
> Review request for hive and Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Most of the vectorization related q-tests operate on ORC tables using Tez. It 
> would be good to add more coverage on a different combination of engine and 
> file-format. We can model existing q-tests using parquet tables and run it 
> using TestSparkCliDriver
> 
> 
> Diffs
> -
> 
>   data/scripts/q_test_cleanup.sql 4620dcd 
>   data/scripts/q_test_init.sql f763c12 
>   itests/src/test/resources/testconfiguration.properties 1d16b65 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java f1d90ff 
>   pom.xml dfb29ce 
>   ql/src/test/queries/clientpositive/parquet_read_backward_compatible_files.q 
> 0abbc2f 
>   ql/src/test/queries/clientpositive/parquet_vectorization_0.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_10.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_11.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_12.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_13.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_14.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_15.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_16.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_17.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_3.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_7.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_8.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_9.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_decimal_date.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_div0.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_limit.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_nested_udf.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_not.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_offset_limit.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part_project.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part_varchar.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_pushdown.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/add_part_exist.q.out f8d50ca 
>   ql/src/test/results/clientpositive/alter1.q.out c2efbe5 
>   ql/src/test/results/clientpositive/alter2.q.out 18032ac 
>   ql/src/test/results/clientpositive/alter3.q.out 3bd7288 
>   ql/src/test/results/clientpositive/alter4.q.out ddcb0ed 
>   ql/src/test/results/clientpositive/alter5.q.out 1eb24c2 
>   ql/src/test/results/clientpositive/alter_index.q.out bca4e12 
>   ql/src/test/results/clientpositive/alter_rename_partition.q.out 5702d39 
>   ql/src/test/results/clientpositive/authorization_9.q.out 39e0a56 
>   ql/src/test/results/clientpositive/authorization_show_grant.q.out

Re: Review Request 63806: HIVE-16756 : Vectorization: LongColModuloLongColumn throws java.lang.ArithmeticException: / by zero

2017-11-15 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63806/
---

(Updated Nov. 16, 2017, 5:21 a.m.)


Review request for hive, Aihua Xu and Matt McCline.


Changes
---

added a test case


Bugs: HIVE-16756
https://issues.apache.org/jira/browse/HIVE-16756


Repository: hive-git


Description
---

HIVE-16756 : Vectorization: LongColModuloLongColumn throws 
java.lang.ArithmeticException: / by zero


Diffs (updated)
-

  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
8b586b1f00ce7d6081f973a5736100d8941f79bc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumn.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
6d3e82e9b96e012d875d947fa397c6c67df6a931 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java 
21f6540512ec795171014d87a6fde0d0ea5f23cf 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
 ea06ea0aefcf5e36f204eaad78131860b44298ae 
  ql/src/test/queries/clientpositive/vectorization_div0.q 
025d457807dd0642965a81c6b093e421c4acd0f8 
  ql/src/test/results/clientpositive/spark/vectorization_div0.q.out 
631b0723fb0d7ab011ad2bfd7be4b33d11d76b1c 
  ql/src/test/results/clientpositive/tez/vectorization_div0.q.out 
6c3354cb4a8cd439d86df7e6b0cf759ea4c04cd0 
  ql/src/test/results/clientpositive/vectorization_div0.q.out 
97f1687b85193e681f26c61107a6d9266c1d87a2 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
e58d4e91938dc266111042fe98b05a3d9c6fc5e9 


Diff: https://reviews.apache.org/r/63806/diff/3/

Changes: https://reviews.apache.org/r/63806/diff/2-3/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 63806: HIVE-16756 : Vectorization: LongColModuloLongColumn throws java.lang.ArithmeticException: / by zero

2017-11-15 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63806/
---

(Updated Nov. 15, 2017, 10:50 p.m.)


Review request for hive, Aihua Xu and Matt McCline.


Changes
---

the fix was needed only in case of long % long expression. Removed changes to 
LongColDivideLongColumn.java and removed the unnecessary file 
ColumnDivideLong.txt


Summary (updated)
-

HIVE-16756 : Vectorization: LongColModuloLongColumn throws 
java.lang.ArithmeticException: / by zero


Bugs: HIVE-16756
https://issues.apache.org/jira/browse/HIVE-16756


Repository: hive-git


Description (updated)
---

HIVE-16756 : Vectorization: LongColModuloLongColumn throws 
java.lang.ArithmeticException: / by zero


Diffs (updated)
-

  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
8b586b1f00ce7d6081f973a5736100d8941f79bc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumn.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
6d3e82e9b96e012d875d947fa397c6c67df6a931 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java 
21f6540512ec795171014d87a6fde0d0ea5f23cf 
  ql/src/test/queries/clientpositive/vectorization_div0.q 
025d457807dd0642965a81c6b093e421c4acd0f8 
  ql/src/test/results/clientpositive/spark/vectorization_div0.q.out 
631b0723fb0d7ab011ad2bfd7be4b33d11d76b1c 
  ql/src/test/results/clientpositive/tez/vectorization_div0.q.out 
6c3354cb4a8cd439d86df7e6b0cf759ea4c04cd0 
  ql/src/test/results/clientpositive/vectorization_div0.q.out 
97f1687b85193e681f26c61107a6d9266c1d87a2 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
e58d4e91938dc266111042fe98b05a3d9c6fc5e9 


Diff: https://reviews.apache.org/r/63806/diff/2/

Changes: https://reviews.apache.org/r/63806/diff/1-2/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 63711: HIVE-17528 Add more q-tests for Hive-on-Spark with Parquet vectorized reader

2017-11-15 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63711/#review191092
---



Thanks for the patch Ferdinand. Just one comment below, rest of the qfiles look 
good to me. We should also do a separate exercise to verify if the output of 
the queries in these qfiles match when vectorization is disabled to make sure 
that golden files are not hiding any bugs.


ql/src/test/results/clientpositive/llap/sysdb.q.out
Lines 2236 (patched)


not sure why this file is changing. Do you know?



ql/src/test/results/clientpositive/llap/sysdb.q.out
Line 3646 (original), 3706 (patched)


how come data values are changing here?


- Vihang Karajgaonkar


On Nov. 15, 2017, 1:34 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63711/
> ---
> 
> (Updated Nov. 15, 2017, 1:34 a.m.)
> 
> 
> Review request for hive and Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Most of the vectorization related q-tests operate on ORC tables using Tez. It 
> would be good to add more coverage on a different combination of engine and 
> file-format. We can model existing q-tests using parquet tables and run it 
> using TestSparkCliDriver
> 
> 
> Diffs
> -
> 
>   data/scripts/q_test_cleanup.sql 4620dcd 
>   data/scripts/q_test_init.sql f763c12 
>   itests/src/test/resources/testconfiguration.properties 1d16b65 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java f1d90ff 
>   pom.xml dfb29ce 
>   ql/src/test/queries/clientpositive/parquet_read_backward_compatible_files.q 
> 0abbc2f 
>   ql/src/test/queries/clientpositive/parquet_vectorization_0.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_10.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_11.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_12.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_13.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_14.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_15.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_16.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_17.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_3.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_7.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_8.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_9.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_decimal_date.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_div0.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_limit.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_nested_udf.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_not.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_offset_limit.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part_project.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_part_varchar.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_vectorization_pushdown.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/add_part_exist.q.out f8d50ca 
>   ql/src/test/results/clientpositive/alter1.q.out c2efbe5 
>   ql/src/test/results/clientpositive/alter2.q.out 18032ac 
>   ql/src/test/results/clientpositive/alter3.q.out 3bd7288 
>   ql/src/test/results/clientpositive/alter4.q.out ddcb0ed 
>   ql/src/test/results/clientpositive/alter5.q.out 1eb24c2 
>   ql/src/test/results/clientpositive/alter_index.q.out bca4e12 
>   ql/src/test/results/clientpositive/alter_rename_partition.q.out 5702d39 
>   ql/src/test/results/clientpositive/authorization_9.q.out 39e0a56 
>   ql/src/test/results/clientpositive/authorization_show_grant.q.out d0fed81 
>

Review Request 63806: HIVE-16756 : Vectorization: LongColModuloLongColumn throws java.lang.ArithmeticException: divide by zero

2017-11-14 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63806/
---

Review request for hive, Aihua Xu and Matt McCline.


Bugs: HIVE-16756
https://issues.apache.org/jira/browse/HIVE-16756


Repository: hive-git


Description
---

HIVE-16756 : Vectorization: LongColModuloLongColumn throws 
java.lang.ArithmeticException: divide by zero


Diffs
-

  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
8b586b1f00ce7d6081f973a5736100d8941f79bc 
  ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideLong.txt 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColDivideLongColumn.java
 446c033ed473adfcfa0db630ceb716aecb623612 
  ql/src/test/queries/clientpositive/vectorization_div0.q 
025d457807dd0642965a81c6b093e421c4acd0f8 
  ql/src/test/results/clientpositive/spark/vectorization_div0.q.out 
631b0723fb0d7ab011ad2bfd7be4b33d11d76b1c 
  ql/src/test/results/clientpositive/tez/vectorization_div0.q.out 
6c3354cb4a8cd439d86df7e6b0cf759ea4c04cd0 
  ql/src/test/results/clientpositive/vectorization_div0.q.out 
97f1687b85193e681f26c61107a6d9266c1d87a2 
  vector-code-gen/src/org/apache/hadoop/hive/tools/GenVectorCode.java 
e58d4e91938dc266111042fe98b05a3d9c6fc5e9 


Diff: https://reviews.apache.org/r/63806/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

Re: Review Request 63586: Fix HIVE-17942: HiveAlterHandler should use the conf from HMS Handler

2017-11-08 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63586/#review190457
---


Ship it!




LGTM. Thanks for the changes!


itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreAlterColumnPar.java
Lines 94 (patched)


I looked up online I think a better way to do this would be 
Assert.fail("Exception not thrown"), but this works too.


- Vihang Karajgaonkar


On Nov. 7, 2017, 9:19 p.m., Janaki Lahorani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63586/
> ---
> 
> (Updated Nov. 7, 2017, 9:19 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Andrew Sherman, Sahil Takiar, 
> and Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HMS handler thread local will have the configuration changes from the user 
> local only to that connection.  HiveAlterHandler should use the thread local 
> to pick up user's configuration changes.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreAlterColumnPar.java
>  PRE-CREATION 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  ccadac1ada6aaae884ab39f5d99e91b8c542404e 
> 
> 
> Diff: https://reviews.apache.org/r/63586/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Janaki Lahorani
> 
>

Re: Review Request 63528: HIVE-17969: Metastore to alter table in batches of partitions when renaming table

2017-11-06 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63528/#review190193
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
Lines 263-264 (original), 275-278 (patched)


Shouldn't the partValues contain only the values from partBatch here?


- Vihang Karajgaonkar


On Nov. 2, 2017, 10:02 p.m., Adam Szita wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63528/
> ---
> 
> (Updated Nov. 2, 2017, 10:02 p.m.)
> 
> 
> Review request for hive, Peter Vary and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-17969
> https://issues.apache.org/jira/browse/HIVE-17969
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactoring alter table code to use batching of partitions when calling the 
> heavy removeUnusedColumnDescriptor method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  ccadac1ada6aaae884ab39f5d99e91b8c542404e 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  62801c53853dbafb7c425cff943ec819dcee4800 
> 
> 
> Diff: https://reviews.apache.org/r/63528/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adam Szita
> 
>

Re: Review Request 62995: HIVE-17806: Create directory for metrics file if it doesn't exist

2017-10-18 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62995/#review188551
---


Ship it!




Ship It!

- Vihang Karajgaonkar


On Oct. 18, 2017, 7:07 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62995/
> ---
> 
> (Updated Oct. 18, 2017, 7:07 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Janaki Lahorani, Sergio 
> Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-17806
> https://issues.apache.org/jira/browse/HIVE-17806
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17806: Create directory for metrics file if it doesn't exist
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
>  96243cb74a154b9a639ffb080256c4b43bd35a4b 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
>  254af7d4310578e3883c0dffa64bed0f823696ea 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
>  f71bb25463b56bc741f989454664397996b6a5cf 
> 
> 
> Diff: https://reviews.apache.org/r/62995/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 62995: HIVE-17806 Create directory for metrics file if it doesn't exist

2017-10-17 Thread Vihang Karajgaonkar via Review Board



> On Oct. 17, 2017, 11:21 p.m., Vihang Karajgaonkar wrote:
> > common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
> > Lines 119 (patched)
> > 
> >
> > I think it is better to throw a RuntimeException here instead of 
> > catching it since there is nothing more that you can do if the metricsDir 
> > could not be created and it doesn't exist.
> 
> Alexander Kolbasov wrote:
> Here we actually avoid creating JSON reporter. My thinking is that it 
> isn't a catastrophic faiilure so you should be able to continue running 
> without it. Do you think that it is better to throw an exception rather then 
> continue running without the reporter?

If we catch the exception and return then the ExecutorService remains 
uninitialized and when CodahaleMetrics class tries to close it sometime later 
it will result in a NPE on ExecutorService.shutdown(). I think throwing this 
exception and handling it at CodahaleMetrics such that this reporter is not 
added in list of reporters would be cleaner way to handle this error.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62995/#review188416
---


On Oct. 17, 2017, 12:35 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62995/
> ---
> 
> (Updated Oct. 17, 2017, 12:35 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Janaki Lahorani, Sergio 
> Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-17806
> https://issues.apache.org/jira/browse/HIVE-17806
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17806 Create directory for metrics file if it doesn't exist
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
>  96243cb74a154b9a639ffb080256c4b43bd35a4b 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
>  254af7d4310578e3883c0dffa64bed0f823696ea 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
>  f71bb25463b56bc741f989454664397996b6a5cf 
> 
> 
> Diff: https://reviews.apache.org/r/62995/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 62995: HIVE-17806 Create directory for metrics file if it doesn't exist

2017-10-17 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62995/#review188416
---




common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
Lines 119 (patched)


I think it is better to throw a RuntimeException here instead of catching 
it since there is nothing more that you can do if the metricsDir could not be 
created and it doesn't exist.


- Vihang Karajgaonkar


On Oct. 17, 2017, 12:35 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62995/
> ---
> 
> (Updated Oct. 17, 2017, 12:35 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Janaki Lahorani, Sergio 
> Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-17806
> https://issues.apache.org/jira/browse/HIVE-17806
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17806 Create directory for metrics file if it doesn't exist
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
>  96243cb74a154b9a639ffb080256c4b43bd35a4b 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
>  254af7d4310578e3883c0dffa64bed0f823696ea 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
>  f71bb25463b56bc741f989454664397996b6a5cf 
> 
> 
> Diff: https://reviews.apache.org/r/62995/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Review Request 62915: HIVE-17534 : Add a config to turn off parquet vectorization

2017-10-11 Thread Vihang Karajgaonkar via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62915/
---

Review request for hive, Matt McCline and Sergey Shelukhin.


Repository: hive-git


Description
---

HIVE-17534 : Add a config to turn off parquet vectorization


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
302569242e687175a98167de61a780fe3b0089ca 
  itests/src/test/resources/testconfiguration.properties 
6b237396e540f58d9bd908fa6f428bac2b2e4294 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
c19978040b4b2d0934731fded008ef68748e0f46 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapDecider.java 
ae05bd52fe25fe1b4ed35affafc21ac85c778736 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
13d7730acdbf87ae735c93d23db70765712a2535 
  ql/src/test/queries/clientpositive/vectorization_input_format_excludes.q 
PRE-CREATION 
  
ql/src/test/results/clientpositive/spark/vectorization_input_format_excludes.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/vectorization_input_format_excludes.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62915/diff/1/


Testing
---


Thanks,

Vihang Karajgaonkar

99 matches

Mail list logo