Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207983
---


Ship it!




Ship It!

- Aihua Xu


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  b61ee81533930c889f23d2551041055cbdd1a6b2 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
>  7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  0445cbf9095285bdcde72946f1b6dd9a9a3b9fff 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MSerDeInfo.java
>  68f07e2569b6531cf3e18919209aed1a17e88bf7 
>   
>

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-27 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207981
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
Line 200 (original), 200 (patched)


Seems it can be static since it doesn't refer to any variables outside. 
Maybe I miss anything?


- Aihua Xu


On Aug. 25, 2018, 12:23 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 25, 2018, 12:23 a.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  571c789eddfd2b1a27c65c48bdc6dccfafaaf676 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  d27224b23580b4662a85c874b657847ed068c9a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  b61ee81533930c889f23d2551041055cbdd1a6b2 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
>  7a0b21b2580d8bb9b256dbc698f125ed15ccdcd3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>

Re: Review Request 68395: HIVE-20306 : Implement projection spec for fetching only requested fields from partitions

2018-08-24 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68395/#review207920
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Line 376 (original), 416 (patched)


Why some functions like extractSqlString() not moving into 
MetastoreDirectSqlUtils class, but extractSqlLong() gets moved? Can we make it 
consistent?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 547 (patched)


Better add comments for this public function.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 886 (patched)


This function is a little confusing actually. Since we are not used 
anywhere else, probably we don't need to create such function seperately.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionProjectionEvaluator.java
Lines 200 (patched)


This function can be static and probably gives the name find() as well? 
findUtil is a little confusing to me.


- Aihua Xu


On Aug. 24, 2018, 6:15 p.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68395/
> ---
> 
> (Updated Aug. 24, 2018, 6:15 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov, Alan Gates, Peter Vary, and 
> Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20306
> https://issues.apache.org/jira/browse/HIVE-20306
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20306 : Implement projection spec for fetching only requested fields 
> from partitions
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  0ad2a2469e0330e050fdb8983078b80617afbbf1 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectSpec.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionFilterMode.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  47f96f323ade4e15753d6fd98709b9a882de624d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  0973f4f3c13554b23ba59b8a1aa1c5a37c094a9e 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  5ed4f71b1dd947c7d4cbb5c290b393b8cd2ea31d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  a595732f04af4304974186178377192227bb80fb 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
>  3c0d0a55b1dcca96dca77676873b68e52703715d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  7fc1e43de03eac1cfe802439ba38f83988299169 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e0c6c02715dab4d9ad457ec710bcb3159206c6c6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
>  e54a7321e2e1e6069a3e598627bc6f6eaed93449 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> feb44d5159f131fae932739923b1a41f5e72e74b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  6b660f55aca5b99517447f164339ed92261c9370 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
>  faf6810420ccf6088a3398711c0c216af722f062 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
>  ac10da2f3e623c1fe2d1cbb0849af00c4520297b 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

2018-04-23 Thread Aihua Xu via Review Board 


> On March 22, 2018, 11:03 a.m., Peter Vary wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 154-158 (original), 154-158 (patched)
> > 
> >
> > It might be a good idea, to use this around our batching as well:
> > - DatabaseProduct.needsInBatching(dbType)
> > 
> > What do you think @Aihua?

Thanks Peter for reviewing. This is slightly different problem. In directSQL, 
some databases may not need batch, some do. While in DN, the limitation is in 
DN, so it applies to all the databases.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/#review199751
---


On April 23, 2018, 10:51 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66188/
> ---
> 
> (Updated April 23, 2018, 10:51 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Yongzhi Chen.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2' && ... .
> 
> I'm breaking the query into multiple smaller queries and then we aggregate 
> the result together.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/alter_rename_table.q 53fb230cf6 
>   ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
>  PRE-CREATION 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  997f5fdb88 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  125d5a79f2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  59749e4947 
> 
> 
> Diff: https://reviews.apache.org/r/66188/diff/3/
> 
> 
> Testing
> ---
> 
> Manual test has been done for large column of tables.
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

2018-04-23 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/
---

(Updated April 23, 2018, 10:51 p.m.)


Review request for hive, Alexander Kolbasov and Yongzhi Chen.


Changes
---

Address comments.


Repository: hive-git


Description
---

If the table contains a lot of columns e.g, 5k, simple table rename would fail 
with the following stack trace. The issue is datanucleus can't handle the query 
with lots of colName='c1' && colName='c2' && ... .

I'm breaking the query into multiple smaller queries and then we aggregate the 
result together.


Diffs (updated)
-

  ql/src/test/queries/clientpositive/alter_rename_table.q 53fb230cf6 
  ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
 PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 997f5fdb88 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 125d5a79f2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 59749e4947 


Diff: https://reviews.apache.org/r/66188/diff/3/

Changes: https://reviews.apache.org/r/66188/diff/2-3/


Testing
---

Manual test has been done for large column of tables.


Thanks,

Aihua Xu

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

2018-04-17 Thread Aihua Xu via Review Board 


> On April 17, 2018, 4:49 p.m., Yongzhi Chen wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 7730 (patched)
> > 
> >
> > Should you call addQueryAfterUse and closeAllQueries ? That's how do 
> > you release the resources held by the batch queries?

In this change, I reuse single query to perform multiple queries and then 
release the query using queryWrapper. I did check how these two are used. Seems 
it's misused in other places that whenever there is new query, we created a new 
Query object which is not necessary. We can improve the logic to reuse the same 
Query object. It doesn't seem to be necessary.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/#review201323
---


On March 21, 2018, 6:57 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66188/
> ---
> 
> (Updated March 21, 2018, 6:57 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Yongzhi Chen.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2' && ... .
> 
> I'm breaking the query into multiple smaller queries and then we aggregate 
> the result together.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/alter_rename_table.q 2061850540 
>   ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
>  PRE-CREATION 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  6ead20aeaf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  88d88ed4df 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  9f822564bd 
> 
> 
> Diff: https://reviews.apache.org/r/66188/diff/2/
> 
> 
> Testing
> ---
> 
> Manual test has been done for large column of tables.
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

2018-03-21 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/
---

(Updated March 21, 2018, 6:57 p.m.)


Review request for hive, Alexander Kolbasov and Yongzhi Chen.


Changes
---

Added unit tests and fix checkstyle.


Repository: hive-git


Description
---

If the table contains a lot of columns e.g, 5k, simple table rename would fail 
with the following stack trace. The issue is datanucleus can't handle the query 
with lots of colName='c1' && colName='c2' && ... .

I'm breaking the query into multiple smaller queries and then we aggregate the 
result together.


Diffs (updated)
-

  ql/src/test/queries/clientpositive/alter_rename_table.q 2061850540 
  ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
 PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 6ead20aeaf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 88d88ed4df 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 9f822564bd 


Diff: https://reviews.apache.org/r/66188/diff/2/

Changes: https://reviews.apache.org/r/66188/diff/1-2/


Testing
---

Manual test has been done for large column of tables.


Thanks,

Aihua Xu

Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

2018-03-20 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/
---

Review request for hive, Alexander Kolbasov and Yongzhi Chen.


Repository: hive-git


Description
---

If the table contains a lot of columns e.g, 5k, simple table rename would fail 
with the following stack trace. The issue is datanucleus can't handle the query 
with lots of colName='c1' && colName='c2' && ... .

I'm breaking the query into multiple smaller queries and then we aggregate the 
result together.


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
 PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 6ead20aeaf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 88d88ed4df 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 9f822564bd 


Diff: https://reviews.apache.org/r/66188/diff/1/


Testing
---

Manual test has been done for large column of tables.


Thanks,

Aihua Xu

Re: Review Request 65500: HIVE-18421 : Vectorized execution handles overflows in a different manner than non-vectorized execution

2018-02-07 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65500/#review197073
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 2980 (patched)


I'm confused that we need this configuration. So we need to manually verify 
if the data is overflow and if there is overflow, we turn that on? 

Even we want to provide both CHECKED and UNCHECKED, I think we need to fail 
the query if there is overflow for unchecked implementation, otherwise, the 
user may get incorrect result. 

Of course, to me, I think we should just have CHECKED implementation, that 
is, we should always handle overflow internally.

BTW: how much performance impact will it be? And why do we  have big 
performance impact from the CHECKED implementation (sorry, I don't follow 
exactly the vecorization work)?



ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt
Lines 131 (patched)


Can we remove #ELSE from here and some other places below? Seems not needed.


- Aihua Xu


On Feb. 7, 2018, 7:13 a.m., Vihang Karajgaonkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65500/
> ---
> 
> (Updated Feb. 7, 2018, 7:13 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Gopal V, Matt McCline, and Sahil Takiar.
> 
> 
> Bugs: HIVE-18421
> https://issues.apache.org/jira/browse/HIVE-18421
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 99e8457c7b2d506cfc7c71ca18bc678fe5cdf049 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticColumn.txt 
> b5011c3adcedf8974d3241994733e0021a851cbd 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnArithmeticScalar.txt 
> cbec1abcc2b66f3ffc91b4778daf5017eff4379d 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnDivideColumn.txt 
> 3e955578933dd7990939865527c3bd11023b3a90 
>   ql/src/gen/vectorization/ExpressionTemplates/ColumnUnaryMinus.txt 
> f0ab4711e79c8a1bfceebcde9a3dda2b4e15a38a 
>   ql/src/gen/vectorization/ExpressionTemplates/ScalarArithmeticColumn.txt 
> e95baa6199e138a4e0c009e62ce495b626e5909c 
>   ql/src/gen/vectorization/TestTemplates/TestClass.txt 
> 62c58fb293fbe2d4d948c6a3409ee31466424a02 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnColumnOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/gen/vectorization/TestTemplates/TestColumnScalarOperationVectorExpressionCheckedEvaluation.txt
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorExpressionDescriptor.java
>  bbe78c8720e16163b642f54d27fdf6b65ba9850b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 8264e8ad285deac29424bd1cb0bf626436d47c75 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/LongColModuloLongColumnChecked.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OverflowUtils.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModDoubleToDouble.java
>  75ec419aa9ea5c3fcc5e7314fbac756d6a5d36d5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/PosModLongToLong.java
>  6b4d714c9a79a55593c4a4d254267a3035abb10f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpression.java
>  710165033627b33d9b238cc847dbac36c07ee5f6 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
> af8552caa02f2896f393a5099abdb1ae5abd4c16 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMod.java 
> e2a638da518a2071ff15b8da6899646ec45c832a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMultiply.java 
> 99d1ad7f203d946fd89d26074bd0e00dec8b3a1a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
> 4e45788936559bbb7cfe65e9ffd083747b37dcc2 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
> b1200e673e6b470b5fd1cc856270a6da615f16cb 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestUnaryMinus.java
>  ab6f6b79316818cac458390dc2d087091057c63b 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorArithmeticExpressions.java
>  02dec659ce421eef06f924bb6973070878d57be3 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorMathFunctions.java
>  1950e92bd6d5e4f818588e691db30cd28193c716 
>   ql/src/test/queries/clientpositive/vectorization_numeric_overflows.q 
> PRE-CREATION 
>

Re: Review Request 65330: HIVE-18526 Backport HIVE-16886 to Hive 2

2018-01-28 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65330/#review196397
---


Ship it!




Ship It!

- Aihua Xu


On Jan. 27, 2018, 12:05 a.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65330/
> ---
> 
> (Updated Jan. 27, 2018, 12:05 a.m.)
> 
> 
> Review request for hive, Aihua Xu, anishek, Andrew Sherman, Janaki Lahorani, 
> Sergio Pena, Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18526
> https://issues.apache.org/jira/browse/HIVE-18526
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18526 Backport HIVE-16886 to Hive 2
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> d16d2a3cd6a08217bb5b5b7bd9226f5e19f68994 
>   metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
> 69e8826f53c9278a7307817c8bfa4151f36631b9 
> 
> 
> Diff: https://reviews.apache.org/r/65330/diff/2/
> 
> 
> Testing
> ---
> 
> I wrote unit test which runs against stand-alone DB and verified that 
> everything is as expected.
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-11-01 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/
---

(Updated Nov. 1, 2017, 10:20 p.m.)


Review request for hive.


Repository: hive-git


Description
---

HIVE-15016: Run tests with Hadoop 3.0.0-beta1


Diffs (updated)
-

  common/pom.xml 868e14dbc3 
  hbase-handler/pom.xml 7f57b77168 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
c6f3b0ff3a 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
9cad97ad4b 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 6054d533f0 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 1ef454572c 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
 4b8f62c913 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 
a25a96faa9 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
b35aea9f60 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
f244ed6562 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
b2bdd19964 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java 
561b0a8952 
  hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
  hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
  hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
4ace16ca6f 
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/rcfile/TestRCFileMapReduceInputFormat.java
 bb6c582d28 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
 22d2cc6bff 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
 d69a7339c1 
  itests/hcatalog-unit/pom.xml bb6b10554f 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
 745aa999f9 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
 4e1384a634 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
 f8f18b3514 
  itests/hive-minikdc/pom.xml 95d2614901 
  itests/hive-unit-hadoop2/pom.xml 339a194f2d 
  itests/hive-unit/pom.xml 14409832c8 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
dbfc23510c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 707bcd10b7 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithLocalClusterSpark.java
 cabddea5c6 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestMultiSessionsHS2WithLocalClusterSpark.java
 e3f9646b59 
  itests/qtest-accumulo/pom.xml 40d0a749c8 
  itests/qtest-spark/pom.xml a506f7f081 
  itests/qtest/pom.xml 02664f3792 
  itests/util/pom.xml e6dc09fb8f 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
0cc9a89085 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
4f8fa05925 
  llap-server/pom.xml 47a04cc310 
  
llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
 6b08da66b6 
  metastore/pom.xml 04c6f47879 
  pom.xml 006e8f8611 
  ql/src/test/org/apache/hadoop/hive/ql/io/TestRCFile.java 19b97e4cf2 
  ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
b582471afd 
  ql/src/test/results/clientnegative/external1.q.out 661d669206 
  ql/src/test/results/clientnegative/external2.q.out eb5518c017 
  serde/pom.xml 7419cfb17a 
  shims/0.23/pom.xml 3ff1d38776 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
e9445eb11d 
  shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
2c37a51cf4 
  shims/scheduler/pom.xml 0eadb69435 
  spark-client/pom.xml 784d908e4a 
  standalone-metastore/pom.xml 8df622f179 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
 c10e36f94a 
  testutils/ptest2/src/main/resources/batch-exec.vm 2d16ca3e8a 


Diff: https://reviews.apache.org/r/63394/diff/3/

Changes: https://reviews.apache.org/r/63394/diff/2-3/


Testing
---


Thanks,

Aihua Xu

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-31 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189762
---



- Aihua Xu


On Oct. 31, 2017, 9:56 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 31, 2017, 9:56 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc3 
>   hbase-handler/pom.xml 7f57b77168 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f0 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
>  4b8f62c913 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  a25a96faa9 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
> b35aea9f60 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
> f244ed6562 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
> b2bdd19964 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java
>  561b0a8952 
>   hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
>   hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
>   hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
> 4ace16ca6f 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
>  22d2cc6bff 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
>  d69a7339c1 
>   itests/hcatalog-unit/pom.xml bb6b10554f 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
>  745aa999f9 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
>  4e1384a634 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
>  f8f18b3514 
>   itests/hive-minikdc/pom.xml 95d2614901 
>   itests/hive-unit-hadoop2/pom.xml 339a194f2d 
>   itests/hive-unit/pom.xml 14409832c8 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  707bcd10b7 
>   itests/qtest-accumulo/pom.xml 40d0a749c8 
>   itests/qtest-spark/pom.xml a506f7f081 
>   itests/qtest/pom.xml 02664f3792 
>   itests/util/pom.xml e6dc09fb8f 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
> 0cc9a89085 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
> 4f8fa05925 
>   llap-server/pom.xml 47a04cc310 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
>  6b08da66b6 
>   metastore/pom.xml 04c6f47879 
>   pom.xml 006e8f8611 
>   ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
> b582471afd 
>   ql/src/test/results/clientnegative/external1.q.out 661d669206 
>   ql/src/test/results/clientnegative/external2.q.out eb5518c017 
>   serde/pom.xml 7419cfb17a 
>   shims/0.23/pom.xml 3ff1d38776 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> e9445eb11d 
>   shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
> 2c37a51cf4 
>   shims/scheduler/pom.xml 0eadb69435 
>   spark-client/pom.xml 784d908e4a 
>   standalone-metastore/pom.xml 8df622f179 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
>  c10e36f94a 
>   testutils/ptest2/src/main/resources/batch-exec.vm 2d16ca3e8a 
> 
> 
> Diff: https://reviews.apache.org/r/63394/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-31 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189747
---




hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
Lines 119-120 (patched)


You mean we want to support NULL for Hbase table locations? I think we 
allow this 

`LOCATION` varchar(4000) DEFAULT NULL.

Seems we just need to correct the comments.



itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
Lines 106 (patched)


I think this will be needed on the cluster as well so hbase will use the 
right native library. Hbase has a doc for that but I can't find it right now.


- Aihua Xu


On Oct. 31, 2017, 9:56 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 31, 2017, 9:56 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc3 
>   hbase-handler/pom.xml 7f57b77168 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f0 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
>  4b8f62c913 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  a25a96faa9 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
> b35aea9f60 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
> f244ed6562 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
> b2bdd19964 
>   
> hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java
>  561b0a8952 
>   hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
>   hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
>   hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
> 4ace16ca6f 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
>  22d2cc6bff 
>   
> hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
>  d69a7339c1 
>   itests/hcatalog-unit/pom.xml bb6b10554f 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
>  745aa999f9 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
>  4e1384a634 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
>  f8f18b3514 
>   itests/hive-minikdc/pom.xml 95d2614901 
>   itests/hive-unit-hadoop2/pom.xml 339a194f2d 
>   itests/hive-unit/pom.xml 14409832c8 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  707bcd10b7 
>   itests/qtest-accumulo/pom.xml 40d0a749c8 
>   itests/qtest-spark/pom.xml a506f7f081 
>   itests/qtest/pom.xml 02664f3792 
>   itests/util/pom.xml e6dc09fb8f 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
> 0cc9a89085 
>   itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
> 4f8fa05925 
>   llap-server/pom.xml 47a04cc310 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
>  6b08da66b6 
>   metastore/pom.xml 04c6f47879 
>   pom.xml 006e8f8611 
>   ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
> b582471afd 
>   ql/src/test/results/clientnegative/external1.q.out 661d669206 
>   ql/src/test/results/clientnegative/external2.q.out eb5518c017 
>   serde/pom.xml 7419cfb17a 
>   shims/0.23/pom.xml 3ff1d38776 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> e9445eb11d 
>   shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
> 2c37a51cf4 
>   shims/scheduler/pom.xml 0eadb69435 
>   spark-client/pom.xml 784d908e4a 
>   standalone-metastore/pom.xml 8df622f179 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
>  c10e36f94a 
>

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-31 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/
---

(Updated Oct. 31, 2017, 9:56 p.m.)


Review request for hive.


Changes
---

Address comments.


Repository: hive-git


Description
---

HIVE-15016: Run tests with Hadoop 3.0.0-beta1


Diffs (updated)
-

  common/pom.xml 868e14dbc3 
  hbase-handler/pom.xml 7f57b77168 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
c6f3b0ff3a 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
9cad97ad4b 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 6054d533f0 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 1ef454572c 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
 4b8f62c913 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 
a25a96faa9 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
b35aea9f60 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
f244ed6562 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
b2bdd19964 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java 
561b0a8952 
  hbase-handler/src/test/queries/positive/hbase_bulk.q 475aafc1ce 
  hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 85581ecdac 
  hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 1f42567a4b 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
4ace16ca6f 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
 22d2cc6bff 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
 d69a7339c1 
  itests/hcatalog-unit/pom.xml bb6b10554f 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
 745aa999f9 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
 4e1384a634 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
 f8f18b3514 
  itests/hive-minikdc/pom.xml 95d2614901 
  itests/hive-unit-hadoop2/pom.xml 339a194f2d 
  itests/hive-unit/pom.xml 14409832c8 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 707bcd10b7 
  itests/qtest-accumulo/pom.xml 40d0a749c8 
  itests/qtest-spark/pom.xml a506f7f081 
  itests/qtest/pom.xml 02664f3792 
  itests/util/pom.xml e6dc09fb8f 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
0cc9a89085 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
4f8fa05925 
  llap-server/pom.xml 47a04cc310 
  
llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
 6b08da66b6 
  metastore/pom.xml 04c6f47879 
  pom.xml 006e8f8611 
  ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
b582471afd 
  ql/src/test/results/clientnegative/external1.q.out 661d669206 
  ql/src/test/results/clientnegative/external2.q.out eb5518c017 
  serde/pom.xml 7419cfb17a 
  shims/0.23/pom.xml 3ff1d38776 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
e9445eb11d 
  shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
2c37a51cf4 
  shims/scheduler/pom.xml 0eadb69435 
  spark-client/pom.xml 784d908e4a 
  standalone-metastore/pom.xml 8df622f179 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/HdfsUtils.java
 c10e36f94a 
  testutils/ptest2/src/main/resources/batch-exec.vm 2d16ca3e8a 


Diff: https://reviews.apache.org/r/63394/diff/2/

Changes: https://reviews.apache.org/r/63394/diff/1-2/


Testing
---


Thanks,

Aihua Xu

Re: Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-31 Thread Aihua Xu via Review Board 


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/pom.xml
> > Line 82 (original), 82 (patched)
> > 
> >
> > scope=test?

Actually the code does have a reference to it. Not only for test.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
> > Lines 46 (patched)
> > 
> >
> > Might be a good idea to extend DefaultHiveMetaHook to take advantage of 
> > other extension points.

I'm not very faimilar with the implementation for insert operation. So I will 
delay that to later jira.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
> > Lines 57 (patched)
> > 
> >
> > Any reason to not do this in this patch?

Actually they are closed.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java
> > Line 742 (original), 742 (patched)
> > 
> >
> > Do we need to retain this for 0.23? That support is long gone :)

I will leave this cleanup for later jira since seems other places also have 
0.23 support.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
> > Lines 106 (patched)
> > 
> >
> > Is this test only or on cluster also we need to set up this property?

This is needed for the cluster as well so habase will pick up right native lib.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > pom.xml
> > Line 841 (original), 907 (patched)
> > 
> >
> > I don't think we need to depend on hbase-sever. If this is only for 
> > unit tests, shall we add test

Actually we need for the production code.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > serde/pom.xml
> > Line 155 (original), 161 (patched)
> > 
> >
> > Do we need this as well? Proabably not.

It's actually needed.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > shims/0.23/pom.xml
> > Line 67 (original), 73 (patched)
> > 
> >
> > This shall not be needed.

This is needed for getMiniDfs() call.


> On Oct. 30, 2017, 4:28 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/pom.xml
> > Line 179 (original), 179 (patched)
> > 
> >
> > Probably not needed.

It's actually needed.


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/#review189595
---


On Oct. 28, 2017, 11:54 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63394/
> ---
> 
> (Updated Oct. 28, 2017, 11:54 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15016: Run tests with Hadoop 3.0.0-beta1
> 
> 
> Diffs
> -
> 
>   common/pom.xml 868e14dbc321be1e4008ba1575568bbc81032791 
>   hbase-handler/pom.xml 7f57b771686aeb09d4d58a4ae591110c10260519 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
> PRE-CREATION 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
> c6f3b0ff3abfd78bb4139eb471beeb0b66211f2a 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
> 9cad97ad4b1a2654b8865d10645963cbe69eaa33 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
>  6054d533f00d4a8ffe4a3649486d0dfe1e19ae68 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
>  1ef454572c258983337a63a27415de990d56aa6f 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
>  4b8f62c913bf5ec0b1ee2cb7408c3da1d0785777 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  a25a96faa99ee751fc8a25cce2e1a0af67e015c4 
>   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
> b35aea9f60524640cd546d15cd88bacc28ceb36b 
>   hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
>

Review Request 63394: HIVE-15016: Run tests with Hadoop 3.0.0-beta1

2017-10-28 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63394/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-15016: Run tests with Hadoop 3.0.0-beta1


Diffs
-

  common/pom.xml 868e14dbc321be1e4008ba1575568bbc81032791 
  hbase-handler/pom.xml 7f57b771686aeb09d4d58a4ae591110c10260519 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
c6f3b0ff3abfd78bb4139eb471beeb0b66211f2a 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
9cad97ad4b1a2654b8865d10645963cbe69eaa33 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 6054d533f00d4a8ffe4a3649486d0dfe1e19ae68 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 1ef454572c258983337a63a27415de990d56aa6f 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
 4b8f62c913bf5ec0b1ee2cb7408c3da1d0785777 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 
a25a96faa99ee751fc8a25cce2e1a0af67e015c4 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java 
b35aea9f60524640cd546d15cd88bacc28ceb36b 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java 
f244ed656271c4db2f742bea6334d8ab1fd5bb89 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
b2bdd19964b041d4a6fe0f75d8836663e106e34d 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestPutResultWritable.java 
561b0a8952448fc40ee5d4e9a9ab040d95370cf3 
  hbase-handler/src/test/queries/positive/hbase_bulk.q 
475aafc1ce00f394481a7c10e2c3adc93341225c 
  hbase-handler/src/test/queries/positive/hbase_handler_bulk.q 
85581ecdacb3e0541c21634b509cf60001797262 
  hbase-handler/src/test/results/positive/hbase_handler_bulk.q.out 
1f42567a4b269a62138c5428e31e2ee112444295 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 
4ace16ca6f24cf2d28846057ddba3a3877ab021b 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestWebHCatE2e.java
 22d2cc6bff5ede76e492efcaca1a28c77e1cd539 
  
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/mock/MockUriInfo.java
 d69a7339c106182fc5f51d96aea0f925931c4c04 
  itests/hcatalog-unit/pom.xml bb6b10554fa768f1d50fdcae15fc1a0f93f37c64 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java
 745aa999f92d2f2c56d444f990482697e9f960f1 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/SkeletonHBaseTest.java
 4e1384a634b21cac4cf754fca4b09b446b9e82bf 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/TestPigHBaseStorageHandler.java
 f8f18b3514e96eaea57ac924b2d3f5c0b95c154c 
  itests/hive-minikdc/pom.xml 95d2614901a247203487e9082b71c881efc1098d 
  itests/hive-unit-hadoop2/pom.xml 339a194f2dfd6ad9765e225b26d72cdc929d5687 
  itests/hive-unit/pom.xml 14409832c80f5fcb5d633d9e0f283f40f33e35d4 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 707bcd10b759ea3783397495a19c99422b0f33fa 
  itests/qtest-accumulo/pom.xml 40d0a749c8406bacb66b92ffc063a4bf781143f2 
  itests/qtest-spark/pom.xml a506f7f081f80f9c3e632d4b80d434b6c71f8c57 
  itests/qtest/pom.xml 02664f3792e1512cc3f051089fbebc73a4ed39bc 
  itests/util/pom.xml e6dc09fb8f4bcc0408e2d4a1694661f2f7f13bef 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
0cc9a890850f7a2652a001f33161d0de5a03bfa2 
  itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseTestSetup.java 
4f8fa05925ac26e78817ba1aa0957db19b53ccd5 
  llap-server/pom.xml 47a04cc31018689c71c39ea48b392406df767c66 
  
llap-server/src/java/org/apache/hadoop/hive/llap/shufflehandler/ShuffleHandler.java
 6b08da66b64e245a9a3679a934a7a22aff99b37b 
  metastore/pom.xml 04c6f47879a4736f7b73199f61709140ad1b8dfd 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
0e24ba70b6c60cfb5f6eb8ff705f01f69c7a8f5e 
  pom.xml 006e8f8611f4f866ef0e55a61930b6165263b677 
  ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 
b582471afd19680efd23f1a318e5d430d47593ed 
  ql/src/test/results/clientnegative/external1.q.out 
661d669206aa58522ef2efeaaa48ce275745a9d3 
  ql/src/test/results/clientnegative/external2.q.out 
eb5518c017160507bd5584ba62558a348d3230f3 
  serde/pom.xml 7419cfb17ae4f52edb36e90ff3ef96e43a0072e1 
  shims/0.23/pom.xml 3ff1d38776ddf1b14dcf9aeb695d7316c39119fd 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
e9445eb11dd74f24167f1e156c824e5d688fdd87 
  shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 
2c37a51cf41fc78a9d1226d8ef09aa010cf799f1 
  shims/scheduler/pom.xml

Re: Review Request 63043: HIVE-17730 Queries can be closed automatically

2017-10-18 Thread Aihua Xu via Review Board 


> On Oct. 17, 2017, 11:25 p.m., Aihua Xu wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 261 (patched)
> > 
> >
> > OK. I started to recall why we did that this way. I think we don't want 
> > to call query.close() in the subclass since when you return query.execute() 
> > actually DN doesn't get the result yet until you consume those results. 
> > 
> > That's why we want the caller to be responsible to call close() 
> > explictly and the caller knows that it has consumed all the data. 
> > 
> > The new implmentation will close the query object inside the subclass 
> > right?
> 
> Alexander Kolbasov wrote:
> Originally Query wasn't Closeable or AutoCLoseable, so you couldn't use 
> it anyway. Now it is and it could be possible to use Query directly, but this 
> raises a problem of handling exceptions on close.
> 
> Answering the second part of your question. The query now is closed 
> immediately after the use (in subclass that creates the query). Query results 
> can be used after query close, but I need to copy them into another 
> collection - that's why there are a bunch of changes like `return new 
> ArrayList<>(result)`.
> 
> Aihua Xu wrote:
> I see. I believe the original solution is to avoid the memory pressure on 
> HMS. With your approach, we are prefetching all the data and save in a 
> collection, while the original approach delays the data retrieval until it's 
> needed. 
> 
> Is that possible to preserve that logic but still let it close 
> automatically?
> 
> Alexander Kolbasov wrote:
> Note that we are not copying the data itself in the new ArrayList, we are 
> copying pointers. So the only extra overhead is the newly allocated array 
> which is mostly a short-living object and not a big deal. The prefetches are 
> present in the old code - I didn't add any new ones.
> 
> Doing what you suggest requires bigger code refactoring since 
> try-with-resource is based on try-blocks, so we need to significantly shuffle 
> the code around.

Maybe I didn't state clearly. When you call query.execute(), what I have 
noticed that DN actually didn't really execute the query. It will only return 
an object and wait until the result is getting consumed and DN will do the 
actual query against the database.

So for the following code, I remember query.execute() doesn't return any data 
for you at that time yet. Now if new ArrayList<> retrieves the data and it 
triggers the actual execution, then you are fetching all the data at a time 
(cause memory pressure); if it doesn't and if q.close() is called, then the 
List you returned to the parent is not accessible since the query is closed.

you will have either issue. That's what I remembered from the past. Can you 
check test failure TestHiveMetaTool.testExecuteJDOQL if it's related to this? 
Right now we are using directSQL access, so we may hide these DN issues from 
the tests.

private List sub() {

  try (queryWrapper q) {
  
 return new ArrayList<>((List) query.execute(dbName, tableName, 
partNameMatcher));
  }
}


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63043/#review188419
---


On Oct. 16, 2017, 6:37 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63043/
> ---
> 
> (Updated Oct. 16, 2017, 6:37 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Alan Gates, Janaki 
> Lahorani, Sergio Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: 17730
> https://issues.apache.org/jira/browse/17730
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17730 Queries can be closed automatically
> 
> The goal of the patch is to replaces all curret uses of QueryWrapper with 
> try-with-resource construct that guarantees that the
> query is closed.
> 
> This change affects the scope of the open query, so in all cases where the 
> code was returning the list of query results, I had to copy the list to a new 
> ArrayList to allow access after query close.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java 
> 22e246f1c914532ed6c67151c261fa709a283ef2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63043/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 63043: HIVE-17730 Queries can be closed automatically

2017-10-18 Thread Aihua Xu via Review Board 


> On Oct. 17, 2017, 11:25 p.m., Aihua Xu wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 261 (patched)
> > 
> >
> > OK. I started to recall why we did that this way. I think we don't want 
> > to call query.close() in the subclass since when you return query.execute() 
> > actually DN doesn't get the result yet until you consume those results. 
> > 
> > That's why we want the caller to be responsible to call close() 
> > explictly and the caller knows that it has consumed all the data. 
> > 
> > The new implmentation will close the query object inside the subclass 
> > right?
> 
> Alexander Kolbasov wrote:
> Originally Query wasn't Closeable or AutoCLoseable, so you couldn't use 
> it anyway. Now it is and it could be possible to use Query directly, but this 
> raises a problem of handling exceptions on close.
> 
> Answering the second part of your question. The query now is closed 
> immediately after the use (in subclass that creates the query). Query results 
> can be used after query close, but I need to copy them into another 
> collection - that's why there are a bunch of changes like `return new 
> ArrayList<>(result)`.

I see. I believe the original solution is to avoid the memory pressure on HMS. 
With your approach, we are prefetching all the data and save in a collection, 
while the original approach delays the data retrieval until it's needed. 

Is that possible to preserve that logic but still let it close automatically?


- Aihua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63043/#review188419
---


On Oct. 16, 2017, 6:37 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63043/
> ---
> 
> (Updated Oct. 16, 2017, 6:37 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Alan Gates, Janaki 
> Lahorani, Sergio Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: 17730
> https://issues.apache.org/jira/browse/17730
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17730 Queries can be closed automatically
> 
> The goal of the patch is to replaces all curret uses of QueryWrapper with 
> try-with-resource construct that guarantees that the
> query is closed.
> 
> This change affects the scope of the open query, so in all cases where the 
> code was returning the list of query results, I had to copy the list to a new 
> ArrayList to allow access after query close.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java 
> 22e246f1c914532ed6c67151c261fa709a283ef2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63043/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 63043: HIVE-17730 Queries can be closed automatically

2017-10-17 Thread Aihua Xu via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63043/#review188419
---




standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 261 (patched)


OK. I started to recall why we did that this way. I think we don't want to 
call query.close() in the subclass since when you return query.execute() 
actually DN doesn't get the result yet until you consume those results. 

That's why we want the caller to be responsible to call close() explictly 
and the caller knows that it has consumed all the data. 

The new implmentation will close the query object inside the subclass right?


- Aihua Xu


On Oct. 16, 2017, 6:37 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63043/
> ---
> 
> (Updated Oct. 16, 2017, 6:37 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Andrew Sherman, Alan Gates, Janaki 
> Lahorani, Sergio Pena, Sahil Takiar, and Vihang Karajgaonkar.
> 
> 
> Bugs: 17730
> https://issues.apache.org/jira/browse/17730
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17730 Queries can be closed automatically
> 
> The goal of the patch is to replaces all curret uses of QueryWrapper with 
> try-with-resource construct that guarantees that the
> query is closed.
> 
> This change affects the scope of the open query, so in all cases where the 
> code was returning the list of query results, I had to copy the list to a new 
> ArrayList to allow access after query close.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java 
> 22e246f1c914532ed6c67151c261fa709a283ef2 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  ffb2abdf6294dbd470525ad888bd95feac7e7f06 
> 
> 
> Diff: https://reviews.apache.org/r/63043/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>