Re: Review Request 34393: HIVE-10427 - collect_list() and collect_set() should accept struct types as argument

[Chao Sun]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34393/
---

(Updated May 21, 2015, 6:44 a.m.)


Review request for hive.


Bugs: HIVE-10427
https://issues.apache.org/jira/browse/HIVE-10427


Repository: hive-git


Description (updated)
---

Currently for collect_list() and collect_set(), only primitive types are 
supported. This patch adds support for struct, list and map types as well.

It turned out I that all I need is loosen the type checking.


Diffs
-

  data/files/customers.txt PRE-CREATION 
  data/files/nested_orders.txt PRE-CREATION 
  data/files/orders.txt PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java 
536c4a7 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java 
6dc424a 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
 efcc8f5 
  ql/src/test/queries/clientnegative/udaf_collect_set_unsupported.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udaf_collect_set_2.q PRE-CREATION 
  ql/src/test/results/clientnegative/udaf_collect_set_unsupported.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/udaf_collect_set_2.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/34393/diff/


Testing (updated)
---

All but one test (which seems unrelated) are passing.
I also added a test: udaf_collect_list_set_2.q


Thanks,

Chao Sun

Re: Review Request 34393: HIVE-10427 - collect_list() and collect_set() should accept struct types as argument

[Chao Sun]

> On May 19, 2015, 5:36 a.m., Lenni Kuff wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java,
> >  line 50
> > 
> >
> > should we also support arrays and unions?

Added support for array. union seems a bit tricky - let's make that as a follow 
up task.


- Chao


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34393/#review84260
---


On May 19, 2015, 4:47 a.m., Chao Sun wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34393/
> ---
> 
> (Updated May 19, 2015, 4:47 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-10427
> https://issues.apache.org/jira/browse/HIVE-10427
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently for collect_list() and collect_set(), only primitive types are 
> supported. This patch adds support for struct and map types as well.
> 
> It turned out I that all I need is loosen the type checking.
> 
> 
> Diffs
> -
> 
>   data/files/customers.txt PRE-CREATION 
>   data/files/nested_orders.txt PRE-CREATION 
>   data/files/orders.txt PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java 
> 536c4a7 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java 
> 6dc424a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
>  efcc8f5 
>   ql/src/test/queries/clientpositive/udaf_collect_list_set_nested.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/udaf_collect_list_set_nested.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/34393/diff/
> 
> 
> Testing
> ---
> 
> All but one test (which seems unrelated) are passing.
> I also added a test: udaf_collect_list_set_nested.q
> 
> 
> Thanks,
> 
> Chao Sun
> 
>

Re: Review Request 34393: HIVE-10427 - collect_list() and collect_set() should accept struct types as argument

[Chao Sun]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34393/
---

(Updated May 21, 2015, 6:44 a.m.)


Review request for hive.


Changes
---

Addressing RB comments.


Bugs: HIVE-10427
https://issues.apache.org/jira/browse/HIVE-10427


Repository: hive-git


Description
---

Currently for collect_list() and collect_set(), only primitive types are 
supported. This patch adds support for struct and map types as well.

It turned out I that all I need is loosen the type checking.


Diffs (updated)
-

  data/files/customers.txt PRE-CREATION 
  data/files/nested_orders.txt PRE-CREATION 
  data/files/orders.txt PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java 
536c4a7 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java 
6dc424a 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
 efcc8f5 
  ql/src/test/queries/clientnegative/udaf_collect_set_unsupported.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udaf_collect_set_2.q PRE-CREATION 
  ql/src/test/results/clientnegative/udaf_collect_set_unsupported.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/udaf_collect_set_2.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/34393/diff/


Testing
---

All but one test (which seems unrelated) are passing.
I also added a test: udaf_collect_list_set_nested.q


Thanks,

Chao Sun

[jira] [Created] (HIVE-10779) LLAP: Daemons should shutdown in case of fatal errors

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-10779:
-

 Summary: LLAP: Daemons should shutdown in case of fatal errors
 Key: HIVE-10779
 URL: https://issues.apache.org/jira/browse/HIVE-10779
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


For example, the scheduler loop exiting. Currently they end up getting stuck - 
while still accepting new work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] hive pull request: Fix bug when OrcFileStripeMergeRecordReader get...

[FanTn]

GitHub user FanTn opened a pull request:

https://github.com/apache/hive/pull/37

Fix bug when OrcFileStripeMergeRecordReader get nextStripe



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/FanTn/hive HIVE-10685

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/37.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #37


commit f5495724f7655223ea9b328b6176a5447fbaff3a
Author: FanTn 
Date:   2015-05-21T05:39:30Z

Fix bug when OrcFileStripeMergeRecordReader get nextStripe




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Review Request 34522: HIVE-10748 Replace StringBuffer with StringBuilder where possible

[Alexander Pivovarov]

> On May 21, 2015, 5:06 a.m., Swarnim Kulkarni wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line 
> > 1833
> > 
> >
> > Out of curiosity, any specific reason for this change?

I changed aggregationKey type from StringBuffer to Appendable on line 1818 
(it's public API. I can not change aggregationKey to StringBuilder. Appendable 
is an interface implemented by both classes.)

Appendable.append() signature has "throws IOException".
StringBuffer.append() does not have it.

This is why line 1829 aggregationKey.append() throw s IOException now which I 
wrap with RuntimeException


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34522/#review84657
---


On May 21, 2015, 2:22 a.m., Alexander Pivovarov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34522/
> ---
> 
> (Updated May 21, 2015, 2:22 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-10748
> https://issues.apache.org/jira/browse/HIVE-10748
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-10748 Replace StringBuffer with StringBuilder where possible
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
>  6d6bbc2ee2bca67645356089046a39a3b6969df0 
>   common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java 
> 012c28b1a0024b7292a97076f42de1097dae6b2a 
>   common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java 
> 309d0427da3f17a85d16da0e0dca46ad29a1c48e 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatException.java
>  265d08dec6d3e260adfadfe7f629592ebeb5039d 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
> 2947c4333b925e0beabd8a85b188419a4d71a2e3 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
>  eae91cbd79ebb47e59263e8e47b8acdb457d576d 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3c2548635b95509da8cbdf474149c01da0662bbb 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  329e5da5c4675ad3d5f57fbdbddfc5ea168a6dbe 
>   jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 
> 8a0671fc28c4e8326df068f7de5cf278c863e362 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  52147bcbd0bd214b62e52d4ed2a6775e04a94143 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 
> 835015f249684820a9f0eb453d3316a98af52e00 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> 7b48b8b87a0c54f482c32e460930978b691bcdb5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 
> 4423cd1a9960c68b74788f41e386bea105cee4eb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
> 4a16b4c196c7080b1ec64d8ffdc25f359698b4d6 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> c5f03d94672a80849400e51a238bcec1db56659d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> acd9bf5017ca23578616a5bd9b902d2c2abed1ef 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> f7e1dbce4ef1c985b8f2987df413aed0ab087051 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 0de74882f3b92aa979c1960ac64023d3c750b876 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
>  e477f04d83715f5488e72bddd8527728faeb6789 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
> 7108a47676a6a8e2765f098c1799d08e587db58e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 086d9a2b1740a8dc8560667c19826b7dff6cb75b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java
>  9e197331bffb8db4b02aa5d5d842d68d55f7001a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> 8dff2fcee46a4d366bef559576348e9ea8ef6336 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
> 87a25480740df061e0918228d71dd9ec8e08a275 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> aa291b9b1f704c682c82d85675c5de17f3965403 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b8e18eafb67307c9b974194de28482fa8a7c6f2a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 847d75199d6d614bd17ea852a4e3e87bf6911be7 
>   ql/src/java/org/apache/hadoop/hive/ql/

Re: Review Request 34522: HIVE-10748 Replace StringBuffer with StringBuilder where possible

[Alexander Pivovarov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34522/#review84659
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java


I changed aggregationKey type from StringBuffer to Appendable on line 1818 
(it's public API. I can not change aggregationKey to StringBuilder. Appendable 
is an interface implemented by both classes.)

Appendable.append() signature has "throws IOException".
StringBuffer.append() does not have it.

This is why line 1829 aggregationKey.append() throw s IOException now which 
I wrap with RuntimeException


- Alexander Pivovarov


On May 21, 2015, 2:22 a.m., Alexander Pivovarov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34522/
> ---
> 
> (Updated May 21, 2015, 2:22 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-10748
> https://issues.apache.org/jira/browse/HIVE-10748
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-10748 Replace StringBuffer with StringBuilder where possible
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
>  6d6bbc2ee2bca67645356089046a39a3b6969df0 
>   common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java 
> 012c28b1a0024b7292a97076f42de1097dae6b2a 
>   common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java 
> 309d0427da3f17a85d16da0e0dca46ad29a1c48e 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatException.java
>  265d08dec6d3e260adfadfe7f629592ebeb5039d 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
> 2947c4333b925e0beabd8a85b188419a4d71a2e3 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
>  eae91cbd79ebb47e59263e8e47b8acdb457d576d 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3c2548635b95509da8cbdf474149c01da0662bbb 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  329e5da5c4675ad3d5f57fbdbddfc5ea168a6dbe 
>   jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 
> 8a0671fc28c4e8326df068f7de5cf278c863e362 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  52147bcbd0bd214b62e52d4ed2a6775e04a94143 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 
> 835015f249684820a9f0eb453d3316a98af52e00 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> 7b48b8b87a0c54f482c32e460930978b691bcdb5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 
> 4423cd1a9960c68b74788f41e386bea105cee4eb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
> 4a16b4c196c7080b1ec64d8ffdc25f359698b4d6 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> c5f03d94672a80849400e51a238bcec1db56659d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> acd9bf5017ca23578616a5bd9b902d2c2abed1ef 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> f7e1dbce4ef1c985b8f2987df413aed0ab087051 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 0de74882f3b92aa979c1960ac64023d3c750b876 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
>  e477f04d83715f5488e72bddd8527728faeb6789 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
> 7108a47676a6a8e2765f098c1799d08e587db58e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 086d9a2b1740a8dc8560667c19826b7dff6cb75b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java
>  9e197331bffb8db4b02aa5d5d842d68d55f7001a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> 8dff2fcee46a4d366bef559576348e9ea8ef6336 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
> 87a25480740df061e0918228d71dd9ec8e08a275 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> aa291b9b1f704c682c82d85675c5de17f3965403 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b8e18eafb67307c9b974194de28482fa8a7c6f2a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 847d75199d6d614bd17ea852a4e3e87bf6911be7 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
> f26225a72c34252c8fdf615bd34b59532376c5de 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspec

Re: Review Request 34522: HIVE-10748 Replace StringBuffer with StringBuilder where possible

[Swarnim Kulkarni]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34522/#review84657
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java


Out of curiosity, any specific reason for this change?


- Swarnim Kulkarni


On May 21, 2015, 2:22 a.m., Alexander Pivovarov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34522/
> ---
> 
> (Updated May 21, 2015, 2:22 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-10748
> https://issues.apache.org/jira/browse/HIVE-10748
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-10748 Replace StringBuffer with StringBuilder where possible
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
>  6d6bbc2ee2bca67645356089046a39a3b6969df0 
>   common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java 
> 012c28b1a0024b7292a97076f42de1097dae6b2a 
>   common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java 
> 309d0427da3f17a85d16da0e0dca46ad29a1c48e 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatException.java
>  265d08dec6d3e260adfadfe7f629592ebeb5039d 
>   
> hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
> 2947c4333b925e0beabd8a85b188419a4d71a2e3 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
>  eae91cbd79ebb47e59263e8e47b8acdb457d576d 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
>  3c2548635b95509da8cbdf474149c01da0662bbb 
>   
> hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
>  329e5da5c4675ad3d5f57fbdbddfc5ea168a6dbe 
>   jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 
> 8a0671fc28c4e8326df068f7de5cf278c863e362 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  52147bcbd0bd214b62e52d4ed2a6775e04a94143 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 
> 835015f249684820a9f0eb453d3316a98af52e00 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
> 7b48b8b87a0c54f482c32e460930978b691bcdb5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 
> 4423cd1a9960c68b74788f41e386bea105cee4eb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
> 4a16b4c196c7080b1ec64d8ffdc25f359698b4d6 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> c5f03d94672a80849400e51a238bcec1db56659d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> acd9bf5017ca23578616a5bd9b902d2c2abed1ef 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> f7e1dbce4ef1c985b8f2987df413aed0ab087051 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 0de74882f3b92aa979c1960ac64023d3c750b876 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
>  e477f04d83715f5488e72bddd8527728faeb6789 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
> 7108a47676a6a8e2765f098c1799d08e587db58e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 086d9a2b1740a8dc8560667c19826b7dff6cb75b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java
>  9e197331bffb8db4b02aa5d5d842d68d55f7001a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> 8dff2fcee46a4d366bef559576348e9ea8ef6336 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
> 87a25480740df061e0918228d71dd9ec8e08a275 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> aa291b9b1f704c682c82d85675c5de17f3965403 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b8e18eafb67307c9b974194de28482fa8a7c6f2a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 847d75199d6d614bd17ea852a4e3e87bf6911be7 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
> f26225a72c34252c8fdf615bd34b59532376c5de 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
>  f3f7d95ef90f3e4f1beacecb4d681030bd69a231 
>   serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 
> 19fe952f5e84755d1e7a8b752997c084dab339b9 
>   service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 
> 3ef55779a6bde85193ca63ec9683cf9f67d6a39d 
> 
> Diff: https://reviews.apache.org/r/34522/diff/
> 
> 
> T

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Chinna Rao Lalam]

Congratulations Chaoyu.

On Thu, May 21, 2015 at 9:42 AM, Naveen Gangam  wrote:

> Congrats Chaoyu,
> Well deserved indeed!!!
>
> On Wed, May 20, 2015 at 10:04 PM, Yongzhi Chen  wrote:
>
> > Congratulations Chaoyu!
> >
> > On Wed, May 20, 2015 at 8:10 PM, Lefty Leverenz  >
> > wrote:
> >
> > > Congratulations Chaoyu!  Thanks for all your contributions.
> > >
> > > -- Lefty
> > >
> > > On Wed, May 20, 2015 at 4:11 PM, Hari Subramaniyan <
> > > hsubramani...@hortonworks.com> wrote:
> > >
> > > > Congrats Chaoyu!
> > > > 
> > > > From: Lenni Kuff 
> > > > Sent: Wednesday, May 20, 2015 4:08 PM
> > > > To: dev@hive.apache.org
> > > > Subject: Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang
> > > >
> > > > Congrats Chaoyu! Well deserved.
> > > >
> > > > On Wed, May 20, 2015 at 4:07 PM, Sushanth Sowmyan <
> khorg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Congrats Chaoyu, welcome aboard! :)
> > > > > On May 20, 2015 3:45 PM, "Vaibhav Gumashta" <
> > vgumas...@hortonworks.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > ‹Vaibhav
> > > > > >
> > > > > > On 5/20/15, 3:40 PM, "Jimmy Xiang"  wrote:
> > > > > >
> > > > > > >Congrats!!
> > > > > > >
> > > > > > >On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach  >
> > > > wrote:
> > > > > > >
> > > > > > >> The Apache Hive PMC has voted to make Chaoyu Tang a committer
> on
> > > the
> > > > > > >>Apache
> > > > > > >> Hive Project.
> > > > > > >>
> > > > > > >> Please join me in congratulating Chaoyu!
> > > > > > >>
> > > > > > >> Thanks.
> > > > > > >>
> > > > > > >> - Carl
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Hope It Helps,
Chinna

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Naveen Gangam]

Congrats Chaoyu,
Well deserved indeed!!!

On Wed, May 20, 2015 at 10:04 PM, Yongzhi Chen  wrote:

> Congratulations Chaoyu!
>
> On Wed, May 20, 2015 at 8:10 PM, Lefty Leverenz 
> wrote:
>
> > Congratulations Chaoyu!  Thanks for all your contributions.
> >
> > -- Lefty
> >
> > On Wed, May 20, 2015 at 4:11 PM, Hari Subramaniyan <
> > hsubramani...@hortonworks.com> wrote:
> >
> > > Congrats Chaoyu!
> > > 
> > > From: Lenni Kuff 
> > > Sent: Wednesday, May 20, 2015 4:08 PM
> > > To: dev@hive.apache.org
> > > Subject: Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang
> > >
> > > Congrats Chaoyu! Well deserved.
> > >
> > > On Wed, May 20, 2015 at 4:07 PM, Sushanth Sowmyan 
> > > wrote:
> > >
> > > > Congrats Chaoyu, welcome aboard! :)
> > > > On May 20, 2015 3:45 PM, "Vaibhav Gumashta" <
> vgumas...@hortonworks.com
> > >
> > > > wrote:
> > > >
> > > > > Congratulations!
> > > > >
> > > > > ‹Vaibhav
> > > > >
> > > > > On 5/20/15, 3:40 PM, "Jimmy Xiang"  wrote:
> > > > >
> > > > > >Congrats!!
> > > > > >
> > > > > >On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach 
> > > wrote:
> > > > > >
> > > > > >> The Apache Hive PMC has voted to make Chaoyu Tang a committer on
> > the
> > > > > >>Apache
> > > > > >> Hive Project.
> > > > > >>
> > > > > >> Please join me in congratulating Chaoyu!
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >> - Carl
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (HIVE-10778) LLAP: Utilities::gWorkMap needs thread-locals for HiveServer2

[Gopal V (JIRA)]

Gopal V created HIVE-10778:
--

 Summary: LLAP: Utilities::gWorkMap needs thread-locals for 
HiveServer2
 Key: HIVE-10778
 URL: https://issues.apache.org/jira/browse/HIVE-10778
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Affects Versions: llap
Reporter: Gopal V
 Fix For: llap


95% of heap is occupied by the Utilities::gWorkMap in the llap branch HS2.

!llap-hs2-heap.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 34522: HIVE-10748 Replace StringBuffer with StringBuilder where possible

[Alexander Pivovarov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34522/
---

Review request for hive.


Bugs: HIVE-10748
https://issues.apache.org/jira/browse/HIVE-10748


Repository: hive-git


Description
---

HIVE-10748 Replace StringBuffer with StringBuilder where possible


Diffs
-

  
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
 6d6bbc2ee2bca67645356089046a39a3b6969df0 
  common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java 
012c28b1a0024b7292a97076f42de1097dae6b2a 
  common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java 
309d0427da3f17a85d16da0e0dca46ad29a1c48e 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatException.java 
265d08dec6d3e260adfadfe7f629592ebeb5039d 
  hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
2947c4333b925e0beabd8a85b188419a4d71a2e3 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
 eae91cbd79ebb47e59263e8e47b8acdb457d576d 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
 3c2548635b95509da8cbdf474149c01da0662bbb 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 329e5da5c4675ad3d5f57fbdbddfc5ea168a6dbe 
  jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 
8a0671fc28c4e8326df068f7de5cf278c863e362 
  
metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 52147bcbd0bd214b62e52d4ed2a6775e04a94143 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 
835015f249684820a9f0eb453d3316a98af52e00 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
7b48b8b87a0c54f482c32e460930978b691bcdb5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 
4423cd1a9960c68b74788f41e386bea105cee4eb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
4a16b4c196c7080b1ec64d8ffdc25f359698b4d6 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
c5f03d94672a80849400e51a238bcec1db56659d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
acd9bf5017ca23578616a5bd9b902d2c2abed1ef 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
f7e1dbce4ef1c985b8f2987df413aed0ab087051 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
0de74882f3b92aa979c1960ac64023d3c750b876 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
 e477f04d83715f5488e72bddd8527728faeb6789 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
7108a47676a6a8e2765f098c1799d08e587db58e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
086d9a2b1740a8dc8560667c19826b7dff6cb75b 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
9e197331bffb8db4b02aa5d5d842d68d55f7001a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
8dff2fcee46a4d366bef559576348e9ea8ef6336 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
87a25480740df061e0918228d71dd9ec8e08a275 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
aa291b9b1f704c682c82d85675c5de17f3965403 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b8e18eafb67307c9b974194de28482fa8a7c6f2a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
847d75199d6d614bd17ea852a4e3e87bf6911be7 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
f26225a72c34252c8fdf615bd34b59532376c5de 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
 f3f7d95ef90f3e4f1beacecb4d681030bd69a231 
  serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 
19fe952f5e84755d1e7a8b752997c084dab339b9 
  service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 
3ef55779a6bde85193ca63ec9683cf9f67d6a39d 

Diff: https://reviews.apache.org/r/34522/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Yongzhi Chen]

Congratulations Chaoyu!

On Wed, May 20, 2015 at 8:10 PM, Lefty Leverenz 
wrote:

> Congratulations Chaoyu!  Thanks for all your contributions.
>
> -- Lefty
>
> On Wed, May 20, 2015 at 4:11 PM, Hari Subramaniyan <
> hsubramani...@hortonworks.com> wrote:
>
> > Congrats Chaoyu!
> > 
> > From: Lenni Kuff 
> > Sent: Wednesday, May 20, 2015 4:08 PM
> > To: dev@hive.apache.org
> > Subject: Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang
> >
> > Congrats Chaoyu! Well deserved.
> >
> > On Wed, May 20, 2015 at 4:07 PM, Sushanth Sowmyan 
> > wrote:
> >
> > > Congrats Chaoyu, welcome aboard! :)
> > > On May 20, 2015 3:45 PM, "Vaibhav Gumashta"  >
> > > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > ‹Vaibhav
> > > >
> > > > On 5/20/15, 3:40 PM, "Jimmy Xiang"  wrote:
> > > >
> > > > >Congrats!!
> > > > >
> > > > >On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach 
> > wrote:
> > > > >
> > > > >> The Apache Hive PMC has voted to make Chaoyu Tang a committer on
> the
> > > > >>Apache
> > > > >> Hive Project.
> > > > >>
> > > > >> Please join me in congratulating Chaoyu!
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >> - Carl
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: please include me as a Hive contributor

[Lefty Leverenz]

Thanks Thejas!

-- Lefty

On Wed, May 20, 2015 at 6:11 PM, Thejas Nair  wrote:

> I have also removed that no longer necessary step from this page -
> https://cwiki.apache.org/confluence/display/Hive/HowToContribute
>
> On Wed, May 20, 2015 at 6:09 PM, Thejas Nair 
> wrote:
> > Hi Dayue,
> > With recent change in jira settings for hive, you should be able to
> > assign yourself bugs as long as you have a jira account.
> > No other changes are needed.
> > Looking forward to your contributions!
> >
> > Thanks,
> > Thejas
> >
> >
> > 2015-05-20 17:20 GMT-07:00 Dayue Gao :
> >> JIRA Username: gaodayue
> >>
> >> Thanks,
> >> Dayue
> >>
> >>
> >>
>

[jira] [Created] (HIVE-10777) LLAP: add pre-fragment and per-table cache details

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-10777:
---

 Summary: LLAP: add pre-fragment and per-table cache details
 Key: HIVE-10777
 URL: https://issues.apache.org/jira/browse/HIVE-10777
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: llap






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: please include me as a Hive contributor

[Thejas Nair]

I have also removed that no longer necessary step from this page -
https://cwiki.apache.org/confluence/display/Hive/HowToContribute

On Wed, May 20, 2015 at 6:09 PM, Thejas Nair  wrote:
> Hi Dayue,
> With recent change in jira settings for hive, you should be able to
> assign yourself bugs as long as you have a jira account.
> No other changes are needed.
> Looking forward to your contributions!
>
> Thanks,
> Thejas
>
>
> 2015-05-20 17:20 GMT-07:00 Dayue Gao :
>> JIRA Username: gaodayue
>>
>> Thanks,
>> Dayue
>>
>>
>>

Re: please include me as a Hive contributor

[Thejas Nair]

Hi Dayue,
With recent change in jira settings for hive, you should be able to
assign yourself bugs as long as you have a jira account.
No other changes are needed.
Looking forward to your contributions!

Thanks,
Thejas


2015-05-20 17:20 GMT-07:00 Dayue Gao :
> JIRA Username: gaodayue
>
> Thanks,
> Dayue
>
>
>

[jira] [Created] (HIVE-10776) Schema on insert for acid tables throwing Nullpointer exception

[Aswathy Chellammal Sreekumar (JIRA)]

Aswathy Chellammal Sreekumar created HIVE-10776:
---

 Summary: Schema on insert for acid tables throwing Nullpointer 
exception
 Key: HIVE-10776
 URL: https://issues.apache.org/jira/browse/HIVE-10776
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.0
 Environment: Linux, Windows
Reporter: Aswathy Chellammal Sreekumar
Assignee: Eugene Koifman
 Fix For: 1.2.0


Hive schema on insert queries, with select * , are failing with below exception

2015-05-15 19:29:01,278 ERROR [main]: ql.Driver 
(SessionState.java:printError(957)) - FAILED: NullPointerException null
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genReduceSinkPlan(SemanticAnalyzer.java:7257)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBucketingSortingDest(SemanticAnalyzer.java:6100)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6271)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8972)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8863)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9708)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9601)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10037)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:323)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10048)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:207)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
at 
org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Steps to reproduce
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set hive.enforce.bucketing=true;

drop table if exists studenttab10k;
create table studenttab10k (age int, name varchar(50),gpa decimal(3,2));
insert into studenttab10k values(1,'foo', 1.1), (2,'bar', 2.3),(3,'baz', 3.1);


drop table if exists student_acid;
create table student_acid (age int, name varchar(50),gpa decimal(3,2), grade 
int) 
clustered by (age) into 2 buckets
stored as orc
tblproperties ('transactional'='true');

insert into student_acid(name,age,gpa) select * from studenttab10k;




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 34515: ReduceSinkDedup undoes Distinct Rewrite

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34515/
---

Review request for hive and John Pullokkaran.


Bugs: HIVE-10774
https://issues.apache.org/jira/browse/HIVE-10774


Repository: hive-git


Description
---

ReduceSinkDedup undoes Distinct Rewrite


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Context.java a74bbbe 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveConfigContext.java 
0e559e0 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveAlgorithmsConf.java
 83454ea 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveAlgorithmsUtil.java
 6840418 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java
 73c7cac 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 404b759 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 4760a22 
  ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 2a41aae 
  ql/src/test/results/clientpositive/tez/mrr.q.out d90b27f 
  ql/src/test/results/clientpositive/tez/vectorization_limit.q.out 1c5b51f 

Diff: https://reviews.apache.org/r/34515/diff/


Testing
---

Existing tests.


Thanks,

Ashutosh Chauhan

[jira] [Created] (HIVE-10775) Frequent calls to printStackTrace() obscuring legitimate problems

[Andrew Cowie (JIRA)]

Andrew Cowie created HIVE-10775:
---

 Summary: Frequent calls to printStackTrace() obscuring legitimate 
problems
 Key: HIVE-10775
 URL: https://issues.apache.org/jira/browse/HIVE-10775
 Project: Hive
  Issue Type: Test
  Components: Metastore, Query Processor
Reporter: Andrew Cowie
Priority: Minor


When running test suites built on top of libraries that build on top of ... 
that use Hive, the signal to noise ratio with exceptions flying past is 
appalling. Most of this is down to calls to printStackTrace() embedded in this 
library. HIVE-7697 showed someone cleaning that up and replacing with logging 
the exception instead. That seems wise (logging can be redirected by the 
calling test suite).

So, if you don't object, I'll hunt down the calls to printStackTrace() and 
replace them with LOG.warn() instead. I'm about half way through the patch now.

AfC




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10774) ReduceSinkDedup undoes Distinct Rewrite

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-10774:
---

 Summary: ReduceSinkDedup undoes Distinct Rewrite
 Key: HIVE-10774
 URL: https://issues.apache.org/jira/browse/HIVE-10774
 Project: Hive
  Issue Type: Improvement
  Components: CBO, Logical Optimizer
Affects Versions: 1.2.0
Reporter: Mostafa Mokhtar
Assignee: Ashutosh Chauhan


Now distinct rewrite can fire in more cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 34514: HIVE-9875 replace new SimpleDateFormat with DateUtils.getDateFormat

[Alexander Pivovarov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34514/
---

Review request for hive.


Bugs: HIVE-9875
https://issues.apache.org/jira/browse/HIVE-9875


Repository: hive-git


Description
---

HIVE-9875 replace new SimpleDateFormat with DateUtils.getDateFormat


Diffs
-

  common/src/java/org/apache/hive/common/util/DateUtils.java 
c749bcb5e67c1ca23bb55242c362c349993ca185 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
7b48b8b87a0c54f482c32e460930978b691bcdb5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
6a6593c3ac4890d7e169b5af6608aee66679933a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
a5c1463be4e9e8200c5f79783b4426b83e963d66 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 1f5d91df7c4dd13fadc4488472fdd7bf541e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MetaDataExportListener.java 
1739fd2381f995b677899812bf7bea2fea78df5e 
  serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 
a2a6c79f937aed561b062ad9275bb9eae2346c8b 
  serde/src/test/org/apache/hadoop/hive/serde2/io/TestDateWritable.java 
fd95ccf467224b0aad37d8348a1267f2102a7082 
  serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampWritable.java 
5fee0197d4b3c9c7c395ee252167b22f6fda9f6e 

Diff: https://reviews.apache.org/r/34514/diff/


Testing
---


Thanks,

Alexander Pivovarov

please include me as a Hive contributor

[Dayue Gao]

JIRA Username: gaodayue

Thanks,
Dayue

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Lefty Leverenz]

Congratulations Chaoyu!  Thanks for all your contributions.

-- Lefty

On Wed, May 20, 2015 at 4:11 PM, Hari Subramaniyan <
hsubramani...@hortonworks.com> wrote:

> Congrats Chaoyu!
> 
> From: Lenni Kuff 
> Sent: Wednesday, May 20, 2015 4:08 PM
> To: dev@hive.apache.org
> Subject: Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang
>
> Congrats Chaoyu! Well deserved.
>
> On Wed, May 20, 2015 at 4:07 PM, Sushanth Sowmyan 
> wrote:
>
> > Congrats Chaoyu, welcome aboard! :)
> > On May 20, 2015 3:45 PM, "Vaibhav Gumashta" 
> > wrote:
> >
> > > Congratulations!
> > >
> > > ‹Vaibhav
> > >
> > > On 5/20/15, 3:40 PM, "Jimmy Xiang"  wrote:
> > >
> > > >Congrats!!
> > > >
> > > >On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach 
> wrote:
> > > >
> > > >> The Apache Hive PMC has voted to make Chaoyu Tang a committer on the
> > > >>Apache
> > > >> Hive Project.
> > > >>
> > > >> Please join me in congratulating Chaoyu!
> > > >>
> > > >> Thanks.
> > > >>
> > > >> - Carl
> > > >>
> > >
> > >
> >
>

Re: Review Request 34443: HIVE-10723 better logging/etc. for stuck metastore

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34443/
---

(Updated May 20, 2015, 11:11 p.m.)


Review request for hive, Alexander Pivovarov, Sushanth Sowmyan, and Thejas Nair.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eff4d30 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
5d8718c 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java fd61333 
  
metastore/src/java/org/apache/hadoop/hive/metastore/TServerSocketKeepAlive.java 
9ac18dc 
  
metastore/src/java/org/apache/hadoop/hive/metastore/TSetIpAddressProcessor.java 
4a56bfa 
  metastore/src/java/org/apache/hadoop/hive/metastore/TUGIBasedProcessor.java 
ec8d608 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DefaultFetchFormatter.java 37852ef 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 20d0304 

Diff: https://reviews.apache.org/r/34443/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Hari Subramaniyan]

Congrats Chaoyu!

From: Lenni Kuff 
Sent: Wednesday, May 20, 2015 4:08 PM
To: dev@hive.apache.org
Subject: Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

Congrats Chaoyu! Well deserved.

On Wed, May 20, 2015 at 4:07 PM, Sushanth Sowmyan 
wrote:

> Congrats Chaoyu, welcome aboard! :)
> On May 20, 2015 3:45 PM, "Vaibhav Gumashta" 
> wrote:
>
> > Congratulations!
> >
> > ‹Vaibhav
> >
> > On 5/20/15, 3:40 PM, "Jimmy Xiang"  wrote:
> >
> > >Congrats!!
> > >
> > >On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach  wrote:
> > >
> > >> The Apache Hive PMC has voted to make Chaoyu Tang a committer on the
> > >>Apache
> > >> Hive Project.
> > >>
> > >> Please join me in congratulating Chaoyu!
> > >>
> > >> Thanks.
> > >>
> > >> - Carl
> > >>
> >
> >
>

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Lenni Kuff]

Congrats Chaoyu! Well deserved.

On Wed, May 20, 2015 at 4:07 PM, Sushanth Sowmyan 
wrote:

> Congrats Chaoyu, welcome aboard! :)
> On May 20, 2015 3:45 PM, "Vaibhav Gumashta" 
> wrote:
>
> > Congratulations!
> >
> > ‹Vaibhav
> >
> > On 5/20/15, 3:40 PM, "Jimmy Xiang"  wrote:
> >
> > >Congrats!!
> > >
> > >On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach  wrote:
> > >
> > >> The Apache Hive PMC has voted to make Chaoyu Tang a committer on the
> > >>Apache
> > >> Hive Project.
> > >>
> > >> Please join me in congratulating Chaoyu!
> > >>
> > >> Thanks.
> > >>
> > >> - Carl
> > >>
> >
> >
>

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Sushanth Sowmyan]

Congrats Chaoyu, welcome aboard! :)
On May 20, 2015 3:45 PM, "Vaibhav Gumashta" 
wrote:

> Congratulations!
>
> ‹Vaibhav
>
> On 5/20/15, 3:40 PM, "Jimmy Xiang"  wrote:
>
> >Congrats!!
> >
> >On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach  wrote:
> >
> >> The Apache Hive PMC has voted to make Chaoyu Tang a committer on the
> >>Apache
> >> Hive Project.
> >>
> >> Please join me in congratulating Chaoyu!
> >>
> >> Thanks.
> >>
> >> - Carl
> >>
>
>

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Vaibhav Gumashta]

Congratulations!

‹Vaibhav

On 5/20/15, 3:40 PM, "Jimmy Xiang"  wrote:

>Congrats!!
>
>On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach  wrote:
>
>> The Apache Hive PMC has voted to make Chaoyu Tang a committer on the
>>Apache
>> Hive Project.
>>
>> Please join me in congratulating Chaoyu!
>>
>> Thanks.
>>
>> - Carl
>>

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Jimmy Xiang]

Congrats!!

On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach  wrote:

> The Apache Hive PMC has voted to make Chaoyu Tang a committer on the Apache
> Hive Project.
>
> Please join me in congratulating Chaoyu!
>
> Thanks.
>
> - Carl
>

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Chao Sun]

Congrats Chaoyu!!!

On Wed, May 20, 2015 at 3:31 PM, Alexander Pivovarov 
wrote:

> Congrats Chaoyu!
>
> On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach  wrote:
>
> > The Apache Hive PMC has voted to make Chaoyu Tang a committer on the
> Apache
> > Hive Project.
> >
> > Please join me in congratulating Chaoyu!
> >
> > Thanks.
> >
> > - Carl
> >
>

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

[Alexander Pivovarov]

Congrats Chaoyu!

On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach  wrote:

> The Apache Hive PMC has voted to make Chaoyu Tang a committer on the Apache
> Hive Project.
>
> Please join me in congratulating Chaoyu!
>
> Thanks.
>
> - Carl
>

[ANNOUNCE] New Hive Committer - Chaoyu Tang

[Carl Steinbach]

The Apache Hive PMC has voted to make Chaoyu Tang a committer on the Apache
Hive Project.

Please join me in congratulating Chaoyu!

Thanks.

- Carl

Re: Review Request 34463: HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin

[Swarnim Kulkarni]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34463/#review84594
---

Ship it!


Ship It!

- Swarnim Kulkarni


On May 20, 2015, 6:02 p.m., Alexander Pivovarov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34463/
> ---
> 
> (Updated May 20, 2015, 6:02 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-10717
> https://issues.apache.org/jira/browse/HIVE-10717
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/encryption_insert_partition_static.q 
> 9bff0bd34b5ef0cf43f1e6738a6f1bbf5a6f372e 
>   
> ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out
>  8966608d7ec74c66dafc563aa2984efe86cab333 
> 
> Diff: https://reviews.apache.org/r/34463/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Pivovarov
> 
>

[jira] [Created] (HIVE-10773) MapJoinOperator times out on loading HashTable

[frank luo (JIRA)]

frank luo created HIVE-10773:


 Summary: MapJoinOperator times out on loading HashTable
 Key: HIVE-10773
 URL: https://issues.apache.org/jira/browse/HIVE-10773
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.14.0
Reporter: frank luo


When running a map join, depends on data, it might timeout with last two lines 
in the log as below. And when I do "set mapreduce.task.timeout=60;", which 
is defaulted to 30, the query can go through fine. The size of hashtable 
file is roughly 400M. 

2015-05-20 13:27:03,237 INFO [main] 
org.apache.hadoop.hive.ql.exec.MapJoinOperator: *** Load from HashTable for 
input file: 
hdfs://nameservice1/tmp/hive/jluo/2ee8914d-1cef-4af4-aac6-51f64d630346/hive_2015-05-20_13-13-35_335_1565066409090716856-1/-mr-10007/00_0
2015-05-20 13:27:03,237 INFO [main] 
org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load back 1 hashtable file 
from tmp file 
uri:file:/data/12/hadoop/yarn/local/usercache/xxy/appcache/application_1430337284339_2087
/container_1430337284339_2087_01_03/Stage-3.tar.gz/MapJoin-mapfile31--.hashtable
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34495: HIVE-10678 : In QTestGenTask.execute() we should not throw an exception right away if we are unable to clean any old files

[Hari Sankar Sivarama Subramaniyan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34495/
---

(Updated May 20, 2015, 9:41 p.m.)


Review request for hive and Xuefu Zhang.


Changes
---

Thanks Xuefu for the comment, I have addressed it in the new patch.


Repository: hive-git


Description
---

Currently in QTestGenTask.execute(), we have code like :
   if (logFile != null) {
 File lf = new File(logFile);
 if (lf.exists()) {
   if (!lf.delete()) {
 throw new Exception("Could not delete log file " + 
lf.getCanonicalPath());
 }
If the log file gets locked due to some other process, it throws the above 
exception and short circuits the entire test suite. We should provide a 
mechanism to generate alternative new log files/temporary files whenever 
possible to avoid unit test failures.


Diffs (updated)
-

  ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 60c7764 

Diff: https://reviews.apache.org/r/34495/diff/


Testing
---


Thanks,

Hari Sankar Sivarama Subramaniyan

Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34455/#review84572
---



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/CacheTran.java


Currently the storage level is memory+disk. Any reason to change it to 
memory_only?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java


Can we keep the old code around. I understand it's not currently used.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapTran.java


I cannot construct a case where a MapTran would need caching. Do you have 
an example?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java


Can we keep the comment around?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/LocalSparkJobStatus.java


We need to consider if it makes to unpersist rdds explicitly.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkRddCachingResolver.java


Nit: variable name.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkRddCachingResolver.java


Nit: class name. Maybe CommonParentWorkMatcher?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkRddCachingResolver.java


This nested loop doesn't seem efficient. SparkWork is basically a graph. 
Finding a node that has multiple children should be fairly easy using graph 
traversing. An example would be in SplitSparkWorkResolver.

Getting the value for the threshold should be outside the look, 
nevertheless.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkRddCachingResolver.java


We need to consider the case where the statistics may not be present.



ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java


I think we need a better name for this variable, something like cachedWorks 
or worksToCache.



spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java


Do you think it makes sense for us to release the cache as soon as the job 
is completed, as it's done here?


- Xuefu Zhang


On May 20, 2015, 2:37 a.m., chengxiang li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34455/
> ---
> 
> (Updated May 20, 2015, 2:37 a.m.)
> 
> 
> Review request for hive, Chao Sun, Jimmy Xiang, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-10550
> https://issues.apache.org/jira/browse/HIVE-10550
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira description
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 43c53fc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/CacheTran.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java 
> 19d3fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 26cfebd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapTran.java 2170243 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ReduceTran.java e60dfac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 
> 8b15099 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java a774395 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java ee5c78a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 
> 3f240f5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 
> e6c845c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/LocalSparkJobStatus.java
>  5d62596 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
>  8e56263 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkRddCachingResolver.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSkewJoinProcFactory.java
>  5990d17 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SplitSparkWorkResolver.java
>  fb20080 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 
> 19aae70 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java bb5dd79 
>   spark-client/src/main/java/org/apache/hive/spark/client/JobContext.java 
> af6332e 
>   spark-client/src/main/java/org/apache/hive/spar

Re: Review Request 34473: HIVE-10749 Implement Insert statement for parquet

[Alexander Pivovarov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34473/#review84574
---



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java


missing space after comma and before tableProperties



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetStructObjectInspector.java


you can use
final ArrayList list = new 
ArrayList(Collections.nCopies(fields.size(), null));
instead


- Alexander Pivovarov


On May 20, 2015, 2:54 p.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34473/
> ---
> 
> (Updated May 20, 2015, 2:54 p.m.)
> 
> 
> Review request for hive, Alan Gates and Sergio Pena.
> 
> 
> Bugs: HIVE-10749
> https://issues.apache.org/jira/browse/HIVE-10749
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement the insert statement for parquet format.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  000eb38 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java
>  8380117 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
>  4e1820c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/acid/ParquetRawRecordMerger.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/acid/ParquetRecordUpdater.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java
>  43c772f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java
>  0a5edbb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetStructObjectInspector.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java
>  0d32e49 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java
>  5f7f597 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/acid/TestParquetRecordUpdater.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/acid_parquet_insert.q PRE-CREATION 
>   ql/src/test/results/clientpositive/acid_parquet_insert.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/34473/diff/
> 
> 
> Testing
> ---
> 
> Newly added qtest and UT passed locally
> 
> 
> Thanks,
> 
> cheng xu
> 
>

Re: Review Request 34143: Fix stats annotation

[pengcheng xiong]

> On May 20, 2015, 4:26 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out, line 
> > 397
> > 
> >
> > Seems like cloning bug? Says Map 5, but its really Map 3.

confirmed with Tez plan, it is really "Map 5 => 1"


> On May 20, 2015, 4:26 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out, line 
> > 735
> > 
> >
> > GBy followed by gby has double the # of rows. Expected it to go lower.

GB33 has two parents, GB13 and GB28, thus the #row is the sum of them.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34143/#review84434
---


On May 14, 2015, 4:50 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34143/
> ---
> 
> (Updated May 14, 2015, 4:50 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a umbrella patch for a bunch of issues: HIVE-8769 Physical optimizer 
> : Incorrect CE results in a shuffle join instead of a Map join (PK/FK pattern 
> not detected) HIVE-9392 JoinStatsRule miscalculates join cardinality as 
> incorrect NDV is used due to column names having duplicated fqColumnName 
> HIVE-10107 Union All : Vertex missing stats resulting in OOM and in-efficient 
> plans
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/external_table_ppd.q.out 6d48edb 
>   hbase-handler/src/test/results/positive/hbase_custom_key2.q.out c9b5a84 
>   hbase-handler/src/test/results/positive/hbase_custom_key3.q.out 76848e0 
>   hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out 6174bfb 
>   hbase-handler/src/test/results/positive/hbase_pushdown.q.out 8a979bf 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 7863f69 
>   hbase-handler/src/test/results/positive/hbase_timestamp.q.out 3aae7d0 
>   hbase-handler/src/test/results/positive/ppd_key_ranges.q.out 5936735 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 0de7488 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  44269f0 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
> 0a83440 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java c420190 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java f66279f 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 508d880 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e8cd06d 
>   ql/src/test/results/clientpositive/annotate_stats_limit.q.out 5f8b6f8 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 241192b 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 753ab4e 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out 9bf82ac 
>   ql/src/test/results/clientpositive/auto_join30.q.out b068493 
>   ql/src/test/results/clientpositive/auto_join31.q.out 1e19dd0 
>   ql/src/test/results/clientpositive/auto_join32.q.out bfc8be8 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 9100762 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out ed09875 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> ce4ad8a 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 383defd 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out e9fb705 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out 43504d8 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out afd5518 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out c089419 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 6e443fa 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out feaea04 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out f64ecf0 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out f039dda 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out e89f548 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 44c037f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 65aa3ef 
>   ql/src/test/results/clientpositive/binarysortable_1.q.out c4ba7e0 
>   ql/src/test/results/clientpositive/bucket_map_join_1.q.out d778203 
>   ql/src/test/results/clientpositive/bucket_map_join_2.q.out aef77aa 
>   ql/src/test/results/clientpositive/bucketmapjoin1.q.out 72f2a07 
>   ql/src/test/results/clientpositiv

Review Request 34495: HIVE-10678 : In QTestGenTask.execute() we should not throw an exception right away if we are unable to clean any old files

[Hari Sankar Sivarama Subramaniyan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34495/
---

Review request for hive.


Repository: hive-git


Description
---

Currently in QTestGenTask.execute(), we have code like :
   if (logFile != null) {
 File lf = new File(logFile);
 if (lf.exists()) {
   if (!lf.delete()) {
 throw new Exception("Could not delete log file " + 
lf.getCanonicalPath());
 }
If the log file gets locked due to some other process, it throws the above 
exception and short circuits the entire test suite. We should provide a 
mechanism to generate alternative new log files/temporary files whenever 
possible to avoid unit test failures.


Diffs
-

  ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 60c7764 

Diff: https://reviews.apache.org/r/34495/diff/


Testing
---


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-10772) LLAP: perf dips with IO elevator disabled

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-10772:
---

 Summary: LLAP: perf dips with IO elevator disabled
 Key: HIVE-10772
 URL: https://issues.apache.org/jira/browse/HIVE-10772
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin


If Q1 TPCH is run 10 times in a row on the same cluster, with IO elevator,the 
perf stays relatively consistent (or improves due to JIT and stuff). Without, 
there's a large perf dip in the middle (usually dag #3 or so for me), then 
recovery.
This dip is not caused directly by GC time, and at least in my case I cannot 
see any particular part becoming slower in YK profiles, or any obvious 
correlates. One can easily see tasks slow down from 3-7 to 10-30 seconds (up to 
60 with slow HDFS reads). One thing that happens is that kernel CPU time goes 
up to 9-12% on all the daemons when the slowdown is "ramping up" and occuring, 
compared to usual 0-3% levels. I didn't investigate much where that comes from 
since this is not a mainline scenario.
Still, interesting to learn what causes this.
YK dumps provided upon request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34463: HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin

[Sergio Pena]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34463/#review84561
---

Ship it!


Ship It!

- Sergio Pena


On May 20, 2015, 6:02 p.m., Alexander Pivovarov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34463/
> ---
> 
> (Updated May 20, 2015, 6:02 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-10717
> https://issues.apache.org/jira/browse/HIVE-10717
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/encryption_insert_partition_static.q 
> 9bff0bd34b5ef0cf43f1e6738a6f1bbf5a6f372e 
>   
> ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out
>  8966608d7ec74c66dafc563aa2984efe86cab333 
> 
> Diff: https://reviews.apache.org/r/34463/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Pivovarov
> 
>

Re: Review Request 34463: HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin

[Alexander Pivovarov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34463/
---

(Updated May 20, 2015, 6:02 p.m.)


Review request for hive.


Changes
---

rebased and rerun the test


Bugs: HIVE-10717
https://issues.apache.org/jira/browse/HIVE-10717


Repository: hive-git


Description
---

HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin


Diffs (updated)
-

  ql/src/test/queries/clientpositive/encryption_insert_partition_static.q 
9bff0bd34b5ef0cf43f1e6738a6f1bbf5a6f372e 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out
 8966608d7ec74c66dafc563aa2984efe86cab333 

Diff: https://reviews.apache.org/r/34463/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-10771) "separatorChar" has no effect in "CREATE TABLE AS SELECT" statement

[Yongzhi Chen (JIRA)]

Yongzhi Chen created HIVE-10771:
---

 Summary: "separatorChar" has no effect in "CREATE TABLE AS SELECT" 
statement
 Key: HIVE-10771
 URL: https://issues.apache.org/jira/browse/HIVE-10771
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


To replicate:
CREATE TABLE separator_test 
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES ("separatorChar" = "|","quoteChar"="\"","escapeChar"="
") 
STORED AS TEXTFILE
AS
SELECT * FROM sample_07;
Then hadoop fs -cat /user/hive/warehouse/separator_test/*
"53-3032","Truck drivers, heavy and tractor-trailer","1693590","37560"
"53-3033","Truck drivers, light or delivery services","922900","28820"
"53-3041","Taxi drivers and chauffeurs","165590","22740"
The separator is till ",", not "|" as specified.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10770) Recognize additional common factors in Filter predicates

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-10770:
--

 Summary: Recognize additional common factors in Filter predicates
 Key: HIVE-10770
 URL: https://issues.apache.org/jira/browse/HIVE-10770
 Project: Hive
  Issue Type: Bug
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Currently, we canonize predicates at the term level (i.e. "a or b or a" becomes 
"a or b" but we do not attempt to recognize terms that are equivalent). 
Further, we do not exploit e.g. the symmetry of '=' (i.e. a = b iff b = a).

- A first extension would be to normalize comparisons between field references 
and literals so that the lower field reference is always on the left. So, "$6 = 
$3" becomes "$3 = $6"; "$6 > $3" becomes "$3< $6". And "literal <= $5" becomes 
"$5 >= literal". This would not damage performance, and would improve a few 
plans.

- Another possible extension. Given the predicate "(a or b) and ((x and a) or 
(y and b))", the first factor can be removed so the expression consists only of 
"(x and a) or (y and b)".
One possible way to recognize such cases is to transform the second factor to 
CNF i.e. "(x or y) and (x or b) and (a or y) and (a or b)", and as it contains 
"(a or b)", we would know that we can discard it. Then we could just use the 
original expression i.e. "(x and a) or (y and b)" in the predicate, once we 
have done the check.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34473: HIVE-10749 Implement Insert statement for parquet

[Sergio Pena]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34473/#review84521
---


I merged master to parquet, so that you don't have to the NullWritable in this 
patch.
Could you update the patch, and upload it again?

- Sergio Pena


On May 20, 2015, 2:54 p.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34473/
> ---
> 
> (Updated May 20, 2015, 2:54 p.m.)
> 
> 
> Review request for hive, Alan Gates and Sergio Pena.
> 
> 
> Bugs: HIVE-10749
> https://issues.apache.org/jira/browse/HIVE-10749
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement the insert statement for parquet format.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  000eb38 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java
>  8380117 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
>  4e1820c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/acid/ParquetRawRecordMerger.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/acid/ParquetRecordUpdater.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java
>  43c772f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java
>  0a5edbb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetStructObjectInspector.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java
>  0d32e49 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java
>  5f7f597 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/acid/TestParquetRecordUpdater.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/acid_parquet_insert.q PRE-CREATION 
>   ql/src/test/results/clientpositive/acid_parquet_insert.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/34473/diff/
> 
> 
> Testing
> ---
> 
> Newly added qtest and UT passed locally
> 
> 
> Thanks,
> 
> cheng xu
> 
>

[jira] [Created] (HIVE-10769) Merge from master (1) [Parquet branch]

[JIRA]

Sergio Peña created HIVE-10769:
--

 Summary: Merge from master (1) [Parquet branch]
 Key: HIVE-10769
 URL: https://issues.apache.org/jira/browse/HIVE-10769
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergio Peña
Assignee: Sergio Peña






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 34473: HIVE-10749 Implement Insert statement for parquet

[cheng xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34473/
---

Review request for hive, Alan Gates and Sergio Pena.


Bugs: HIVE-10749
https://issues.apache.org/jira/browse/HIVE-10749


Repository: hive-git


Description
---

Implement the insert statement for parquet format.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
000eb38 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java 
8380117 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
 4e1820c 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/acid/ParquetRawRecordMerger.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/acid/ParquetRecordUpdater.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java
 43c772f 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java
 0a5edbb 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetStructObjectInspector.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java
 0d32e49 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java 
5f7f597 
  
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/acid/TestParquetRecordUpdater.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/acid_parquet_insert.q PRE-CREATION 
  ql/src/test/results/clientpositive/acid_parquet_insert.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/34473/diff/


Testing
---

Newly added qtest and UT passed locally


Thanks,

cheng xu

Review Request 34463: HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin

[Alexander Pivovarov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34463/
---

Review request for hive.


Bugs: HIVE-10717
https://issues.apache.org/jira/browse/HIVE-10717


Repository: hive-git


Description
---

HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin


Diffs
-

  ql/src/test/queries/clientpositive/encryption_insert_partition_static.q 
10fb96582fb4952e194540e9243a0df97f971495 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out
 96f8b6abdb12f44a1c39d9e9180dbc84b5aa5bee 

Diff: https://reviews.apache.org/r/34463/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-10768) In QTestGenTask.execute() we should not throw an exception right away if we are unable to clean any old files

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-10768:


 Summary: In QTestGenTask.execute() we should not throw an 
exception right away if we are unable to clean any old files
 Key: HIVE-10768
 URL: https://issues.apache.org/jira/browse/HIVE-10768
 Project: Hive
  Issue Type: Bug
  Components: Testing Infrastructure
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


Currently in QTestGenTask.execute(), we have code like :
{code}
   if (logFile != null) {
 File lf = new File(logFile);
 if (lf.exists()) {
   if (!lf.delete()) {
 throw new Exception("Could not delete log file " + 
lf.getCanonicalPath());
 }
{code}

If the log file gets locked due to some other process, it throws the above 
exception and short circuits the entire test suite. We should provide a 
mechanism to generate alternative new log files/temporary files whenever 
possible to avoid unit test failures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)