subject:"Re\: Review Request 65422\: HIVE\-17626"

Re: Review Request 65422: HIVE-17626

2018-03-06 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated March 6, 2018, 11:06 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

11 ; address review comments; disable opstats collection for successfull first 
query ; added a conf variable to enable it for tests.


Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edea129579 
  data/conf/llap/hive-site.xml c4c299c5de 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
 cbeac2c00a 
  itests/src/test/resources/testconfiguration.properties e8aa827523 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 07e6eaad88 
  itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
c01d87bf51 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 6999777297 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
08b791ad42 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
131127e50d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 77e9263e0f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
1e28ca843f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
2b345d6ec7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
085d6a7d94 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
0d31e6e422 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 8c1bcb3f62 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java
 d1c53cf345 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 0c6aece1df 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 35f9edfecc 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
714cf3914b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 
7332693513 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
5d3fdb8b63 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 
3837a49934 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2

Re: Review Request 65422: HIVE-17626

2018-03-05 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated March 5, 2018, 10:12 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

10


Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edea129579 
  data/conf/llap/hive-site.xml c4c299c5de 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
 cbeac2c00a 
  itests/src/test/resources/testconfiguration.properties e8aa827523 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 07e6eaad88 
  itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
c01d87bf51 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 6999777297 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
08b791ad42 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
131127e50d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 77e9263e0f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
1e28ca843f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
2b345d6ec7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
085d6a7d94 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
0d31e6e422 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 8c1bcb3f62 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java
 d1c53cf345 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 0c6aece1df 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 35f9edfecc 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
714cf3914b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 
7332693513 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
5d3fdb8b63 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 
3837a49934 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2 
  ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a26 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09

Re: Review Request 65422: HIVE-17626

2018-03-05 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review198652
---




ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 653 (patched)


I meant this should be run only if first query execution failed. Also when 
it runs it should simultaneously collect stats from executed tree.

Actually I am not sure how its computing even signatures, since its not 
passing any reference to plan itself.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java
Lines 35 (patched)


Add comment about what is key and what is value.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java
Lines 68 (patched)


I think we agreed on that signature of operator is stored in a flat 
structure which can be compared without traversing full tree. Can you leave a 
TODO for this?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 2506 (patched)


Whats the need for this condition? Runtime stats are always accurate even 
when state is determined as complete during compile time.


- Ashutosh Chauhan


On March 5, 2018, 8:08 a.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated March 5, 2018, 8:08 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17626
> https://issues.apache.org/jira/browse/HIVE-17626
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> eca35731292df6e1adef7ec04164e39a7baf13e1 
>   data/conf/llap/hive-site.xml cdda875ddbcc7b6fe90523779487d30a043a 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> 32b75518cfea11367e0adee26b08c47887a59b54 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 2d0aca0a11bda6b9dce31ccf6acfcfd81e412d5a 
>   itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
> c01d87bf51942f01a38e21ffbd66a9313a91e9a6 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 
> dba2dbb15b6dadee4a5cd640c8cea39669869477 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 2d7e4597de623d892702cae6e732ec5eb09d87da 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 
> 49d2bf5f335c6806460fb6b83ee4da8bf842bd5a 
>   ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 
> d4494cc72ed706886f829195a58514082793be4f 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
> 76e85636d1ecddc720d6b6e3680194354a6e157c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 
> 199b1812903a1cff9d3f52078eff92fc2aa3cec5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> 395a5f450f91151773edd49f42145209b8877cc8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
> 085d6a7d94ca40faed3ec87f95962cb7fd774100 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
> 0d31e6e42203b3a9146b678b1694c457dd56ab2a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
>  8dd7cfe58c467df479be0b5984fe2a9bf9f5ab7a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
>  134fc0ff0b1412ebb9783da3bf28a9c1b9f10dbc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
>  1eb72ce4d981ae95f02d1d5acded1d8661272128 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
>  384bd746869540df35c235ab28a33d2625e9a8b5 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java 
> b75850760f9cbe1ed969bd7ac53196dd49d081a3 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
>

Re: Review Request 65422: HIVE-17626

2018-03-05 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated March 5, 2018, 8:08 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

08


Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
eca35731292df6e1adef7ec04164e39a7baf13e1 
  data/conf/llap/hive-site.xml cdda875ddbcc7b6fe90523779487d30a043a 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
 PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 
32b75518cfea11367e0adee26b08c47887a59b54 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
2d0aca0a11bda6b9dce31ccf6acfcfd81e412d5a 
  itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
c01d87bf51942f01a38e21ffbd66a9313a91e9a6 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java 
dba2dbb15b6dadee4a5cd640c8cea39669869477 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
2d7e4597de623d892702cae6e732ec5eb09d87da 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 
49d2bf5f335c6806460fb6b83ee4da8bf842bd5a 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 
d4494cc72ed706886f829195a58514082793be4f 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
76e85636d1ecddc720d6b6e3680194354a6e157c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 
199b1812903a1cff9d3f52078eff92fc2aa3cec5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
395a5f450f91151773edd49f42145209b8877cc8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
085d6a7d94ca40faed3ec87f95962cb7fd774100 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
0d31e6e42203b3a9146b678b1694c457dd56ab2a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c467df479be0b5984fe2a9bf9f5ab7a 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b1412ebb9783da3bf28a9c1b9f10dbc 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d981ae95f02d1d5acded1d8661272128 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd746869540df35c235ab28a33d2625e9a8b5 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java 
b75850760f9cbe1ed969bd7ac53196dd49d081a3 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e1cbae2386cd2f475e47b44f750d088 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
190771ea6b1cbf4b669a8919271b25a689af941b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/Signature.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/SignatureUtils.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java
 d1c53cf34515d73f24425a7b3a6afdf2d076b9d4 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 cbadfa4f07e04f6575faa9147df687cb860cb5ca 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 
78cbf25c436e44258947c459e3e0c0e24eb9d622 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
907362312da329010dcde53d8fb6f5790f022fbd 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 
35f9edfecc74b85fe64a2cbe6290ca44a341c63e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
714cf3914b0190c360a1246934902db1c8ab4012 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 
7d5be6ba819fd73d1bded1a7a246049a0a2f420d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 
7332693513fe4a9374422a7671e82351590988f2

Re: Review Request 65422: HIVE-17626

2018-03-04 Thread Zoltan Haindrich



> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote:
> > data/conf/hive-site.xml
> > Lines 338 (patched)
> > 
> >
> > Is there a reason for this? AFAICT existing tests should not get 
> > affected by this, so this shouldn't impact tests, so we might as well turn 
> > it to true.

at some point I was not sure if this patch is broking some tests or not;
but I'll submit it with and without this for testing right now...I think some 
negative tests will be broken - because the reexecution will cause some new log 
lines...I'll remove this if possible in the next patch


> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> > Lines 652 (patched)
> > 
> >
> > We want to compute operator signatures and store stats corresponding to 
> > it only if we decide to reexecute it. Currently, it seems it 
> > unconditionally does this. Also, at this point plan hasn't been executed, 
> > so there are no stats for it. So, is this only computing signatures?

added a conditional; yes, it's only computing the signatures - the idea is that 
this call will calculate the "plan independent" part of the planmapper.


> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
> > Lines 1581 (patched)
> > 
> >
> > Also add a comment that its currently used only when reoptimization is 
> > attempted to match operators on 2 trees.

I think in HIVE-18703 I will probably move this method to somewhere else...


> On March 2, 2018, 11:19 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java
> > Lines 111 (patched)
> > 
> >
> > Can there be a semantics about whether subsequent plugin run depends on 
> > results on plugins before it? I am thinking of a scenario where ReOptimize 
> > plugin executes first but it fails to generate an alternate plan (either 
> > because it can't get stats for an operator it was looking for or can't 
> > match op trees or whatever). So, we will know before executing second time 
> > via Reoptimize plugin that it won't be successful. At that time we make 
> > Overlay plugin go with auto.convert.join=off. I think this will be useful, 
> > although can be a follow-up.

that would be an interesting extension...I don't know how would it be the best 
right now...but I think this should be investigated in a followup


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review198543
---


On March 1, 2018, 3:35 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated March 1, 2018, 3:35 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17626
> https://issues.apache.org/jira/browse/HIVE-17626
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
>   data/conf/hive-site.xml b56cbd2469 
>   data/conf/llap/hive-site.xml c4c299c5de 
>   data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7 
>   data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f 
>   data/conf/rlist/hive-site.xml 630e481edc 
>   data/conf/spark/local/hive-site.xml 8ff6256638 
>   data/conf/spark/standalone/hive-site.xml 84851c769c 
>   data/conf/spark/yarn-client/hive-site.xml 6c633623ab 
>   data/conf/tez/hive-site.xml 236adc7087 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
>  cbeac2c00a 
>   itests/src/test/resources/testconfiguration.properties 2776fe95f1 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 6cd7a136ae 
>   itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
> c01d87bf51 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
>   ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
> 08b791ad42 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
> 131127e50d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java

Re: Review Request 65422: HIVE-17626

2018-03-02 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review198543
---




data/conf/hive-site.xml
Lines 338 (patched)


Is there a reason for this? AFAICT existing tests should not get affected 
by this, so this shouldn't impact tests, so we might as well turn it to true.



data/conf/llap/hive-site.xml
Lines 363 (patched)


Set to true ?



ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 652 (patched)


We want to compute operator signatures and store stats corresponding to it 
only if we decide to reexecute it. Currently, it seems it unconditionally does 
this. Also, at this point plan hasn't been executed, so there are no stats for 
it. So, is this only computing signatures?



ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
Lines 1581 (patched)


Also add a comment that its currently used only when reoptimization is 
attempted to match operators on 2 trees.



ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
Lines 306 (patched)


Add it in non reserved list in IdentifiersParser.g



ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java
Lines 36 (patched)


These are not just filters, better name it as operators?



ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java
Lines 111 (patched)


Can there be a semantics about whether subsequent plugin run depends on 
results on plugins before it? I am thinking of a scenario where ReOptimize 
plugin executes first but it fails to generate an alternate plan (either 
because it can't get stats for an operator it was looking for or can't match op 
trees or whatever). So, we will know before executing second time via 
Reoptimize plugin that it won't be successful. At that time we make Overlay 
plugin go with auto.convert.join=off. I think this will be useful, although can 
be a follow-up.


- Ashutosh Chauhan


On March 1, 2018, 3:35 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated March 1, 2018, 3:35 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17626
> https://issues.apache.org/jira/browse/HIVE-17626
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
>   data/conf/hive-site.xml b56cbd2469 
>   data/conf/llap/hive-site.xml c4c299c5de 
>   data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7 
>   data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f 
>   data/conf/rlist/hive-site.xml 630e481edc 
>   data/conf/spark/local/hive-site.xml 8ff6256638 
>   data/conf/spark/standalone/hive-site.xml 84851c769c 
>   data/conf/spark/yarn-client/hive-site.xml 6c633623ab 
>   data/conf/tez/hive-site.xml 236adc7087 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
>  cbeac2c00a 
>   itests/src/test/resources/testconfiguration.properties 2776fe95f1 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 6cd7a136ae 
>   itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
> c01d87bf51 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
>   ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
> 08b791ad42 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
> 131127e50d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
> 1e28ca843f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
> 2b345d6ec7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> 395a5f450f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
> 085d6a7d94 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
> 0d31e6e422 
>   
>

Re: Review Request 65422: HIVE-17626

2018-03-01 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated March 1, 2018, 3:35 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

06


Bugs: HIVE-17626
https://issues.apache.org/jira/browse/HIVE-17626


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
  data/conf/hive-site.xml b56cbd2469 
  data/conf/llap/hive-site.xml c4c299c5de 
  data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7 
  data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f 
  data/conf/rlist/hive-site.xml 630e481edc 
  data/conf/spark/local/hive-site.xml 8ff6256638 
  data/conf/spark/standalone/hive-site.xml 84851c769c 
  data/conf/spark/yarn-client/hive-site.xml 6c633623ab 
  data/conf/tez/hive-site.xml 236adc7087 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
 cbeac2c00a 
  itests/src/test/resources/testconfiguration.properties 2776fe95f1 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae 
  itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
c01d87bf51 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
08b791ad42 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
131127e50d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
1e28ca843f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
2b345d6ec7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
085d6a7d94 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
0d31e6e422 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 8c1bcb3f62 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java
 d1c53cf345 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
714cf3914b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java 7d5be6ba81 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java 
7332693513 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
5d3fdb8b63 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 9c651ab3ab 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 
3837a49934 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2

Re: Review Request 65422: HIVE-17626

2018-02-27 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated Feb. 27, 2018, 6:36 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

5


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
  data/conf/hive-site.xml b56cbd2469 
  data/conf/llap/hive-site.xml c4c299c5de 
  data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7 
  data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f 
  data/conf/rlist/hive-site.xml 630e481edc 
  data/conf/spark/local/hive-site.xml 8ff6256638 
  data/conf/spark/standalone/hive-site.xml 84851c769c 
  data/conf/spark/yarn-client/hive-site.xml 6c633623ab 
  data/conf/tez/hive-site.xml 236adc7087 
  
druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java
 cbeac2c00a 
  itests/src/test/resources/testconfiguration.properties 2776fe95f1 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae 
  itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java 
c01d87bf51 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java 
08b791ad42 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
131127e50d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java 
1e28ca843f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java 
2b345d6ec7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java 
085d6a7d94 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java 
0d31e6e422 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java
 8c1bcb3f62 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java
 6c469bf944

Re: Review Request 65422: HIVE-17626

2018-02-27 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated Feb. 27, 2018, 11:17 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

03


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
  data/conf/hive-site.xml b56cbd2469 
  data/conf/llap/hive-site.xml c4c299c5de 
  data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7 
  data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f 
  data/conf/rlist/hive-site.xml 630e481edc 
  data/conf/spark/local/hive-site.xml 8ff6256638 
  data/conf/spark/standalone/hive-site.xml 84851c769c 
  data/conf/spark/yarn-client/hive-site.xml 6c633623ab 
  data/conf/tez/hive-site.xml 236adc7087 
  itests/src/test/resources/testconfiguration.properties 2776fe95f1 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae 
  ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java b75850760f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java
 fa3f89fda1 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b 
  ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1 
  ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68 
  ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a 
  ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708

Re: Review Request 65422: HIVE-17626

2018-02-27 Thread Zoltan Haindrich



> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
> > Lines 139 (patched)
> > 
> >
> > Should also compare with parent to check rowcount and other stats are 
> > not greater than parent.

ok


> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
> > Lines 310 (patched)
> > 
> >
> > Should also compare with parent to check rowcount and other stats are 
> > not greater than parent and normalize to parent stats in that case.

ok


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review197649
---


On Feb. 26, 2018, 6:19 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated Feb. 26, 2018, 6:19 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
>   data/conf/llap/hive-site.xml c4c299c5de 
>   itests/src/test/resources/testconfiguration.properties 2776fe95f1 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 6cd7a136ae 
>   ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
>   ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
>   ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> 395a5f450f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
>  8dd7cfe58c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
>  134fc0ff0b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
>  1eb72ce4d9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
>  384bd74686 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
> b0cf3bd94e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 783a672c47 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  9a3f81c98f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java
>  PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java
>  fa3f89fda1 
>   ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
>

Re: Review Request 65422: HIVE-17626

2018-02-26 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated Feb. 26, 2018, 6:19 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

update to 02


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 
  data/conf/llap/hive-site.xml c4c299c5de 
  itests/src/test/resources/testconfiguration.properties 2776fe95f1 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 6cd7a136ae 
  ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
783a672c47 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java
 fa3f89fda1 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b 
  ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out 
cab45c9410 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 43ca0d9ad7 
  ql/src/test/results/clientpositive/llap/join32_lessSize.q.out 5b5be13702 
  ql/src/test/results/clientpositive/llap/join46.q.out 4625b9fed8 
  ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 4dd7432ac5 
  ql/src/test/results/clientpositive/llap/llap_partitioned.q.out ac754674ea 
  ql/src/test/results/clientpositive/llap/mapjoin46.q.out f530c1e8e1 
  ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68 
  ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out

Re: Review Request 65422: HIVE-17626

2018-02-20 Thread Zoltan Haindrich


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/
---

(Updated Feb. 20, 2018, 6:41 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

preview


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3d777f992b 
  itests/src/test/resources/testconfiguration.properties d4f2e539fb 
  ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java d00e639643 
  ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 
  ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 
  ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 
  ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 395a5f450f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 8dd7cfe58c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
 134fc0ff0b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1eb72ce4d9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
 384bd74686 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
b0cf3bd94e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
27b53b8b33 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 9a3f81c98f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g e431271d3a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java f9a6386ecf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION 
  ql/src/test/queries/clientpositive/retry_failure_stat_changes.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/65422/diff/2/

Changes: https://reviews.apache.org/r/65422/diff/1-2/


Testing
---


Thanks,

Zoltan Haindrich

Re: Review Request 65422: HIVE-17626

2018-02-19 Thread Zoltan Haindrich



> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 3691 (patched)
> > 
> >
> > Instead of config this should be explain modifier. WE already have 
> > explain rewrite select .. We similarly can add explain reoptimize select ...

yes...I agree; it turned out that its very inconvinient to use it this way...

I've employed a semanticAnlayzer hook to handle the reoptimize keyword


> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 5066 (patched)
> > 
> >
> > Instead of iterating over _this_ which can be very large, more 
> > efficient is to iterate on other list.

I wasn't aware that the iterator() creates a new map on the flyI'm now 
using getProps() to get access to the actual values


> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
> > Lines 127 (patched)
> > 
> >
> > Currently its only reexcuted once. Alternatively, we can keep 
> > re-running it if it fails again. e.g. in case of OOM, its possible that 
> > there are many joins which are mis-planed, but we get stats only for first 
> > join.
> > To avoid, very large number of retrials we can limit to some max 
> > attempts.

I aggree


> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
> > Lines 21 (patched)
> > 
> >
> > Incorrect import ?

I've just taken a look at null analysis; but it detects too many issues to just 
turn on...so I'll remove it for now :)


> On Feb. 7, 2018, 1:58 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java
> > Lines 50 (patched)
> > 
> >
> > Instead of relying on ids, better is to use (and extend) logic in 
> > SharedWorkOptimizer::compareOperator() ?

t


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review196950
---


On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated Jan. 30, 2018, 6:13 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java 
> ad31287879 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 
> 533f0bcd6f 
>   itests/src/test/resources/testconfiguration.properties d86ff58840 
>   ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08 
>   ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> 395a5f450f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
>  8dd7cfe58c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
>  134fc0ff0b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
>  1eb72ce4d9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
>  384bd74686 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  cbadfa4f07 
>

Re: Review Request 65422: HIVE-17626

2018-02-19 Thread Zoltan Haindrich



> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
> > Lines 131 (patched)
> > 
> >
> > This is hackish.. as pointed above it needs to happen via explain 
> > modifier.

I agree


> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
> > Lines 21 (patched)
> > 
> >
> > Use java's nonnull annotation.

I've not found any "standard" annotation...I may just as well remove these 
markers...
https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use/42695253#42695253


> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
> > Lines 54 (patched)
> > 
> >
> > Why is this needed?

this is not needed...but enables the user to set a different set of 
configuration during re-executions


> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java
> > Lines 36 (patched)
> > 
> >
> > A flat map of operators looses hierarichal info in which operators are 
> > organized which is tree. So, this match needs to happen  via sub-graph 
> > matching pattern. See SharedWorkOptimizer::areMergeable() .

I will try to retain this concept for now at least; the idea is that imagine 
that we have N operator stats gathered; and the current plan consist of M 
operators; if we have only a cmp(A,B) oracle; that means we will have to do N*M 
comparisions; which could become really bad if N starts to become large...

I'm thinking of serving the existing operator infos in a map alike fashion - at 
least it should be visible as one for the outside.

If an operator could self-describe its whole context; then it could be match... 
for example a matching the textual representation of a RelNode contains all the 
upstream operations as well; and enables matching.

It looked promising to do it; I wanted to do it with HIVE-18703 - but 
unfortunately there were some complications...


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review197649
---


On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated Jan. 30, 2018, 6:13 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java 
> ad31287879 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 
> 533f0bcd6f 
>   itests/src/test/resources/testconfiguration.properties d86ff58840 
>   ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08 
>   ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> 395a5f450f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
>  8dd7cfe58c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
>  134fc0ff0b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
>  1eb72ce4d9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java
>  384bd74686 
>   ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 190771ea6b 
>   
>

Re: Review Request 65422: HIVE-17626

2018-02-15 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review197649
---




ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
Lines 131 (patched)


This is hackish.. as pointed above it needs to happen via explain modifier.



ql/src/java/org/apache/hadoop/hive/ql/Context.java
Lines 72 (patched)


Use  java.util.Optional instead



ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
Lines 21 (patched)


Use java's nonnull annotation.



ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java
Lines 40 (patched)


Need to check for 
org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionError and 
java.lang.OutOfMemoryError only.



ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
Lines 43 (patched)


Need to check for 
org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionError and 
java.lang.OutOfMemoryError only.



ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
Lines 54 (patched)


Why is this needed?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 139 (patched)


Should also compare with parent to check rowcount and other stats are not 
greater than parent.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 310 (patched)


Should also compare with parent to check rowcount and other stats are not 
greater than parent and normalize to parent stats in that case.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 2314 (patched)


Should also compare with parent to check rowcount and other stats are not 
greater than parent and normalize to parent stats in that case.



ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java
Lines 36 (patched)


A flat map of operators looses hierarichal info in which operators are 
organized which is tree. So, this match needs to happen  via sub-graph matching 
pattern. See SharedWorkOptimizer::areMergeable() .



ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java
Lines 75 (patched)


LOG.debug



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java
Lines 64 (patched)


throw MapJoinMemoryExhaustionError


- Ashutosh Chauhan


On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated Jan. 30, 2018, 6:13 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java 
> ad31287879 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 
> 533f0bcd6f 
>   itests/src/test/resources/testconfiguration.properties d86ff58840 
>   ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08 
>   ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> 395a5f450f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
>  8dd7cfe58c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java
>  134fc0ff0b 
>   
>

Re: Review Request 65422: HIVE-17626

2018-02-06 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65422/#review196950
---



First pass review.


common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3689 (patched)


Default should be reoptimize.



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3691 (patched)


Instead of config this should be explain modifier. WE already have explain 
rewrite select .. We similarly can add explain reoptimize select ...



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 5066 (patched)


Instead of iterating over _this_ which can be very large, more efficient is 
to iterate on other list.



ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
Lines 36 (patched)


Comments on what this driver does?



ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java
Lines 127 (patched)


Currently its only reexcuted once. Alternatively, we can keep re-running it 
if it fails again. e.g. in case of OOM, its possible that there are many joins 
which are mis-planed, but we get stats only for first join.
To avoid, very large number of retrials we can limit to some max attempts.



ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java
Lines 21 (patched)


Incorrect import ?



ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java
Lines 25 (patched)


Add comments on what this Driver does.



ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java
Lines 29 (patched)


java class docs.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
Lines 4748-4749 (patched)


To avoid this  we should switch to find equivalence of 2 operators based on 
their signature. e.g., Operator::logicalEquals().



ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Lines 2364 (patched)


This should extend to all operators.



ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java
Lines 33 (patched)


javadoc



ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java
Lines 32 (patched)


Operator::logicalEquals() ?



ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java
Lines 50 (patched)


Instead of relying on ids, better is to use (and extend) logic in 
SharedWorkOptimizer::compareOperator() ?



ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java
Lines 56 (patched)


LOG.debug


- Ashutosh Chauhan


On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65422/
> ---
> 
> (Updated Jan. 30, 2018, 6:13 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> preview
> 
> 
> Diffs
> -
> 
>   cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java 
> ad31287879 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java 
> 533f0bcd6f 
>   itests/src/test/resources/testconfiguration.properties d86ff58840 
>   ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33 
>   ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08 
>   ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
> 395a5f450f 
>   
>

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

Re: Review Request 65422: HIVE-17626

16 matches

Site Navigation

Mail list logo

Footer information