Re: Review Request 59998: HIVE-16867

2017-06-21 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/#review178517
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
Line 76 (original), 80 (patched)


Having a Comparator like this looks brittle. 
Better design would have been to add compareTo(Operator other) in Operator 
base class and then each operator overriding that in its own class. That way 
contract is explicit for what matters in comparison. Similiar to isSame() of 
ExprNodeDesc. This current design is brittle since any change in fields of 
operators will break these comparators.
We should file a follow-up jira to clean this up.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
Line 342 (original), 347 (patched)


I am not sure if outputColNames need to match, since these could be 
generated names so on different branches names may differ but are semantically 
same columns.
However since including them is more conservative, its ok to have them for 
now.



ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java
Lines 588-618 (patched)


Can refactor this in a method which can be called for TS and Filter, since 
lot of repeated code in two branches.


- Ashutosh Chauhan


On June 21, 2017, 3:44 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59998/
> ---
> 
> (Updated June 21, 2017, 3:44 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16867
> https://issues.apache.org/jira/browse/HIVE-16867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16867
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> be38f381e6a7638be96fa6060bf296f91e92c18c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
> d5006bd52db42e3bb2b650e099d656746834b497 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  a9099b868001f4a917b91540fb305db25ccac664 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
>  1da91641b989a43b4ce5f6a09e0eabe6487e9157 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
> e31119fd081b5989e64e00c6d903c53040dfd8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
> ca544b45491cee62709957642750552eb0ac7620 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
> 20f16fbb2baa3cca050ae4c52ef3fad102d21884 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
> 73bbebd888c508cf7add51287326ba133df9e4d3 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> 04686f7fb98b558d7c7d671341f6e121e7c282c6 
>   ql/src/test/results/clientpositive/llap/auto_join0.q.out 
> 6d051ea3f5117c043f79cfc9c641f1a8ace3c87e 
>   ql/src/test/results/clientpositive/llap/auto_join30.q.out 
> 90c4241f192cfe8f40acc8a1fc0e19fbc846bd97 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
> bdb30d735bc481b11fa6b44563ba74861841071b 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
> 042c60bf17f4d696e771dfb24d7f7a4fc146f309 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out 
> cdae4fbfbf107394ccd58e7d9d1d2aeaeaa285ec 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 
> 3e715463a48a18986a82c4e55d8c8576821b0a05 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer6.q.out 
> 82dae9afbb3ae101b7b7cf8214ecfdb08f9d5d34 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
> 2875e13bbaccb4e5bfee3e682e5b2c88acaf31a5 
>   ql/src/test/results/clientpositive/llap/except_distinct.q.out 
> e4c2941f67858e0171d00032945e4e7d2b7500c9 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
> 28ca13b6d448d11d5baa1c4c96c959d413cf24f7 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
> a250fd6527327a44e97dd31ced826c12a56922cc 
>   ql/src/test/results/clientpositive/llap/intersect_merge.q.out 
> a31296672023080cc91865fbb86cdbeb891b2167 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 
> 57594e016484ea9020b349339a8a9be63709a6cb 
>   ql/src/test/results/clientpositive/llap/mrr.q.out 
> 726349c7f16823f1ff54a8575d8d8a27a50f1772 
>   ql/src/test/results/clientpositive/llap/multiMapJoin2.q.out 
> b4b0e93c82d73c60bcfe345cea1cb9c8e26ab145 
>

Re: Review Request 59998: HIVE-16867

2017-06-21 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/
---

(Updated June 21, 2017, 3:44 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16867
https://issues.apache.org/jira/browse/HIVE-16867


Repository: hive-git


Description
---

HIVE-16867


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
be38f381e6a7638be96fa6060bf296f91e92c18c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
d5006bd52db42e3bb2b650e099d656746834b497 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 a9099b868001f4a917b91540fb305db25ccac664 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java 
1da91641b989a43b4ce5f6a09e0eabe6487e9157 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
e31119fd081b5989e64e00c6d903c53040dfd8d3 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
ca544b45491cee62709957642750552eb0ac7620 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
20f16fbb2baa3cca050ae4c52ef3fad102d21884 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
73bbebd888c508cf7add51287326ba133df9e4d3 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
04686f7fb98b558d7c7d671341f6e121e7c282c6 
  ql/src/test/results/clientpositive/llap/auto_join0.q.out 
6d051ea3f5117c043f79cfc9c641f1a8ace3c87e 
  ql/src/test/results/clientpositive/llap/auto_join30.q.out 
90c4241f192cfe8f40acc8a1fc0e19fbc846bd97 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
bdb30d735bc481b11fa6b44563ba74861841071b 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
042c60bf17f4d696e771dfb24d7f7a4fc146f309 
  ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out 
cdae4fbfbf107394ccd58e7d9d1d2aeaeaa285ec 
  ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 
3e715463a48a18986a82c4e55d8c8576821b0a05 
  ql/src/test/results/clientpositive/llap/correlationoptimizer6.q.out 
82dae9afbb3ae101b7b7cf8214ecfdb08f9d5d34 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
2875e13bbaccb4e5bfee3e682e5b2c88acaf31a5 
  ql/src/test/results/clientpositive/llap/except_distinct.q.out 
e4c2941f67858e0171d00032945e4e7d2b7500c9 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
28ca13b6d448d11d5baa1c4c96c959d413cf24f7 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
a250fd6527327a44e97dd31ced826c12a56922cc 
  ql/src/test/results/clientpositive/llap/intersect_merge.q.out 
a31296672023080cc91865fbb86cdbeb891b2167 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 
57594e016484ea9020b349339a8a9be63709a6cb 
  ql/src/test/results/clientpositive/llap/mrr.q.out 
726349c7f16823f1ff54a8575d8d8a27a50f1772 
  ql/src/test/results/clientpositive/llap/multiMapJoin2.q.out 
b4b0e93c82d73c60bcfe345cea1cb9c8e26ab145 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
c89ca6b5cd9fbb26cbf0b416e4ca71ee38d32a3f 
  ql/src/test/results/clientpositive/llap/partition_shared_scan.q.out 
34ba87cc9158b2d23ce8d3c6b3defd2b8a4d36f0 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out 
1f9c9e447416c153f909ebbe960a8343454dde14 
  ql/src/test/results/clientpositive/llap/subquery_multi.q.out 
29516eff82a63a1c551a4173ea1e7ea640bdcef0 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 
b4af91579bb44cd30503d48de72a2f8f36e98501 
  ql/src/test/results/clientpositive/llap/subquery_null_agg.q.out 
bff27810e9b0999a3794b1fd505bb602236d40ad 
  ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
e94edff262685a2e2e3c4ab4ba6382b5d2cc186d 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 
202980e975b4bdd988df856bc1e533990e5c664b 
  ql/src/test/results/clientpositive/llap/subquery_views.q.out 
1a21a02a309d7fc931513d2c89f0aee3bc8c0fc6 
  ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out 
b4b601993b6eddbe3a62a917267e214a1c09ceef 
  ql/src/test/results/clientpositive/llap/union_top_level.q.out 
2fac8ccf0c8f0c1117874c502c2970d48060e476 
  ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets4.q.out 
e1ad06c7de5d4fe729ba56ea89be1a55c6b4d487 
  ql/src/test/results/clientpositive/llap/vector_join30.q.out 
ec767507003f09d93f8491f454abf2351bd4226b 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
 d9fc6b5f5834a041d1f19495234c30948aca2751 
  ql/src/test/results/clientpositive/perf/query1.q.out 
da4a65c86286555f63b25b8e63117d1df5430bc4 
  ql/src/test/results/clientpositive/perf/query10.q.out 
9b6621c1aa03728ea2736474d23bf0b89e42d8f6 
  ql/src/test/results/clientpositive/perf/query11.q.out

Re: Review Request 59998: HIVE-16867

2017-06-21 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/#review178499
---




ql/src/test/results/clientpositive/perf/query94.q.out
Lines 89-96 (original), 88-89 (patched)


I run locally and it seems to be a q file that was not up-to-date in my 
patch as I had not rebased, thus the additional GBy is still there. I am 
waiting for q tests and will upload latest version.


- Jesús Camacho Rodríguez


On June 21, 2017, 9:06 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59998/
> ---
> 
> (Updated June 21, 2017, 9:06 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16867
> https://issues.apache.org/jira/browse/HIVE-16867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16867
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> be38f381e6a7638be96fa6060bf296f91e92c18c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
> d5006bd52db42e3bb2b650e099d656746834b497 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  a9099b868001f4a917b91540fb305db25ccac664 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
>  1da91641b989a43b4ce5f6a09e0eabe6487e9157 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
> e31119fd081b5989e64e00c6d903c53040dfd8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
> ca544b45491cee62709957642750552eb0ac7620 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
> 20f16fbb2baa3cca050ae4c52ef3fad102d21884 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
> 73bbebd888c508cf7add51287326ba133df9e4d3 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> 04686f7fb98b558d7c7d671341f6e121e7c282c6 
>   ql/src/test/results/clientpositive/llap/auto_join0.q.out 
> 6d051ea3f5117c043f79cfc9c641f1a8ace3c87e 
>   ql/src/test/results/clientpositive/llap/auto_join30.q.out 
> 90c4241f192cfe8f40acc8a1fc0e19fbc846bd97 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
> bdb30d735bc481b11fa6b44563ba74861841071b 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
> 042c60bf17f4d696e771dfb24d7f7a4fc146f309 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out 
> cdae4fbfbf107394ccd58e7d9d1d2aeaeaa285ec 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 
> 3e715463a48a18986a82c4e55d8c8576821b0a05 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer6.q.out 
> 82dae9afbb3ae101b7b7cf8214ecfdb08f9d5d34 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
> 2875e13bbaccb4e5bfee3e682e5b2c88acaf31a5 
>   ql/src/test/results/clientpositive/llap/except_distinct.q.out 
> e4c2941f67858e0171d00032945e4e7d2b7500c9 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
> 28ca13b6d448d11d5baa1c4c96c959d413cf24f7 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
> a250fd6527327a44e97dd31ced826c12a56922cc 
>   ql/src/test/results/clientpositive/llap/intersect_merge.q.out 
> a31296672023080cc91865fbb86cdbeb891b2167 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 
> 57594e016484ea9020b349339a8a9be63709a6cb 
>   ql/src/test/results/clientpositive/llap/mrr.q.out 
> 726349c7f16823f1ff54a8575d8d8a27a50f1772 
>   ql/src/test/results/clientpositive/llap/multiMapJoin2.q.out 
> b4b0e93c82d73c60bcfe345cea1cb9c8e26ab145 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> c89ca6b5cd9fbb26cbf0b416e4ca71ee38d32a3f 
>   ql/src/test/results/clientpositive/llap/partition_shared_scan.q.out 
> 34ba87cc9158b2d23ce8d3c6b3defd2b8a4d36f0 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out 
> 1f9c9e447416c153f909ebbe960a8343454dde14 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out 
> 29516eff82a63a1c551a4173ea1e7ea640bdcef0 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 
> b4af91579bb44cd30503d48de72a2f8f36e98501 
>   ql/src/test/results/clientpositive/llap/subquery_null_agg.q.out 
> bff27810e9b0999a3794b1fd505bb602236d40ad 
>   ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
> e94edff262685a2e2e3c4ab4ba6382b5d2cc186d 
>   ql/src/test/results/clientpositive/llap/subquery_select.q.out 
> 202980e975b4bdd988df856bc1e533990e5c664b 
>   ql/src/test/results/clientpositive/llap/subquery_views.q.out 
>

Re: Review Request 59998: HIVE-16867

2017-06-21 Thread Ashutosh Chauhan 


> On June 21, 2017, 2:03 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/except_distinct.q.out
> > Lines 421-425 (original), 413-417 (patched)
> > 
> >
> > Missed opportunity. Could have removed this Gby operator as well.
> 
> Jesús Camacho Rodríguez wrote:
> In this case, we have three branches.
> TS[0]-SEL[1]-GBY[3]-RS[4]-...
> TS[27]-SEL[28]-GBY[30]-RS[31]-...
> TS[47]-SEL[48]-GBY[50]-RS[51]-...
> 
> The first two branches merge up to Select operator, as GBY operators are 
> different (_count(2)_ is considered different to _count(1)_ as we do not do 
> any special reasoning around expressions right now).
> TS[0]-SEL[1]-GBY[3]-RS[4]-...
> -GBY[30]-RS[31]-...
> TS[47]-SEL[48]-GBY[50]-RS[51]-...
> 
> Then, we try to merge the third branch. However, currently there is a 
> limitation in the implementation where follow-up reutilizations (third branch 
> in this case) cannot go further that the last reused operator (SEL[1] in this 
> case). That is why we end up with following plan:
> TS[0]-SEL[1]-GBY[3]-RS[4]-
> -GBY[30]-RS[31]-
> -GBY[50]-RS[51]-
> 
> I think I can lift that restriction: I introduced it because 
> multi-branching of the children makes the code more complicated (we would 
> have to follow different branches and end up using a heuristic to choose 
> which one to reuse, e.g., longest path?) and I was already changing almost 
> completely the logic in SharedScanOptimizer.
> However, I think this is something that can be tackled in a follow-up (in 
> fact, I had already left TODO comments in the code, see 
> _extractSharedOptimizationInfo_ in SharedScanOptimizer).
> 
> Btw, another idea for a follow-up: rewrite all _count_ on constant to the 
> same expression in a Calcite rule so 1) we can prune more columns, and 2) 
> find more equivalences for reutilization and MVs rewriting.
> 
> Jesús Camacho Rodríguez wrote:
> On the follow-up on count, more specifically: transform _count(x)_ where 
> _x_ is a not-nullable expression to either _count(1)_ or _count(*)_ (I guess 
> the second option might be better since it would not inspect any expression 
> and thus run faster?)

Agreed, it can be a follow-up.


> On June 21, 2017, 2:03 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
> > Lines 824-839 (original), 819-830 (patched)
> > 
> >
> > Future improvement: Zip together 2 Gbys with different aggregations in 
> > a single Gby with projects after it projecting diff aggregates for diff 
> > branches.
> 
> Jesús Camacho Rodríguez wrote:
> I can create a follow-up for this too. However, I think the question that 
> remains is how much further we want to take this rule vs implementing proper 
> Spool operator at the logical level. The complexity of this kind of extension 
> to the existing code is that currently we do not rewrite the operator tree in 
> any way, we just reutilize what is already there; this extension would imply 
> creating/inserting new operators.

Agreed.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/#review178443
---


On June 21, 2017, 9:06 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59998/
> ---
> 
> (Updated June 21, 2017, 9:06 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16867
> https://issues.apache.org/jira/browse/HIVE-16867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16867
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> be38f381e6a7638be96fa6060bf296f91e92c18c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
> d5006bd52db42e3bb2b650e099d656746834b497 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  a9099b868001f4a917b91540fb305db25ccac664 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
>  1da91641b989a43b4ce5f6a09e0eabe6487e9157 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
> e31119fd081b5989e64e00c6d903c53040dfd8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
> ca544b45491cee62709957642750552eb0ac7620 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
>

Re: Review Request 59998: HIVE-16867

2017-06-21 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/#review178496
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Line 1659 (original), 1659 (patched)


yes.. lets change config name as well.



ql/src/test/results/clientpositive/perf/query94.q.out
Lines 89-96 (original), 88-89 (patched)


This seems to be undoing recently introduced hive.optimize.countdistinct 
optimization, which is surprising since that optimization runs before this. Is 
this expected? This may result in loss of performance.


- Ashutosh Chauhan


On June 21, 2017, 9:06 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59998/
> ---
> 
> (Updated June 21, 2017, 9:06 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16867
> https://issues.apache.org/jira/browse/HIVE-16867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16867
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> be38f381e6a7638be96fa6060bf296f91e92c18c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
> d5006bd52db42e3bb2b650e099d656746834b497 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  a9099b868001f4a917b91540fb305db25ccac664 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
>  1da91641b989a43b4ce5f6a09e0eabe6487e9157 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
> e31119fd081b5989e64e00c6d903c53040dfd8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
> ca544b45491cee62709957642750552eb0ac7620 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
> 20f16fbb2baa3cca050ae4c52ef3fad102d21884 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
> 73bbebd888c508cf7add51287326ba133df9e4d3 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> 04686f7fb98b558d7c7d671341f6e121e7c282c6 
>   ql/src/test/results/clientpositive/llap/auto_join0.q.out 
> 6d051ea3f5117c043f79cfc9c641f1a8ace3c87e 
>   ql/src/test/results/clientpositive/llap/auto_join30.q.out 
> 90c4241f192cfe8f40acc8a1fc0e19fbc846bd97 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
> bdb30d735bc481b11fa6b44563ba74861841071b 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
> 042c60bf17f4d696e771dfb24d7f7a4fc146f309 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out 
> cdae4fbfbf107394ccd58e7d9d1d2aeaeaa285ec 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 
> 3e715463a48a18986a82c4e55d8c8576821b0a05 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer6.q.out 
> 82dae9afbb3ae101b7b7cf8214ecfdb08f9d5d34 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
> 2875e13bbaccb4e5bfee3e682e5b2c88acaf31a5 
>   ql/src/test/results/clientpositive/llap/except_distinct.q.out 
> e4c2941f67858e0171d00032945e4e7d2b7500c9 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
> 28ca13b6d448d11d5baa1c4c96c959d413cf24f7 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
> a250fd6527327a44e97dd31ced826c12a56922cc 
>   ql/src/test/results/clientpositive/llap/intersect_merge.q.out 
> a31296672023080cc91865fbb86cdbeb891b2167 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 
> 57594e016484ea9020b349339a8a9be63709a6cb 
>   ql/src/test/results/clientpositive/llap/mrr.q.out 
> 726349c7f16823f1ff54a8575d8d8a27a50f1772 
>   ql/src/test/results/clientpositive/llap/multiMapJoin2.q.out 
> b4b0e93c82d73c60bcfe345cea1cb9c8e26ab145 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> c89ca6b5cd9fbb26cbf0b416e4ca71ee38d32a3f 
>   ql/src/test/results/clientpositive/llap/partition_shared_scan.q.out 
> 34ba87cc9158b2d23ce8d3c6b3defd2b8a4d36f0 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out 
> 1f9c9e447416c153f909ebbe960a8343454dde14 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out 
> 29516eff82a63a1c551a4173ea1e7ea640bdcef0 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 
> b4af91579bb44cd30503d48de72a2f8f36e98501 
>   ql/src/test/results/clientpositive/llap/subquery_null_agg.q.out 
> bff27810e9b0999a3794b1fd505bb602236d40ad 
>   ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
> e94edff262685a2e2e3c4ab4ba6382b5d2cc186d 
>

Re: Review Request 59998: HIVE-16867

2017-06-21 Thread Jesús Camacho Rodríguez 


> On June 21, 2017, 2:03 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/except_distinct.q.out
> > Lines 421-425 (original), 413-417 (patched)
> > 
> >
> > Missed opportunity. Could have removed this Gby operator as well.
> 
> Jesús Camacho Rodríguez wrote:
> In this case, we have three branches.
> TS[0]-SEL[1]-GBY[3]-RS[4]-...
> TS[27]-SEL[28]-GBY[30]-RS[31]-...
> TS[47]-SEL[48]-GBY[50]-RS[51]-...
> 
> The first two branches merge up to Select operator, as GBY operators are 
> different (_count(2)_ is considered different to _count(1)_ as we do not do 
> any special reasoning around expressions right now).
> TS[0]-SEL[1]-GBY[3]-RS[4]-...
> -GBY[30]-RS[31]-...
> TS[47]-SEL[48]-GBY[50]-RS[51]-...
> 
> Then, we try to merge the third branch. However, currently there is a 
> limitation in the implementation where follow-up reutilizations (third branch 
> in this case) cannot go further that the last reused operator (SEL[1] in this 
> case). That is why we end up with following plan:
> TS[0]-SEL[1]-GBY[3]-RS[4]-
> -GBY[30]-RS[31]-
> -GBY[50]-RS[51]-
> 
> I think I can lift that restriction: I introduced it because 
> multi-branching of the children makes the code more complicated (we would 
> have to follow different branches and end up using a heuristic to choose 
> which one to reuse, e.g., longest path?) and I was already changing almost 
> completely the logic in SharedScanOptimizer.
> However, I think this is something that can be tackled in a follow-up (in 
> fact, I had already left TODO comments in the code, see 
> _extractSharedOptimizationInfo_ in SharedScanOptimizer).
> 
> Btw, another idea for a follow-up: rewrite all _count_ on constant to the 
> same expression in a Calcite rule so 1) we can prune more columns, and 2) 
> find more equivalences for reutilization and MVs rewriting.

On the follow-up on count, more specifically: transform _count(x)_ where _x_ is 
a not-nullable expression to either _count(1)_ or _count(*)_ (I guess the 
second option might be better since it would not inspect any expression and 
thus run faster?)


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/#review178443
---


On June 21, 2017, 9:06 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59998/
> ---
> 
> (Updated June 21, 2017, 9:06 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16867
> https://issues.apache.org/jira/browse/HIVE-16867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16867
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> be38f381e6a7638be96fa6060bf296f91e92c18c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
> d5006bd52db42e3bb2b650e099d656746834b497 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  a9099b868001f4a917b91540fb305db25ccac664 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
>  1da91641b989a43b4ce5f6a09e0eabe6487e9157 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
> e31119fd081b5989e64e00c6d903c53040dfd8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
> ca544b45491cee62709957642750552eb0ac7620 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
> 20f16fbb2baa3cca050ae4c52ef3fad102d21884 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
> 73bbebd888c508cf7add51287326ba133df9e4d3 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> 04686f7fb98b558d7c7d671341f6e121e7c282c6 
>   ql/src/test/results/clientpositive/llap/auto_join0.q.out 
> 6d051ea3f5117c043f79cfc9c641f1a8ace3c87e 
>   ql/src/test/results/clientpositive/llap/auto_join30.q.out 
> 90c4241f192cfe8f40acc8a1fc0e19fbc846bd97 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
> bdb30d735bc481b11fa6b44563ba74861841071b 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
> 042c60bf17f4d696e771dfb24d7f7a4fc146f309 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out 
> cdae4fbfbf107394ccd58e7d9d1d2aeaeaa285ec 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 
> 3e715463a48a18986a82c4e55d8c8576821b0a05 
>

Re: Review Request 59998: HIVE-16867

2017-06-21 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/
---

(Updated June 21, 2017, 9:06 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16867
https://issues.apache.org/jira/browse/HIVE-16867


Repository: hive-git


Description
---

HIVE-16867


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
be38f381e6a7638be96fa6060bf296f91e92c18c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
d5006bd52db42e3bb2b650e099d656746834b497 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 a9099b868001f4a917b91540fb305db25ccac664 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java 
1da91641b989a43b4ce5f6a09e0eabe6487e9157 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
e31119fd081b5989e64e00c6d903c53040dfd8d3 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
ca544b45491cee62709957642750552eb0ac7620 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 
20f16fbb2baa3cca050ae4c52ef3fad102d21884 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
73bbebd888c508cf7add51287326ba133df9e4d3 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
04686f7fb98b558d7c7d671341f6e121e7c282c6 
  ql/src/test/results/clientpositive/llap/auto_join0.q.out 
6d051ea3f5117c043f79cfc9c641f1a8ace3c87e 
  ql/src/test/results/clientpositive/llap/auto_join30.q.out 
90c4241f192cfe8f40acc8a1fc0e19fbc846bd97 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
bdb30d735bc481b11fa6b44563ba74861841071b 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
042c60bf17f4d696e771dfb24d7f7a4fc146f309 
  ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out 
cdae4fbfbf107394ccd58e7d9d1d2aeaeaa285ec 
  ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 
3e715463a48a18986a82c4e55d8c8576821b0a05 
  ql/src/test/results/clientpositive/llap/correlationoptimizer6.q.out 
82dae9afbb3ae101b7b7cf8214ecfdb08f9d5d34 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
2875e13bbaccb4e5bfee3e682e5b2c88acaf31a5 
  ql/src/test/results/clientpositive/llap/except_distinct.q.out 
e4c2941f67858e0171d00032945e4e7d2b7500c9 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
28ca13b6d448d11d5baa1c4c96c959d413cf24f7 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
a250fd6527327a44e97dd31ced826c12a56922cc 
  ql/src/test/results/clientpositive/llap/intersect_merge.q.out 
a31296672023080cc91865fbb86cdbeb891b2167 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 
57594e016484ea9020b349339a8a9be63709a6cb 
  ql/src/test/results/clientpositive/llap/mrr.q.out 
726349c7f16823f1ff54a8575d8d8a27a50f1772 
  ql/src/test/results/clientpositive/llap/multiMapJoin2.q.out 
b4b0e93c82d73c60bcfe345cea1cb9c8e26ab145 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
c89ca6b5cd9fbb26cbf0b416e4ca71ee38d32a3f 
  ql/src/test/results/clientpositive/llap/partition_shared_scan.q.out 
34ba87cc9158b2d23ce8d3c6b3defd2b8a4d36f0 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out 
1f9c9e447416c153f909ebbe960a8343454dde14 
  ql/src/test/results/clientpositive/llap/subquery_multi.q.out 
29516eff82a63a1c551a4173ea1e7ea640bdcef0 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 
b4af91579bb44cd30503d48de72a2f8f36e98501 
  ql/src/test/results/clientpositive/llap/subquery_null_agg.q.out 
bff27810e9b0999a3794b1fd505bb602236d40ad 
  ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
e94edff262685a2e2e3c4ab4ba6382b5d2cc186d 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 
202980e975b4bdd988df856bc1e533990e5c664b 
  ql/src/test/results/clientpositive/llap/subquery_views.q.out 
1a21a02a309d7fc931513d2c89f0aee3bc8c0fc6 
  ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out 
b4b601993b6eddbe3a62a917267e214a1c09ceef 
  ql/src/test/results/clientpositive/llap/union_top_level.q.out 
2fac8ccf0c8f0c1117874c502c2970d48060e476 
  ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets4.q.out 
e1ad06c7de5d4fe729ba56ea89be1a55c6b4d487 
  ql/src/test/results/clientpositive/llap/vector_join30.q.out 
ec767507003f09d93f8491f454abf2351bd4226b 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
 d9fc6b5f5834a041d1f19495234c30948aca2751 
  ql/src/test/results/clientpositive/perf/query1.q.out 
da4a65c86286555f63b25b8e63117d1df5430bc4 
  ql/src/test/results/clientpositive/perf/query10.q.out 
9b6621c1aa03728ea2736474d23bf0b89e42d8f6 
  ql/src/test/results/clientpositive/perf/query11.q.out

Re: Review Request 59998: HIVE-16867

2017-06-21 Thread Jesús Camacho Rodríguez 


> On June 21, 2017, 2:03 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java
> > Line 77 (original), 108 (patched)
> > 
> >
> > Shall rename this to SharedWorkOptimizer.

I changed this. Should I change the name of the HiveConf property too since we 
have not had a release since it was introduced (I think it will help avoiding 
confusion)?


> On June 21, 2017, 2:03 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
> > Lines 2242-2252 (original), 2242-2246 (patched)
> > 
> >
> > Is this change expected? Filter is removed from both branches. 
> > Technically, this is correct since ds is partition column, but logic to 
> > remove filters on partition column is in PartitionConditionRemover. Does 
> > SharedScan has logic to remove redundant filters in tree?

Actually there were no Filter operators before SharedScanOptimizer was 
introduced. These Filter operators were introduced by the SharedScanOptimizer, 
as previous version was pushing the filter expressions above the Scan when it 
was reusing it. Currently, as logic for reutilization has changed, it does not 
do that anymore, which is a good thing.


> On June 21, 2017, 2:03 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/except_distinct.q.out
> > Lines 421-425 (original), 413-417 (patched)
> > 
> >
> > Missed opportunity. Could have removed this Gby operator as well.

In this case, we have three branches.
TS[0]-SEL[1]-GBY[3]-RS[4]-...
TS[27]-SEL[28]-GBY[30]-RS[31]-...
TS[47]-SEL[48]-GBY[50]-RS[51]-...

The first two branches merge up to Select operator, as GBY operators are 
different (_count(2)_ is considered different to _count(1)_ as we do not do any 
special reasoning around expressions right now).
TS[0]-SEL[1]-GBY[3]-RS[4]-...
-GBY[30]-RS[31]-...
TS[47]-SEL[48]-GBY[50]-RS[51]-...

Then, we try to merge the third branch. However, currently there is a 
limitation in the implementation where follow-up reutilizations (third branch 
in this case) cannot go further that the last reused operator (SEL[1] in this 
case). That is why we end up with following plan:
TS[0]-SEL[1]-GBY[3]-RS[4]-
-GBY[30]-RS[31]-
-GBY[50]-RS[51]-

I think I can lift that restriction: I introduced it because multi-branching of 
the children makes the code more complicated (we would have to follow different 
branches and end up using a heuristic to choose which one to reuse, e.g., 
longest path?) and I was already changing almost completely the logic in 
SharedScanOptimizer.
However, I think this is something that can be tackled in a follow-up (in fact, 
I had already left TODO comments in the code, see 
_extractSharedOptimizationInfo_ in SharedScanOptimizer).

Btw, another idea for a follow-up: rewrite all _count_ on constant to the same 
expression in a Calcite rule so 1) we can prune more columns, and 2) find more 
equivalences for reutilization and MVs rewriting.


> On June 21, 2017, 2:03 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
> > Lines 824-839 (original), 819-830 (patched)
> > 
> >
> > Future improvement: Zip together 2 Gbys with different aggregations in 
> > a single Gby with projects after it projecting diff aggregates for diff 
> > branches.

I can create a follow-up for this too. However, I think the question that 
remains is how much further we want to take this rule vs implementing proper 
Spool operator at the logical level. The complexity of this kind of extension 
to the existing code is that currently we do not rewrite the operator tree in 
any way, we just reutilize what is already there; this extension would imply 
creating/inserting new operators.


> On June 21, 2017, 2:03 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
> > Lines 2491-2509 (original), 2482-2495 (patched)
> > 
> >
> > Gbys doing diff aggregations but on same key can be further zipped.

Same comment as above.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/#review178443
---


On June 12, 2017, 10:42 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59998/
>

Re: Review Request 59998: HIVE-16867

2017-06-20 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/#review178443
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java
Line 77 (original), 108 (patched)


Shall rename this to SharedWorkOptimizer.



ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
Lines 2242-2252 (original), 2242-2246 (patched)


Is this change expected? Filter is removed from both branches. Technically, 
this is correct since ds is partition column, but logic to remove filters on 
partition column is in PartitionConditionRemover. Does SharedScan has logic to 
remove redundant filters in tree?



ql/src/test/results/clientpositive/llap/except_distinct.q.out
Lines 421-425 (original), 413-417 (patched)


Missed opportunity. Could have removed this Gby operator as well.



ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
Lines 824-839 (original), 819-830 (patched)


Future improvement: Zip together 2 Gbys with different aggregations in a 
single Gby with projects after it projecting diff aggregates for diff branches.



ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
Lines 2491-2509 (original), 2482-2495 (patched)


Gbys doing diff aggregations but on same key can be further zipped.


- Ashutosh Chauhan


On June 12, 2017, 10:42 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59998/
> ---
> 
> (Updated June 12, 2017, 10:42 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16867
> https://issues.apache.org/jira/browse/HIVE-16867
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16867
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
> d5006bd52db42e3bb2b650e099d656746834b497 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  a9099b868001f4a917b91540fb305db25ccac664 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java
>  1da91641b989a43b4ce5f6a09e0eabe6487e9157 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
> e31119fd081b5989e64e00c6d903c53040dfd8d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
> a9c1e61ba94574d786c3be912a0b4f9eab20db96 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
> 73bbebd888c508cf7add51287326ba133df9e4d3 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
> 04686f7fb98b558d7c7d671341f6e121e7c282c6 
>   ql/src/test/results/clientpositive/llap/auto_join0.q.out 
> 6d051ea3f5117c043f79cfc9c641f1a8ace3c87e 
>   ql/src/test/results/clientpositive/llap/auto_join30.q.out 
> 90c4241f192cfe8f40acc8a1fc0e19fbc846bd97 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
> bdb30d735bc481b11fa6b44563ba74861841071b 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
> 042c60bf17f4d696e771dfb24d7f7a4fc146f309 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out 
> cdae4fbfbf107394ccd58e7d9d1d2aeaeaa285ec 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 
> 3e715463a48a18986a82c4e55d8c8576821b0a05 
>   ql/src/test/results/clientpositive/llap/correlationoptimizer6.q.out 
> 82dae9afbb3ae101b7b7cf8214ecfdb08f9d5d34 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
> 2875e13bbaccb4e5bfee3e682e5b2c88acaf31a5 
>   ql/src/test/results/clientpositive/llap/except_distinct.q.out 
> e4c2941f67858e0171d00032945e4e7d2b7500c9 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
> 8b04bc9261478fee17b82c780a161410a704f4ac 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
> e3f70b097f11255276ffc80082c6c8c115a76a1e 
>   ql/src/test/results/clientpositive/llap/intersect_merge.q.out 
> a31296672023080cc91865fbb86cdbeb891b2167 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 
> 57594e016484ea9020b349339a8a9be63709a6cb 
>   ql/src/test/results/clientpositive/llap/mrr.q.out 
> 726349c7f16823f1ff54a8575d8d8a27a50f1772 
>   ql/src/test/results/clientpositive/llap/multiMapJoin2.q.out 
> b4b0e93c82d73c60bcfe345cea1cb9c8e26ab145 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> c89ca6b5cd9fbb26cbf0b416e4ca71ee38d32a3f 
>   ql/src/test/results/clientpositive/llap/partition_shared_scan.q.out 
>

Review Request 59998: HIVE-16867

2017-06-12 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59998/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16867
https://issues.apache.org/jira/browse/HIVE-16867


Repository: hive-git


Description
---

HIVE-16867


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 
d5006bd52db42e3bb2b650e099d656746834b497 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 a9099b868001f4a917b91540fb305db25ccac664 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java 
1da91641b989a43b4ce5f6a09e0eabe6487e9157 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedScanOptimizer.java 
e31119fd081b5989e64e00c6d903c53040dfd8d3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
a9c1e61ba94574d786c3be912a0b4f9eab20db96 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java 
73bbebd888c508cf7add51287326ba133df9e4d3 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
04686f7fb98b558d7c7d671341f6e121e7c282c6 
  ql/src/test/results/clientpositive/llap/auto_join0.q.out 
6d051ea3f5117c043f79cfc9c641f1a8ace3c87e 
  ql/src/test/results/clientpositive/llap/auto_join30.q.out 
90c4241f192cfe8f40acc8a1fc0e19fbc846bd97 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
bdb30d735bc481b11fa6b44563ba74861841071b 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
042c60bf17f4d696e771dfb24d7f7a4fc146f309 
  ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out 
cdae4fbfbf107394ccd58e7d9d1d2aeaeaa285ec 
  ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out 
3e715463a48a18986a82c4e55d8c8576821b0a05 
  ql/src/test/results/clientpositive/llap/correlationoptimizer6.q.out 
82dae9afbb3ae101b7b7cf8214ecfdb08f9d5d34 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
2875e13bbaccb4e5bfee3e682e5b2c88acaf31a5 
  ql/src/test/results/clientpositive/llap/except_distinct.q.out 
e4c2941f67858e0171d00032945e4e7d2b7500c9 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 
8b04bc9261478fee17b82c780a161410a704f4ac 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 
e3f70b097f11255276ffc80082c6c8c115a76a1e 
  ql/src/test/results/clientpositive/llap/intersect_merge.q.out 
a31296672023080cc91865fbb86cdbeb891b2167 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 
57594e016484ea9020b349339a8a9be63709a6cb 
  ql/src/test/results/clientpositive/llap/mrr.q.out 
726349c7f16823f1ff54a8575d8d8a27a50f1772 
  ql/src/test/results/clientpositive/llap/multiMapJoin2.q.out 
b4b0e93c82d73c60bcfe345cea1cb9c8e26ab145 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
c89ca6b5cd9fbb26cbf0b416e4ca71ee38d32a3f 
  ql/src/test/results/clientpositive/llap/partition_shared_scan.q.out 
34ba87cc9158b2d23ce8d3c6b3defd2b8a4d36f0 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out 
1f9c9e447416c153f909ebbe960a8343454dde14 
  ql/src/test/results/clientpositive/llap/subquery_multi.q.out 
29516eff82a63a1c551a4173ea1e7ea640bdcef0 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 
b4af91579bb44cd30503d48de72a2f8f36e98501 
  ql/src/test/results/clientpositive/llap/subquery_null_agg.q.out 
bff27810e9b0999a3794b1fd505bb602236d40ad 
  ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 
e94edff262685a2e2e3c4ab4ba6382b5d2cc186d 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 
202980e975b4bdd988df856bc1e533990e5c664b 
  ql/src/test/results/clientpositive/llap/subquery_views.q.out 
1a21a02a309d7fc931513d2c89f0aee3bc8c0fc6 
  ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out 
b4b601993b6eddbe3a62a917267e214a1c09ceef 
  ql/src/test/results/clientpositive/llap/union_top_level.q.out 
2fac8ccf0c8f0c1117874c502c2970d48060e476 
  ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets4.q.out 
e1ad06c7de5d4fe729ba56ea89be1a55c6b4d487 
  ql/src/test/results/clientpositive/llap/vector_join30.q.out 
ec767507003f09d93f8491f454abf2351bd4226b 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
 d9fc6b5f5834a041d1f19495234c30948aca2751 
  ql/src/test/results/clientpositive/perf/query1.q.out 
da4a65c86286555f63b25b8e63117d1df5430bc4 
  ql/src/test/results/clientpositive/perf/query10.q.out 
9b6621c1aa03728ea2736474d23bf0b89e42d8f6 
  ql/src/test/results/clientpositive/perf/query14.q.out 
048a17f92f67c4ca78ab1fcfcdbc4572d784e39a 
  ql/src/test/results/clientpositive/perf/query16.q.out 
a7f93f9ec28f38675429429c1668ae7a09b0e5c0 
  ql/src/test/results/clientpositive/perf/query2.q.out 
50d7f7bcfacc81cdc5e3d7485d9ea2c5b5bf1d52 
  ql/src/test/results/clientpositive/perf/query23.q.out 
1fd8cb4f259a12e4b805843e480575b055bb