subject:"Review Request 71820\: HIVE\-20150"

Re: Review Request 71820: HIVE-20150

2019-12-19 Thread Ramesh Kumar Thangarajan



> On Dec. 17, 2019, 10:01 p.m., Ramesh Kumar Thangarajan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
> > Lines 4341 (patched)
> > 
> >
> > What is the bug in Decimal64 to Decimal conversion? 
> > 
> > Do we need a call to the function 
> > fixDecimalDataTypePhysicalVariations() ? Because I see the function 
> > getVectorExpressionsUpConvertDecimal64() is calling 
> > wrapWithDecimal64ToDecimalConversion() on all the child expressions.
> 
> Krisztian Kasa wrote:
> Please consider the following query: 
> ```
> CREATE TABLE decimal_test_small_n0 STORED AS ORC AS SELECT cdouble, CAST 
> (((cdouble*22.1)/37) AS DECIMAL(10,3)) AS cdecimal1, CAST (((cdouble*9.3)/13) 
> AS DECIMAL(7,2)) AS cdecimal2 FROM alltypesorc;
> 
> SELECT cdecimal1 - (2*cdecimal2) as c2 FROM decimal_test_small_n0
> ORDER BY c2
> LIMIT 10;
> ```
> 
> With given keyColumns: GenericUDFOPMinus(Column[cdecimal1], 
> GenericUDFOPMultiply(Const decimal(1,0) 2, Column[cdecimal2])) 
> ```
> keyExpressions = 
> vContext.getVectorExpressionsUpConvertDecimal64(keyColumns);
> ```
> will produce keyExpressions: 
> DecimalColSubtractDecimalColumn(col 4:decimal(10,3), col 
> 5:decimal(9,2)/DECIMAL_64)
> (children: 
> ConvertDecimal64ToDecimal(col 1:decimal(10,3)/DECIMAL_64) -> 
> 4:decimal(10,3), 
> Decimal64ScalarMultiplyDecimal64ColumnUnscaled(decimal64Val 2, 
> decimalVal 2, col 2:decimal(7,2)/DECIMAL_64) -> 5:decimal(9,2)/DECIMAL_64
> ) -> 6:decimal(11,3)
> 
> So the 2nd child of DecimalColSubtractDecimalColumn is not converted from 
> decimal64 to decimal and I got a 
> ```
>  java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColSubtractDecimalColumn.evaluate(DecimalColSubtractDecimalColumn.java:69)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorTopNKeyOperator.process(VectorTopNKeyOperator.java:101)
> ```
> 
> Checking the code of `getVectorExpressionsUpConvertDecimal64` i found 
> that it calls `getVectorExpressions` and for each result expression if that 
> expression's output type is decimal64 then it wraps it with conversion. But 
> it does not checks the children expressions.
> 
> Probably we don't need both function calls, please let me check in a 
> follow up patch if `fixDecimalDataTypePhysicalVariations` is enough.

Makes sense! Thanks for the explanation


- Ramesh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/#review219047
---


On Dec. 14, 2019, 10:31 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71820/
> ---
> 
> (Updated Dec. 14, 2019, 10:31 a.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-20150
> https://issues.apache.org/jira/browse/HIVE-20150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TopNKey pushdown
> 
> 1. Apply patch: 
> https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
> 2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
> 3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
> sink, left outer join, other topnkey
> 4. Add sort order and null sort order direction check when determining if the 
> topnkey op can be pushed
> 5. Implement handling cases when topnkey op and the parent op has a common 
> key prefix only.
> 6. fix Key object inspectors non-vectorized mode 
> 7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
> keyExpression during TopnKey vectorization
> 
> 
> Diffs
> -
> 
>   kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 
> 1324b27f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
> 0d6cf3c755 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 6876787e11 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
>  PRE-CREATION 
>   ql/

Re: Review Request 71820: HIVE-20150

2019-12-17 Thread Krisztian Kasa



> On Dec. 17, 2019, 10:01 p.m., Ramesh Kumar Thangarajan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
> > Lines 4341 (patched)
> > 
> >
> > What is the bug in Decimal64 to Decimal conversion? 
> > 
> > Do we need a call to the function 
> > fixDecimalDataTypePhysicalVariations() ? Because I see the function 
> > getVectorExpressionsUpConvertDecimal64() is calling 
> > wrapWithDecimal64ToDecimalConversion() on all the child expressions.

Please consider the following query: 
```
CREATE TABLE decimal_test_small_n0 STORED AS ORC AS SELECT cdouble, CAST 
(((cdouble*22.1)/37) AS DECIMAL(10,3)) AS cdecimal1, CAST (((cdouble*9.3)/13) 
AS DECIMAL(7,2)) AS cdecimal2 FROM alltypesorc;

SELECT cdecimal1 - (2*cdecimal2) as c2 FROM decimal_test_small_n0
ORDER BY c2
LIMIT 10;
```

With given keyColumns: GenericUDFOPMinus(Column[cdecimal1], 
GenericUDFOPMultiply(Const decimal(1,0) 2, Column[cdecimal2])) 
```
keyExpressions = vContext.getVectorExpressionsUpConvertDecimal64(keyColumns);
```
will produce keyExpressions: 
DecimalColSubtractDecimalColumn(col 4:decimal(10,3), col 
5:decimal(9,2)/DECIMAL_64)
(children: 
ConvertDecimal64ToDecimal(col 1:decimal(10,3)/DECIMAL_64) -> 
4:decimal(10,3), 
Decimal64ScalarMultiplyDecimal64ColumnUnscaled(decimal64Val 2, decimalVal 
2, col 2:decimal(7,2)/DECIMAL_64) -> 5:decimal(9,2)/DECIMAL_64
) -> 6:decimal(11,3)

So the 2nd child of DecimalColSubtractDecimalColumn is not converted from 
decimal64 to decimal and I got a 
```
 java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to 
org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColSubtractDecimalColumn.evaluate(DecimalColSubtractDecimalColumn.java:69)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorTopNKeyOperator.process(VectorTopNKeyOperator.java:101)
```

Checking the code of `getVectorExpressionsUpConvertDecimal64` i found that it 
calls `getVectorExpressions` and for each result expression if that 
expression's output type is decimal64 then it wraps it with conversion. But it 
does not checks the children expressions.

Probably we don't need both function calls, please let me check in a follow up 
patch if `fixDecimalDataTypePhysicalVariations` is enough.


- Krisztian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/#review219047
---


On Dec. 14, 2019, 10:31 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71820/
> ---
> 
> (Updated Dec. 14, 2019, 10:31 a.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-20150
> https://issues.apache.org/jira/browse/HIVE-20150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TopNKey pushdown
> 
> 1. Apply patch: 
> https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
> 2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
> 3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
> sink, left outer join, other topnkey
> 4. Add sort order and null sort order direction check when determining if the 
> topnkey op can be pushed
> 5. Implement handling cases when topnkey op and the parent op has a common 
> key prefix only.
> 6. fix Key object inspectors non-vectorized mode 
> 7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
> keyExpression during TopnKey vectorization
> 
> 
> Diffs
> -
> 
>   kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 
> 1324b27f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
> 0d6cf3c755 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 6876787e11 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
>   ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
>   ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 
> 31862546b0 
>   ql/src/test/results/clientpositive/druid/druidmini_test1.

Re: Review Request 71820: HIVE-20150

2019-12-17 Thread Ramesh Kumar Thangarajan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/#review219047
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
Lines 4341 (patched)


What is the bug in Decimal64 to Decimal conversion? 

Do we need a call to the function fixDecimalDataTypePhysicalVariations() ? 
Because I see the function getVectorExpressionsUpConvertDecimal64() is calling 
wrapWithDecimal64ToDecimalConversion() on all the child expressions.


- Ramesh Kumar Thangarajan


On Dec. 14, 2019, 10:31 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71820/
> ---
> 
> (Updated Dec. 14, 2019, 10:31 a.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-20150
> https://issues.apache.org/jira/browse/HIVE-20150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TopNKey pushdown
> 
> 1. Apply patch: 
> https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
> 2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
> 3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
> sink, left outer join, other topnkey
> 4. Add sort order and null sort order direction check when determining if the 
> topnkey op can be pushed
> 5. Implement handling cases when topnkey op and the parent op has a common 
> key prefix only.
> 6. fix Key object inspectors non-vectorized mode 
> 7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
> keyExpression during TopnKey vectorization
> 
> 
> Diffs
> -
> 
>   kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 
> 1324b27f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
> 0d6cf3c755 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 6876787e11 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
>   ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
>   ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 
> 31862546b0 
>   ql/src/test/results/clientpositive/druid/druidmini_test1.q.out fe75ddd894 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
>   ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 53e57b892f 
>   ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
>   ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
> b6d210becf 
>   ql/src/test/results/clientpositive/llap/ctas.q.out c10d75d081 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
> 4e62779446 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
> 6688944b78 
>   ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
> 9343e078b7 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 64fdc481eb 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
>   ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
> 545cce75a9 
>   ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
>   ql/src/test/results/clientpositive/llap/groupby_groupingset_bug.q.out 
> 63d56172b7 
>   ql/src/test/results/clientpositive/llap/insert1_overwrite_partitions.q.out 
> d51654bc5b 
>   ql/src/test/results/clientpositive/llap/insert_into1.q.out dd9140d3b6 
>   ql/src/test/results/clientpositive/llap/insert_into2.q.out 161318b720 
>   ql/src/test/results/clientpositive/llap/lateral_view.q.out b742230a71 
>   ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 
> 6258f4fe1e 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
>   ql/src/test/results/clientpositive/llap/materialized_view_rewrite_8.q.out 
> 303111a7d4 
>   ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 4ecb7bc46d 
>   ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
> 2fd8

Re: Review Request 71820: HIVE-20150

2019-12-14 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 14, 2019, 10:31 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. fix Key object inspectors non-vectorized mode 
7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
keyExpression during TopnKey vectorization


Diffs (updated)
-

  kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 1324b27f8e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
6876787e11 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 
31862546b0 
  ql/src/test/results/clientpositive/druid/druidmini_test1.q.out fe75ddd894 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 53e57b892f 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/ctas.q.out c10d75d081 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
4e62779446 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
6688944b78 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 64fdc481eb 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/groupby_groupingset_bug.q.out 
63d56172b7 
  ql/src/test/results/clientpositive/llap/insert1_overwrite_partitions.q.out 
d51654bc5b 
  ql/src/test/results/clientpositive/llap/insert_into1.q.out dd9140d3b6 
  ql/src/test/results/clientpositive/llap/insert_into2.q.out 161318b720 
  ql/src/test/results/clientpositive/llap/lateral_view.q.out b742230a71 
  ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 6258f4fe1e 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_8.q.out 
303111a7d4 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
2fd8faa7b5 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
8c900157c5 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out 5bba57633b 
  ql/src/test/results/clientpositive/llap/smb_mapjoin_15.q.out 8ebc139faa 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out 479e60fda9 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 3dc247591c 
  ql/src/test/results/cli

Re: Review Request 71820: HIVE-20150

2019-12-13 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 13, 2019, 7:32 p.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. fix Key object inspectors non-vectorized mode 
7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
keyExpression during TopnKey vectorization


Diffs (updated)
-

  kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 1324b27f8e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
6876787e11 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 
31862546b0 
  ql/src/test/results/clientpositive/druid/druidmini_test1.q.out fe75ddd894 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 53e57b892f 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/ctas.q.out c10d75d081 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
4e62779446 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
6688944b78 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 64fdc481eb 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/groupby_groupingset_bug.q.out 
63d56172b7 
  ql/src/test/results/clientpositive/llap/insert1_overwrite_partitions.q.out 
d51654bc5b 
  ql/src/test/results/clientpositive/llap/insert_into1.q.out dd9140d3b6 
  ql/src/test/results/clientpositive/llap/insert_into2.q.out 161318b720 
  ql/src/test/results/clientpositive/llap/lateral_view.q.out b742230a71 
  ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 6258f4fe1e 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_8.q.out 
303111a7d4 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
2fd8faa7b5 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
8c900157c5 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out 5bba57633b 
  ql/src/test/results/clientpositive/llap/smb_mapjoin_15.q.out 8ebc139faa 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out 479e60fda9 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 3dc247591c 
  ql/src/test/results/clie

Re: Review Request 71820: HIVE-20150

2019-12-12 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 13, 2019, 6:28 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. fix Key object inspectors non-vectorized mode 
7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
keyExpression during TopnKey vectorization


Diffs (updated)
-

  kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 1324b27f8e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
6876787e11 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 
31862546b0 
  ql/src/test/results/clientpositive/druid/druidmini_test1.q.out fe75ddd894 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 53e57b892f 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/ctas.q.out c10d75d081 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
4e62779446 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
6688944b78 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/groupby_groupingset_bug.q.out 
63d56172b7 
  ql/src/test/results/clientpositive/llap/insert1_overwrite_partitions.q.out 
d51654bc5b 
  ql/src/test/results/clientpositive/llap/insert_into1.q.out dd9140d3b6 
  ql/src/test/results/clientpositive/llap/insert_into2.q.out 161318b720 
  ql/src/test/results/clientpositive/llap/lateral_view.q.out b742230a71 
  ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 6258f4fe1e 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_8.q.out 
303111a7d4 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
2fd8faa7b5 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
8c900157c5 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out 5bba57633b 
  ql/src/test/results/clientpositive/llap/smb_mapjoin_15.q.out 8ebc139faa 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out b24ff5ca1e 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out b08e8f3e2f 
  ql/src/test/results/clie

Re: Review Request 71820: HIVE-20150

2019-12-12 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 12, 2019, 1:29 p.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description (updated)
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. fix Key object inspectors non-vectorized mode 
7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
keyExpression during TopnKey vectorization


Diffs (updated)
-

  kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 1324b27f8e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
6876787e11 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 
31862546b0 
  ql/src/test/results/clientpositive/druid/druidmini_test1.q.out fe75ddd894 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 53e57b892f 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/ctas.q.out c10d75d081 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
4e62779446 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
6688944b78 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/groupby_groupingset_bug.q.out 
63d56172b7 
  ql/src/test/results/clientpositive/llap/insert1_overwrite_partitions.q.out 
d51654bc5b 
  ql/src/test/results/clientpositive/llap/insert_into1.q.out dd9140d3b6 
  ql/src/test/results/clientpositive/llap/insert_into2.q.out 161318b720 
  ql/src/test/results/clientpositive/llap/lateral_view.q.out b742230a71 
  ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 6258f4fe1e 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_8.q.out 
303111a7d4 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
2fd8faa7b5 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out 
8c900157c5 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out 5bba57633b 
  ql/src/test/results/clientpositive/llap/smb_mapjoin_15.q.out 8ebc139faa 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out b24ff5ca1e 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out b08e8f3e2f 
  ql/src/test/re

Re: Review Request 71820: HIVE-20150

2019-12-09 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 10, 2019, 7:38 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query54.q.out e96d4c60a6 
  ql/src/test/results/clientpositive/perf/tez/cbo_query58.q.out 523f02e7d7 
  ql/src/test/results/clientpositive/perf/tez/cbo_query6.q.out 944b113e7b 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query54.q.out 
ece76c3e4e 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query58.q.out 
954bdc4dd1 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query6.q.out 
eb872d7e6d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f75361df1 
  ql/src/test/results/clientpositive/perf/tez/constraints/mv_query44.q.out 
61c01c5c60 
  ql/src/test/results/cli

Re: Review Request 71820: HIVE-20150

2019-12-09 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 9, 2019, 3:46 p.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7724f9084 
  data/conf/perf-reg/tez/hive-site.xml ab945f5f95 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query54.q.out e96d4c60a6 
  ql/src/test/results/clientpositive/perf/tez/cbo_query58.q.out 523f02e7d7 
  ql/src/test/results/clientpositive/perf/tez/cbo_query6.q.out 944b113e7b 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query54.q.out 
ece76c3e4e 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query58.q.out 
954bdc4dd1 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query6.q.out 
eb872d7e6d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f7

Re: Review Request 71820: HIVE-20150

2019-12-09 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 9, 2019, 3:32 p.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7724f9084 
  data/conf/perf-reg/tez/hive-site.xml ab945f5f95 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query54.q.out e96d4c60a6 
  ql/src/test/results/clientpositive/perf/tez/cbo_query58.q.out 523f02e7d7 
  ql/src/test/results/clientpositive/perf/tez/cbo_query6.q.out 944b113e7b 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query54.q.out 
ece76c3e4e 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query58.q.out 
954bdc4dd1 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query6.q.out 
eb872d7e6d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f7

Re: Review Request 71820: HIVE-20150

2019-12-09 Thread Krisztian Kasa



> On Dec. 7, 2019, 3:03 a.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
> > Lines 93 (patched)
> > 
> >
> > _replaceChild_?

OperatorFactory.getAndMakeChild adds the newly created operator to parent 
operator's children operators. At this point we have two children the original 
children and the newly created TNK op.


> On Dec. 7, 2019, 3:03 a.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
> > Lines 95 (patched)
> > 
> >
> > _replaceParent_?

I think calling `child.replaceParent(parents.get(0), newOperator)` has a risk 
of ArrayIndexOutOfBoundsException. It should not happen but I added 
```
child.getParentOperators().clear();
child.getParentOperators().add(newOperator);
```
instead.


> On Dec. 7, 2019, 3:03 a.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
> > Lines 18 (patched)
> > 
> >
> > Let's move all these classes (TopNKeyPushdownProcessor, 
> > TopNKeyProcessor, CommonKeyPrefix, etc.) into a new package 
> > _org.apache.hadoop.hive.ql.optimizer.topnkey_. Thoughts?

It makes sense. And we can always change this package in the future if needed.


> On Dec. 7, 2019, 3:03 a.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
> > Lines 105 (patched)
> > 
> >
> > It seems the logic in this method cannot throw a Semantic Exception?

It calls `moveDown` and `pushdown`. Those ones can throw `SemanticException` 
because they call `Operator.removeChildAndAdoptItsChildren(topNKey)`


> On Dec. 7, 2019, 3:03 a.m., Jesús Camacho Rodríguez wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
> > Lines 278 (patched)
> > 
> >
> > Should this method be moved to _CommonKeyPrefix_ too?

This is not a factory method of `CommonKeyPrefix`. It used when pushing through 
LOJ. I beleive it will be used when pushing through ROJ and FOJ so I would like 
to refactor this in a follow-up patch which implements those scenarios.
Anyway I removed `static` because it is used only in this class.


- Krisztian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/#review218963
---


On Dec. 9, 2019, 10:21 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71820/
> ---
> 
> (Updated Dec. 9, 2019, 10:21 a.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-20150
> https://issues.apache.org/jira/browse/HIVE-20150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TopNKey pushdown
> 
> 1. Apply patch: 
> https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
> 2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
> 3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
> sink, left outer join, other topnkey
> 4. Add sort order and null sort order direction check when determining if the 
> topnkey op can be pushed
> 5. Implement handling cases when topnkey op and the parent op has a common 
> key prefix only.
> 6. turn off topnkey optimization by default
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7724f9084 
>   data/conf/perf-reg/tez/hive-site.xml ab945f5f95 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
> 0d6cf3c755 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
>   ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
>   ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
>   ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
> b6d210becf 
>   ql/sr

Re: Review Request 71820: HIVE-20150

2019-12-09 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 9, 2019, 10:21 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7724f9084 
  data/conf/perf-reg/tez/hive-site.xml ab945f5f95 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f75361df1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out 
bb3b1b6660 
  ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 
228b20a8d7 
  ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out 
5268ed3ecf 
  ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out 
d96222d9e1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out 
adabb76e04 
  ql/src/test/results/clientpositive/perf/tez/constraints/

Re: Review Request 71820: HIVE-20150

2019-12-06 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/#review218965
---




ql/src/test/results/clientpositive/tez/topnkey.q.out
Lines 105 (patched)


It seems these two should be merged: Second one supersedes the first one.


- Jesús Camacho Rodríguez


On Dec. 3, 2019, 9:27 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71820/
> ---
> 
> (Updated Dec. 3, 2019, 9:27 a.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-20150
> https://issues.apache.org/jira/browse/HIVE-20150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TopNKey pushdown
> 
> 1. Apply patch: 
> https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
> 2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
> 3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
> sink, left outer join, other topnkey
> 4. Add sort order and null sort order direction check when determining if the 
> topnkey op can be pushed
> 5. Implement handling cases when topnkey op and the parent op has a common 
> key prefix only.
> 6. turn off topnkey optimization by default
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
> 0d6cf3c755 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
>   ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestCommonKeyPrefix.java 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
>   ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
>   ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
>   ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
> b6d210becf 
>   ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
> 9343e078b7 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
>   ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
> 545cce75a9 
>   ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
>   ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
> ffe5f6fb22 
>   ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 4ecb7bc46d 
>   ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
> 0eac389eb7 
>   
> ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
>  4362fb6f2e 
>   
> ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
> 24468c9a1b 
>   
> ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out
>  45890a1890 
>   ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
>   ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
>   ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
>   ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
>   ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out 
> cc2dc47280 
>   ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  6fd15e7101 
>   ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
> d6325982e3 
>   ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
> 4d417b9c3d 
>   ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
> 97a211cfc6 
>   ql/src/test/results/clientpositive/llap/vector_string_concat.q.out 
> a8019be7aa 
>   ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
>   ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 
> 7326adf522 
>   ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
>   ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
>   ql/src/test/re

Re: Review Request 71820: HIVE-20150

2019-12-06 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/#review218963
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Line 2386 (original), 2386 (patched)


Just for reference: Are you planning to enable back the optimization once 
all other fixes are applied? Or does it have to do with performance concerns 
for some cases?

Even if we disable it here, can we enable it for Perf cli drivers (you can 
do that by adding the property value in _data/conf/perf-reg/tez/hive-site.xml_? 
I believe it would be useful to already see how far are we pushing topn in 
those tpcds queries.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java
Lines 35 (patched)


It seems this class can only be created using the static building methods? 
If that is the case, can you create a private constructor to defeat 
instantiation?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java
Lines 37 (patched)


Can you add comments to each of these static methods explaining shortly how 
the mapping will be performed?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java
Lines 93 (patched)


I believe we should use _ExprNodeDesc.isSame_ here, since equals refers to 
identify for ExprNodeDesc objects.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
Line 107 (original), 89 (patched)


Do we need this since they are already passed when we create the new 
operator?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
Line 108 (original), 90 (patched)


We can just add the child to the child operators instead of creating the 
list multiple times.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
Lines 93 (patched)


_replaceChild_?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
Lines 95 (patched)


_replaceParent_?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 18 (patched)


Let's move all these classes (TopNKeyPushdownProcessor, TopNKeyProcessor, 
CommonKeyPrefix, etc.) into a new package 
_org.apache.hadoop.hive.ql.optimizer.topnkey_. Thoughts?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 72 (patched)


Although it is very simple, can we add this to new method as with the rest 
of operators? It helps encapsulating so same logic may be used for other simple 
operators in the future.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 91 (patched)


Typo



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 105 (patched)


It seems the logic in this method cannot throw a Semantic Exception?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 202 (patched)


Short comment on this method? _Only LOJ supported_.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 257 (patched)


Comment?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 258 (patched)


What happens is the key is the same but the value for top is different? It 
seems we do not cover that case here but it is important in case you end up 
with multiple top n operators derived from different RS operators.
I believe in that situation we could still remove one of them, but we 
should keep the smallest value for top n.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 278 (patched)


Should this method be moved to _CommonKeyPrefix_ too?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
Lines 314 (patched)


Same comment as above: It seems we do not need to create lists to 
encapsulate other lists here, we can just add the operator to the existing list 
in the operator?



ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompile

Re: Review Request 71820: HIVE-20150

2019-12-03 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 3, 2019, 9:27 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestCommonKeyPrefix.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f75361df1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out 
bb3b1b6660 
  ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 
228b20a8d7 
  ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out 
5268ed3ecf 
  ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out 
d96222d9e1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out 
adabb76e04 
  ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out 
824bbe6769 
  ql/src/test/results/clientpositive/perf/tez/const

Re: Review Request 71820: HIVE-20150

2019-12-02 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Dec. 3, 2019, 6:28 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestCommonKeyPrefix.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f75361df1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out 
bb3b1b6660 
  ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 
228b20a8d7 
  ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out 
5268ed3ecf 
  ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out 
d96222d9e1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out 
adabb76e04 
  ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out 
824bbe6769 
  ql/src/test/results/clientpositive/perf/tez/const

Re: Review Request 71820: HIVE-20150

2019-11-29 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Nov. 29, 2019, 1:27 p.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestCommonKeyPrefix.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 735296f814 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out b90fb55fd1 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f75361df1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out 
7ad55b5c5f 
  ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 
9b07ceb3a7 
  ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out 
7c0e6cf8cd 
  ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out 
44fd104a9b 
  ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out 
1ca55ca290 
  ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out 
90a56acafc 
  ql/src/test/results/clientpositive/perf/tez/cons

Re: Review Request 71820: HIVE-20150

2019-11-28 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Nov. 28, 2019, 8 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
4b4cf99503 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestCommonKeyPrefix.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 735296f814 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out b90fb55fd1 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f75361df1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out 
7ad55b5c5f 
  ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 
9b07ceb3a7 
  ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out 
7c0e6cf8cd 
  ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out 
44fd104a9b 
  ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out 
1ca55ca290 
  ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out 
90a56acafc 
  ql/src/test/results/clientpositive/perf/tez/constra

Re: Review Request 71820: HIVE-20150

2019-11-27 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

(Updated Nov. 27, 2019, 10:20 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description (updated)
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
4b4cf99503 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestCommonKeyPrefix.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 735296f814 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out b90fb55fd1 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/query10.q.out 30628f4829 
  ql/src/test/results/clientpositive/perf/tez/query15.q.out 418bffddca 
  ql/src/test/results/clientpositive/perf/tez/query17.q.out e7eda149e8 
  ql/src/test/results/clientpositive/perf/tez/query27.q.out d915989654 
  ql/src/test/results/clientpositive/perf/tez/query35.q.out 1936f22b61 
  ql/src/test/results/clientpositive/perf/tez/query40.q.out 61f3d7c748 
  ql/src/test/results/clientpositive/perf/tez/query43.q.out 2e6f86c0a0 
  ql/src/test/results/clientpositive/perf/tez/query45.q.out ab7181d7a4 
  ql/src/test/results/clientpositive/perf/tez/query50.q.out 2a287ce6d3 
  ql/src/test/results/clientpositive/perf/tez/query66.q.out 23ffd637a3 
  ql/src/test/results/clientpositive/perf/tez/query69.q.out d27f4060f6 
  ql/src/test/results/clientpositive/tez/topnkey.q.out 45947d0be8 
  ql/src/test/results/clientpositive/tez/vector_topnkey.q.out d179013e28 
  ql/src/test/results/clientpositive/topnkey.q.out cecbe89b1c 


Diff: https://reviews.apache.org/r/71820/diff/2/

Changes: https://reviews.apache.org/r/71820/diff/1-2/


Testing
---

Run q tests using TestMiniLlapLocalCliDriver
topnkey.q
vector_topnkey.q


Thanks,

Krisztian Kasa

Review Request 71820: HIVE-20150

2019-11-26 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
---

Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
---

TopNKey pushdown

1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
4b4cf99503 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestCommonKeyPrefix.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 735296f814 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out b90fb55fd1 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/query10.q.out 30628f4829 
  ql/src/test/results/clientpositive/perf/tez/query15.q.out 418bffddca 
  ql/src/test/results/clientpositive/perf/tez/query17.q.out e7eda149e8 
  ql/src/test/results/clientpositive/perf/tez/query27.q.out d915989654 
  ql/src/test/results/clientpositive/perf/tez/query35.q.out 1936f22b61 
  ql/src/test/results/clientpositive/perf/tez/query40.q.out 61f3d7c748 
  ql/src/test/results/clientpositive/perf/tez/query43.q.out 2e6f86c0a0 
  ql/src/test/results/clientpositive/perf/tez/query45.q.out ab7181d7a4 
  ql/src/test/results/clientpositive/perf/tez/query50.q.out 2a287ce6d3 
  ql/src/test/results/clientpositive/perf/tez/query66.q.out 23ffd637a3 
  ql/src/test/results/clientpositive/perf/tez/query69.q.out d27f4060f6 
  ql/src/test/results/clientpositive/tez/topnkey.q.out 45947d0be8 
  ql/src/test/results/clientpositive/tez/vector_topnkey.q.out d179013e28 
  ql/src/test/results/clientpositive/topnkey.q.out cecbe89b1c 


Diff: https://reviews.apache.org/r/71820/diff/1/


Testing
---

Run q tests using TestMiniLlapLocalCliDriver
topnkey.q
vector_topnkey.q


Thanks,

Krisztian Kasa

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Re: Review Request 71820: HIVE-20150

Review Request 71820: HIVE-20150

20 matches

Site Navigation

Mail list logo

Footer information