-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
-----------------------------------------------------------
(Updated Nov. 28, 2019, 8 a.m.)
Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150
Repository: hive-git
Description
-------
TopNKey pushdown
================
1. Apply patch:
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key
prefix only.
6. turn off topnkey optimization by default
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/CommonKeyPrefix.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
4b4cf99503
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdownProcessor.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestCommonKeyPrefix.java
PRE-CREATION
ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba
ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6
ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b
ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0
ql/src/test/results/clientpositive/llap/constraints_optimization.q.out
b6d210becf
ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out
9343e078b7
ql/src/test/results/clientpositive/llap/explainuser_1.q.out 735296f814
ql/src/test/results/clientpositive/llap/explainuser_2.q.out b90fb55fd1
ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out
545cce75a9
ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0
ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802
ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c
ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out
ffe5f6fb22
ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5
ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out
4ecb7bc46d
ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out
0eac389eb7
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
4362fb6f2e
ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out
24468c9a1b
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out
45890a1890
ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3
ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d
ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e
ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82
ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280
ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
6fd15e7101
ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
d6325982e3
ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out
4d417b9c3d
ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out
97a211cfc6
ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa
ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37
ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522
ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709
ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d
ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out
43e1b2b5c2
ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out
2f75361df1
ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out
7ad55b5c5f
ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out
9b07ceb3a7
ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out
7c0e6cf8cd
ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out
44fd104a9b
ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out
1ca55ca290
ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out
90a56acafc
ql/src/test/results/clientpositive/perf/tez/constraints/query27.q.out
58b2b86cdf
ql/src/test/results/clientpositive/perf/tez/constraints/query29.q.out
073ae52f1f
ql/src/test/results/clientpositive/perf/tez/constraints/query35.q.out
4ca9d3cb61
ql/src/test/results/clientpositive/perf/tez/constraints/query37.q.out
83759fd6d8
ql/src/test/results/clientpositive/perf/tez/constraints/query40.q.out
8dff7e8984
ql/src/test/results/clientpositive/perf/tez/constraints/query43.q.out
f4ead2363c
ql/src/test/results/clientpositive/perf/tez/constraints/query45.q.out
b73dc876da
ql/src/test/results/clientpositive/perf/tez/constraints/query49.q.out
db3129dca0
ql/src/test/results/clientpositive/perf/tez/constraints/query5.q.out
629d2ff013
ql/src/test/results/clientpositive/perf/tez/constraints/query50.q.out
0e210fb5cc
ql/src/test/results/clientpositive/perf/tez/constraints/query60.q.out
f2b494e706
ql/src/test/results/clientpositive/perf/tez/constraints/query66.q.out
4889093f21
ql/src/test/results/clientpositive/perf/tez/constraints/query69.q.out
71bb17e27a
ql/src/test/results/clientpositive/perf/tez/constraints/query7.q.out
e4e28188dd
ql/src/test/results/clientpositive/perf/tez/constraints/query70.q.out
6fc46fbd83
ql/src/test/results/clientpositive/perf/tez/constraints/query76.q.out
08c26cf74f
ql/src/test/results/clientpositive/perf/tez/constraints/query77.q.out
0f1eafc4d8
ql/src/test/results/clientpositive/perf/tez/constraints/query8.q.out
f036baf048
ql/src/test/results/clientpositive/perf/tez/constraints/query80.q.out
d4e913cb4f
ql/src/test/results/clientpositive/perf/tez/constraints/query82.q.out
185629c47c
ql/src/test/results/clientpositive/perf/tez/constraints/query99.q.out
edcf1c20d7
ql/src/test/results/clientpositive/perf/tez/query10.q.out 30628f4829
ql/src/test/results/clientpositive/perf/tez/query14.q.out abfc5e85d4
ql/src/test/results/clientpositive/perf/tez/query15.q.out 418bffddca
ql/src/test/results/clientpositive/perf/tez/query17.q.out e7eda149e8
ql/src/test/results/clientpositive/perf/tez/query25.q.out 201de7c376
ql/src/test/results/clientpositive/perf/tez/query26.q.out 4d26c4be5b
ql/src/test/results/clientpositive/perf/tez/query27.q.out d915989654
ql/src/test/results/clientpositive/perf/tez/query29.q.out 8d947e280b
ql/src/test/results/clientpositive/perf/tez/query35.q.out 1936f22b61
ql/src/test/results/clientpositive/perf/tez/query37.q.out 53b1cee165
ql/src/test/results/clientpositive/perf/tez/query40.q.out 61f3d7c748
ql/src/test/results/clientpositive/perf/tez/query43.q.out 2e6f86c0a0
ql/src/test/results/clientpositive/perf/tez/query45.q.out ab7181d7a4
ql/src/test/results/clientpositive/perf/tez/query49.q.out aefacb03f4
ql/src/test/results/clientpositive/perf/tez/query5.q.out a345d5caf1
ql/src/test/results/clientpositive/perf/tez/query50.q.out 2a287ce6d3
ql/src/test/results/clientpositive/perf/tez/query60.q.out 87eaa70d50
ql/src/test/results/clientpositive/perf/tez/query66.q.out 23ffd637a3
ql/src/test/results/clientpositive/perf/tez/query69.q.out d27f4060f6
ql/src/test/results/clientpositive/perf/tez/query7.q.out c00b543a0e
ql/src/test/results/clientpositive/perf/tez/query70.q.out 26b1ca1f5d
ql/src/test/results/clientpositive/perf/tez/query76.q.out c49be8e699
ql/src/test/results/clientpositive/perf/tez/query77.q.out ca68776a34
ql/src/test/results/clientpositive/perf/tez/query8.q.out b35b381a82
ql/src/test/results/clientpositive/perf/tez/query80.q.out e831f68dcf
ql/src/test/results/clientpositive/perf/tez/query82.q.out c481ce47fe
ql/src/test/results/clientpositive/perf/tez/query99.q.out fadd30cd14
ql/src/test/results/clientpositive/tez/topnkey.q.out 45947d0be8
ql/src/test/results/clientpositive/tez/vector_topnkey.q.out d179013e28
ql/src/test/results/clientpositive/topnkey.q.out cecbe89b1c
Diff: https://reviews.apache.org/r/71820/diff/3/
Changes: https://reviews.apache.org/r/71820/diff/2-3/
Testing
-------
Run q tests using TestMiniLlapLocalCliDriver
topnkey.q
vector_topnkey.q
Thanks,
Krisztian Kasa
