-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
-----------------------------------------------------------

(Updated Dec. 9, 2019, 10:21 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
    https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
-------

TopNKey pushdown
================
1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7724f9084 
  data/conf/perf-reg/tez/hive-site.xml ab945f5f95 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f75361df1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out 
bb3b1b6660 
  ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 
228b20a8d7 
  ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out 
5268ed3ecf 
  ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out 
d96222d9e1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out 
adabb76e04 
  ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out 
824bbe6769 
  ql/src/test/results/clientpositive/perf/tez/constraints/query27.q.out 
abbd02d6c9 
  ql/src/test/results/clientpositive/perf/tez/constraints/query29.q.out 
c308771dfb 
  ql/src/test/results/clientpositive/perf/tez/constraints/query35.q.out 
23b3399123 
  ql/src/test/results/clientpositive/perf/tez/constraints/query37.q.out 
187ad5c5b5 
  ql/src/test/results/clientpositive/perf/tez/constraints/query40.q.out 
070b5cb1f5 
  ql/src/test/results/clientpositive/perf/tez/constraints/query43.q.out 
b5a6c746d1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query45.q.out 
3f5dbf4beb 
  ql/src/test/results/clientpositive/perf/tez/constraints/query49.q.out 
b384aea779 
  ql/src/test/results/clientpositive/perf/tez/constraints/query5.q.out 
d3f79820f2 
  ql/src/test/results/clientpositive/perf/tez/constraints/query50.q.out 
8c9754967f 
  ql/src/test/results/clientpositive/perf/tez/constraints/query60.q.out 
06a5689938 
  ql/src/test/results/clientpositive/perf/tez/constraints/query66.q.out 
be612609cf 
  ql/src/test/results/clientpositive/perf/tez/constraints/query69.q.out 
d7469ae5a9 
  ql/src/test/results/clientpositive/perf/tez/constraints/query7.q.out 
b2eccdbe90 
  ql/src/test/results/clientpositive/perf/tez/constraints/query76.q.out 
ce4f7cb061 
  ql/src/test/results/clientpositive/perf/tez/constraints/query77.q.out 
95ab61bed2 
  ql/src/test/results/clientpositive/perf/tez/constraints/query8.q.out 
170bccf406 
  ql/src/test/results/clientpositive/perf/tez/constraints/query80.q.out 
b18f89373c 
  ql/src/test/results/clientpositive/perf/tez/constraints/query82.q.out 
8dd6ae9f0f 
  ql/src/test/results/clientpositive/perf/tez/constraints/query99.q.out 
c77a73f4d5 
  ql/src/test/results/clientpositive/perf/tez/query10.q.out b346a5c5fb 
  ql/src/test/results/clientpositive/perf/tez/query14.q.out 069fad2b4a 
  ql/src/test/results/clientpositive/perf/tez/query15.q.out 3670a718b3 
  ql/src/test/results/clientpositive/perf/tez/query17.q.out df70fbc46e 
  ql/src/test/results/clientpositive/perf/tez/query25.q.out d006795c79 
  ql/src/test/results/clientpositive/perf/tez/query26.q.out a1bf3b099b 
  ql/src/test/results/clientpositive/perf/tez/query27.q.out 6f49de2344 
  ql/src/test/results/clientpositive/perf/tez/query29.q.out 5066893829 
  ql/src/test/results/clientpositive/perf/tez/query35.q.out 265c51bb72 
  ql/src/test/results/clientpositive/perf/tez/query37.q.out 2724fd44dc 
  ql/src/test/results/clientpositive/perf/tez/query40.q.out 4b65c82e00 
  ql/src/test/results/clientpositive/perf/tez/query43.q.out eb19d41926 
  ql/src/test/results/clientpositive/perf/tez/query45.q.out 4538a6540d 
  ql/src/test/results/clientpositive/perf/tez/query49.q.out 9c34eccceb 
  ql/src/test/results/clientpositive/perf/tez/query5.q.out 38fba27a8e 
  ql/src/test/results/clientpositive/perf/tez/query50.q.out 6e34831de6 
  ql/src/test/results/clientpositive/perf/tez/query60.q.out e77c89ba69 
  ql/src/test/results/clientpositive/perf/tez/query66.q.out 7ddcc21f92 
  ql/src/test/results/clientpositive/perf/tez/query69.q.out d11b5494e0 
  ql/src/test/results/clientpositive/perf/tez/query7.q.out c17ec8aeb9 
  ql/src/test/results/clientpositive/perf/tez/query76.q.out c0d60e88cc 
  ql/src/test/results/clientpositive/perf/tez/query77.q.out ab2b3dc570 
  ql/src/test/results/clientpositive/perf/tez/query8.q.out 0af8fdf3df 
  ql/src/test/results/clientpositive/perf/tez/query80.q.out 47844158fa 
  ql/src/test/results/clientpositive/perf/tez/query82.q.out c7721acffe 
  ql/src/test/results/clientpositive/perf/tez/query99.q.out c01122f435 
  ql/src/test/results/clientpositive/tez/topnkey.q.out cf2ecf7133 
  ql/src/test/results/clientpositive/tez/vector_topnkey.q.out d179013e28 
  ql/src/test/results/clientpositive/topnkey.q.out cecbe89b1c 


Diff: https://reviews.apache.org/r/71820/diff/7/

Changes: https://reviews.apache.org/r/71820/diff/6-7/


Testing
-------

Run q tests using TestMiniLlapLocalCliDriver
topnkey.q
vector_topnkey.q


Thanks,

Krisztian Kasa

Reply via email to