-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
-----------------------------------------------------------

(Updated Dec. 9, 2019, 3:32 p.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-20150
    https://issues.apache.org/jira/browse/HIVE-20150


Repository: hive-git


Description
-------

TopNKey pushdown
================
1. Apply patch: 
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the 
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key 
prefix only.
6. turn off topnkey optimization by default


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7724f9084 
  data/conf/perf-reg/tez/hive-site.xml ab945f5f95 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
0d6cf3c755 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
  ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
  ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
b6d210becf 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
9343e078b7 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
  ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
545cce75a9 
  ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
  ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out 
ffe5f6fb22 
  ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
4ecb7bc46d 
  ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out 
0eac389eb7 
  
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
 4362fb6f2e 
  ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out 
24468c9a1b 
  
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out 
45890a1890 
  ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3 
  ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d 
  ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82 
  ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280 
  ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 6fd15e7101 
  ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out 
d6325982e3 
  ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out 
4d417b9c3d 
  ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out 
97a211cfc6 
  ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37 
  ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522 
  ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709 
  ql/src/test/results/clientpositive/perf/tez/cbo_query54.q.out e96d4c60a6 
  ql/src/test/results/clientpositive/perf/tez/cbo_query58.q.out 523f02e7d7 
  ql/src/test/results/clientpositive/perf/tez/cbo_query6.q.out 944b113e7b 
  ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
43e1b2b5c2 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query54.q.out 
ece76c3e4e 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query58.q.out 
954bdc4dd1 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query6.q.out 
eb872d7e6d 
  ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
2f75361df1 
  ql/src/test/results/clientpositive/perf/tez/constraints/mv_query44.q.out 
61c01c5c60 
  ql/src/test/results/clientpositive/perf/tez/constraints/query1.q.out 
3aea12dda6 
  ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out 
bb3b1b6660 
  ql/src/test/results/clientpositive/perf/tez/constraints/query11.q.out 
cc47c7e7a1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query12.q.out 
57c16347e5 
  ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 
228b20a8d7 
  ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out 
5268ed3ecf 
  ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out 
d96222d9e1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query18.q.out 
c3c17a2efc 
  ql/src/test/results/clientpositive/perf/tez/constraints/query19.q.out 
fff6c4951d 
  ql/src/test/results/clientpositive/perf/tez/constraints/query1b.q.out 
a6276715b4 
  ql/src/test/results/clientpositive/perf/tez/constraints/query20.q.out 
b71831c7c6 
  ql/src/test/results/clientpositive/perf/tez/constraints/query21.q.out 
af2e33e785 
  ql/src/test/results/clientpositive/perf/tez/constraints/query22.q.out 
cd3c0cc431 
  ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out 
adabb76e04 
  ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out 
824bbe6769 
  ql/src/test/results/clientpositive/perf/tez/constraints/query27.q.out 
abbd02d6c9 
  ql/src/test/results/clientpositive/perf/tez/constraints/query29.q.out 
c308771dfb 
  ql/src/test/results/clientpositive/perf/tez/constraints/query3.q.out 
4c1f168168 
  ql/src/test/results/clientpositive/perf/tez/constraints/query30.q.out 
ea1622883b 
  ql/src/test/results/clientpositive/perf/tez/constraints/query33.q.out 
f00567eeb1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query35.q.out 
23b3399123 
  ql/src/test/results/clientpositive/perf/tez/constraints/query36.q.out 
f7bccf94f7 
  ql/src/test/results/clientpositive/perf/tez/constraints/query37.q.out 
187ad5c5b5 
  ql/src/test/results/clientpositive/perf/tez/constraints/query4.q.out 
68796c2274 
  ql/src/test/results/clientpositive/perf/tez/constraints/query40.q.out 
070b5cb1f5 
  ql/src/test/results/clientpositive/perf/tez/constraints/query42.q.out 
b2dc7b3a8a 
  ql/src/test/results/clientpositive/perf/tez/constraints/query43.q.out 
b5a6c746d1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query44.q.out 
a243417159 
  ql/src/test/results/clientpositive/perf/tez/constraints/query45.q.out 
3f5dbf4beb 
  ql/src/test/results/clientpositive/perf/tez/constraints/query46.q.out 
804013fb80 
  ql/src/test/results/clientpositive/perf/tez/constraints/query47.q.out 
701861a888 
  ql/src/test/results/clientpositive/perf/tez/constraints/query49.q.out 
b384aea779 
  ql/src/test/results/clientpositive/perf/tez/constraints/query5.q.out 
d3f79820f2 
  ql/src/test/results/clientpositive/perf/tez/constraints/query50.q.out 
8c9754967f 
  ql/src/test/results/clientpositive/perf/tez/constraints/query51.q.out 
4255d5cc33 
  ql/src/test/results/clientpositive/perf/tez/constraints/query52.q.out 
56d73b8d46 
  ql/src/test/results/clientpositive/perf/tez/constraints/query53.q.out 
f8f495d3a8 
  ql/src/test/results/clientpositive/perf/tez/constraints/query54.q.out 
d85597af02 
  ql/src/test/results/clientpositive/perf/tez/constraints/query55.q.out 
fb52de2aba 
  ql/src/test/results/clientpositive/perf/tez/constraints/query56.q.out 
cd438d4ae3 
  ql/src/test/results/clientpositive/perf/tez/constraints/query57.q.out 
7896f97fed 
  ql/src/test/results/clientpositive/perf/tez/constraints/query58.q.out 
5fa126bf6e 
  ql/src/test/results/clientpositive/perf/tez/constraints/query59.q.out 
82148531bc 
  ql/src/test/results/clientpositive/perf/tez/constraints/query6.q.out 
d6031153b1 
  ql/src/test/results/clientpositive/perf/tez/constraints/query60.q.out 
06a5689938 
  ql/src/test/results/clientpositive/perf/tez/constraints/query63.q.out 
8d45d8260d 
  ql/src/test/results/clientpositive/perf/tez/constraints/query65.q.out 
8b8a4146a6 
  ql/src/test/results/clientpositive/perf/tez/constraints/query66.q.out 
be612609cf 
  ql/src/test/results/clientpositive/perf/tez/constraints/query67.q.out 
ee1bf12068 
  ql/src/test/results/clientpositive/perf/tez/constraints/query68.q.out 
97b97ce19d 
  ql/src/test/results/clientpositive/perf/tez/constraints/query69.q.out 
d7469ae5a9 
  ql/src/test/results/clientpositive/perf/tez/constraints/query7.q.out 
b2eccdbe90 
  ql/src/test/results/clientpositive/perf/tez/constraints/query70.q.out 
d5efae9aeb 
  ql/src/test/results/clientpositive/perf/tez/constraints/query72.q.out 
9d6a3f6a00 
  ql/src/test/results/clientpositive/perf/tez/constraints/query74.q.out 
68ff9ebf26 
  ql/src/test/results/clientpositive/perf/tez/constraints/query75.q.out 
b1341a2fdb 
  ql/src/test/results/clientpositive/perf/tez/constraints/query76.q.out 
ce4f7cb061 
  ql/src/test/results/clientpositive/perf/tez/constraints/query77.q.out 
95ab61bed2 
  ql/src/test/results/clientpositive/perf/tez/constraints/query78.q.out 
ecb434d3cd 
  ql/src/test/results/clientpositive/perf/tez/constraints/query79.q.out 
9e9bbec9e2 
  ql/src/test/results/clientpositive/perf/tez/constraints/query8.q.out 
170bccf406 
  ql/src/test/results/clientpositive/perf/tez/constraints/query80.q.out 
b18f89373c 
  ql/src/test/results/clientpositive/perf/tez/constraints/query81.q.out 
2b859c230c 
  ql/src/test/results/clientpositive/perf/tez/constraints/query82.q.out 
8dd6ae9f0f 
  ql/src/test/results/clientpositive/perf/tez/constraints/query83.q.out 
a2163802ad 
  ql/src/test/results/clientpositive/perf/tez/constraints/query84.q.out 
89b11e66e2 
  ql/src/test/results/clientpositive/perf/tez/constraints/query85.q.out 
6bd833fd12 
  ql/src/test/results/clientpositive/perf/tez/constraints/query86.q.out 
0b6ee2c156 
  ql/src/test/results/clientpositive/perf/tez/constraints/query89.q.out 
28d604a433 
  ql/src/test/results/clientpositive/perf/tez/constraints/query93.q.out 
193a542b53 
  ql/src/test/results/clientpositive/perf/tez/constraints/query99.q.out 
c77a73f4d5 
  ql/src/test/results/clientpositive/perf/tez/query1.q.out f7affa4f14 
  ql/src/test/results/clientpositive/perf/tez/query10.q.out b346a5c5fb 
  ql/src/test/results/clientpositive/perf/tez/query11.q.out 96bdd7b504 
  ql/src/test/results/clientpositive/perf/tez/query12.q.out 4183059b2d 
  ql/src/test/results/clientpositive/perf/tez/query14.q.out 069fad2b4a 
  ql/src/test/results/clientpositive/perf/tez/query15.q.out 3670a718b3 
  ql/src/test/results/clientpositive/perf/tez/query17.q.out df70fbc46e 
  ql/src/test/results/clientpositive/perf/tez/query18.q.out d61a57d469 
  ql/src/test/results/clientpositive/perf/tez/query19.q.out 5cdeab2bf0 
  ql/src/test/results/clientpositive/perf/tez/query1b.q.out b7975918c3 
  ql/src/test/results/clientpositive/perf/tez/query20.q.out 2553c1ed76 
  ql/src/test/results/clientpositive/perf/tez/query21.q.out 5e158834d7 
  ql/src/test/results/clientpositive/perf/tez/query22.q.out e721f09f82 
  ql/src/test/results/clientpositive/perf/tez/query25.q.out d006795c79 
  ql/src/test/results/clientpositive/perf/tez/query26.q.out a1bf3b099b 
  ql/src/test/results/clientpositive/perf/tez/query27.q.out 6f49de2344 
  ql/src/test/results/clientpositive/perf/tez/query29.q.out 5066893829 
  ql/src/test/results/clientpositive/perf/tez/query3.q.out fe97d402c4 
  ql/src/test/results/clientpositive/perf/tez/query30.q.out ac683c2515 
  ql/src/test/results/clientpositive/perf/tez/query33.q.out dc4f6b17c1 
  ql/src/test/results/clientpositive/perf/tez/query35.q.out 265c51bb72 
  ql/src/test/results/clientpositive/perf/tez/query36.q.out 544d883846 
  ql/src/test/results/clientpositive/perf/tez/query37.q.out 2724fd44dc 
  ql/src/test/results/clientpositive/perf/tez/query4.q.out 99552295f0 
  ql/src/test/results/clientpositive/perf/tez/query40.q.out 4b65c82e00 
  ql/src/test/results/clientpositive/perf/tez/query42.q.out 3c059f85f6 
  ql/src/test/results/clientpositive/perf/tez/query43.q.out eb19d41926 
  ql/src/test/results/clientpositive/perf/tez/query44.q.out 38a281c8f9 
  ql/src/test/results/clientpositive/perf/tez/query45.q.out 4538a6540d 
  ql/src/test/results/clientpositive/perf/tez/query46.q.out 3e6358d907 
  ql/src/test/results/clientpositive/perf/tez/query47.q.out e54c366848 
  ql/src/test/results/clientpositive/perf/tez/query49.q.out 9c34eccceb 
  ql/src/test/results/clientpositive/perf/tez/query5.q.out 38fba27a8e 
  ql/src/test/results/clientpositive/perf/tez/query50.q.out 6e34831de6 
  ql/src/test/results/clientpositive/perf/tez/query51.q.out 906d702207 
  ql/src/test/results/clientpositive/perf/tez/query52.q.out b579d5fbff 
  ql/src/test/results/clientpositive/perf/tez/query53.q.out 1bee5e099c 
  ql/src/test/results/clientpositive/perf/tez/query54.q.out a19bd171e5 
  ql/src/test/results/clientpositive/perf/tez/query55.q.out 5cee50ed50 
  ql/src/test/results/clientpositive/perf/tez/query56.q.out 2be886a3eb 
  ql/src/test/results/clientpositive/perf/tez/query57.q.out b732cc873f 
  ql/src/test/results/clientpositive/perf/tez/query58.q.out 57f3477e48 
  ql/src/test/results/clientpositive/perf/tez/query59.q.out 566f888d40 
  ql/src/test/results/clientpositive/perf/tez/query6.q.out 5c49ab4517 
  ql/src/test/results/clientpositive/perf/tez/query60.q.out e77c89ba69 
  ql/src/test/results/clientpositive/perf/tez/query63.q.out 3cdf37b734 
  ql/src/test/results/clientpositive/perf/tez/query65.q.out dbc5c79a87 
  ql/src/test/results/clientpositive/perf/tez/query66.q.out 7ddcc21f92 
  ql/src/test/results/clientpositive/perf/tez/query67.q.out cbacaae150 
  ql/src/test/results/clientpositive/perf/tez/query68.q.out b9321679a9 
  ql/src/test/results/clientpositive/perf/tez/query69.q.out d11b5494e0 
  ql/src/test/results/clientpositive/perf/tez/query7.q.out c17ec8aeb9 
  ql/src/test/results/clientpositive/perf/tez/query70.q.out 7fe3a62399 
  ql/src/test/results/clientpositive/perf/tez/query72.q.out bebc3852b7 
  ql/src/test/results/clientpositive/perf/tez/query74.q.out f98b9c8815 
  ql/src/test/results/clientpositive/perf/tez/query75.q.out b426bf2d1a 
  ql/src/test/results/clientpositive/perf/tez/query76.q.out c0d60e88cc 
  ql/src/test/results/clientpositive/perf/tez/query77.q.out ab2b3dc570 
  ql/src/test/results/clientpositive/perf/tez/query78.q.out 4d090af463 
  ql/src/test/results/clientpositive/perf/tez/query79.q.out e1ef37ad38 
  ql/src/test/results/clientpositive/perf/tez/query8.q.out 0af8fdf3df 
  ql/src/test/results/clientpositive/perf/tez/query80.q.out 47844158fa 
  ql/src/test/results/clientpositive/perf/tez/query81.q.out 7feb6175fe 
  ql/src/test/results/clientpositive/perf/tez/query82.q.out c7721acffe 
  ql/src/test/results/clientpositive/perf/tez/query83.q.out 54c1ba6161 
  ql/src/test/results/clientpositive/perf/tez/query84.q.out 4fd546bc07 
  ql/src/test/results/clientpositive/perf/tez/query85.q.out c411a92145 
  ql/src/test/results/clientpositive/perf/tez/query86.q.out 37414f697f 
  ql/src/test/results/clientpositive/perf/tez/query89.q.out 927da252da 
  ql/src/test/results/clientpositive/perf/tez/query93.q.out 0df41d120b 
  ql/src/test/results/clientpositive/perf/tez/query99.q.out c01122f435 
  ql/src/test/results/clientpositive/tez/topnkey.q.out cf2ecf7133 
  ql/src/test/results/clientpositive/tez/vector_topnkey.q.out d179013e28 
  ql/src/test/results/clientpositive/topnkey.q.out cecbe89b1c 


Diff: https://reviews.apache.org/r/71820/diff/8/

Changes: https://reviews.apache.org/r/71820/diff/7-8/


Testing
-------

Run q tests using TestMiniLlapLocalCliDriver
topnkey.q
vector_topnkey.q


Thanks,

Krisztian Kasa

Reply via email to