-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/
-----------------------------------------------------------
(Updated Dec. 10, 2019, 7:38 a.m.)
Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
Bugs: HIVE-20150
https://issues.apache.org/jira/browse/HIVE-20150
Repository: hive-git
Description
-------
TopNKey pushdown
================
1. Apply patch:
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
3. Implement TopNKey operator pushdown through: projection, group by, redeuce
sink, left outer join, other topnkey
4. Add sort order and null sort order direction check when determining if the
topnkey op can be pushed
5. Implement handling cases when topnkey op and the parent op has a common key
prefix only.
6. turn off topnkey optimization by default
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java
0d6cf3c755
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java bf58bd8bb8
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
PRE-CREATION
ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba
ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6
ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b
ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0
ql/src/test/results/clientpositive/llap/constraints_optimization.q.out
b6d210becf
ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out
9343e078b7
ql/src/test/results/clientpositive/llap/explainuser_1.q.out 283a665a20
ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833
ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out
545cce75a9
ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0
ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802
ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c
ql/src/test/results/clientpositive/llap/llap_decimal64_reader.q.out
ffe5f6fb22
ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5
ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out
4ecb7bc46d
ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out
0eac389eb7
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out
4362fb6f2e
ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out
24468c9a1b
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out
45890a1890
ql/src/test/results/clientpositive/llap/semijoin_reddedup.q.out 0e9723b8f3
ql/src/test/results/clientpositive/llap/subquery_ALL.q.out d910c1a79d
ql/src/test/results/clientpositive/llap/subquery_ANY.q.out 91472d631e
ql/src/test/results/clientpositive/llap/topnkey.q.out 1e77587f82
ql/src/test/results/clientpositive/llap/vector_cast_constant.q.out cc2dc47280
ql/src/test/results/clientpositive/llap/vector_char_2.q.out f7e76e5a8b
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
6fd15e7101
ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
d6325982e3
ql/src/test/results/clientpositive/llap/vector_mr_diff_schema_alias.q.out
4d417b9c3d
ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out
97a211cfc6
ql/src/test/results/clientpositive/llap/vector_string_concat.q.out a8019be7aa
ql/src/test/results/clientpositive/llap/vector_topnkey.q.out c140bdfd37
ql/src/test/results/clientpositive/llap/vectorization_limit.q.out 7326adf522
ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out e9308cd709
ql/src/test/results/clientpositive/perf/tez/cbo_query54.q.out e96d4c60a6
ql/src/test/results/clientpositive/perf/tez/cbo_query58.q.out 523f02e7d7
ql/src/test/results/clientpositive/perf/tez/cbo_query6.q.out 944b113e7b
ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out 02caf99f7d
ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out
43e1b2b5c2
ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query54.q.out
ece76c3e4e
ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query58.q.out
954bdc4dd1
ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query6.q.out
eb872d7e6d
ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out
2f75361df1
ql/src/test/results/clientpositive/perf/tez/constraints/mv_query44.q.out
61c01c5c60
ql/src/test/results/clientpositive/perf/tez/constraints/query1.q.out
3aea12dda6
ql/src/test/results/clientpositive/perf/tez/constraints/query10.q.out
bb3b1b6660
ql/src/test/results/clientpositive/perf/tez/constraints/query11.q.out
cc47c7e7a1
ql/src/test/results/clientpositive/perf/tez/constraints/query12.q.out
57c16347e5
ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out
228b20a8d7
ql/src/test/results/clientpositive/perf/tez/constraints/query15.q.out
5268ed3ecf
ql/src/test/results/clientpositive/perf/tez/constraints/query17.q.out
d96222d9e1
ql/src/test/results/clientpositive/perf/tez/constraints/query18.q.out
c3c17a2efc
ql/src/test/results/clientpositive/perf/tez/constraints/query19.q.out
fff6c4951d
ql/src/test/results/clientpositive/perf/tez/constraints/query1b.q.out
a6276715b4
ql/src/test/results/clientpositive/perf/tez/constraints/query20.q.out
b71831c7c6
ql/src/test/results/clientpositive/perf/tez/constraints/query21.q.out
af2e33e785
ql/src/test/results/clientpositive/perf/tez/constraints/query22.q.out
cd3c0cc431
ql/src/test/results/clientpositive/perf/tez/constraints/query25.q.out
adabb76e04
ql/src/test/results/clientpositive/perf/tez/constraints/query26.q.out
824bbe6769
ql/src/test/results/clientpositive/perf/tez/constraints/query27.q.out
abbd02d6c9
ql/src/test/results/clientpositive/perf/tez/constraints/query29.q.out
c308771dfb
ql/src/test/results/clientpositive/perf/tez/constraints/query3.q.out
4c1f168168
ql/src/test/results/clientpositive/perf/tez/constraints/query30.q.out
ea1622883b
ql/src/test/results/clientpositive/perf/tez/constraints/query33.q.out
f00567eeb1
ql/src/test/results/clientpositive/perf/tez/constraints/query35.q.out
23b3399123
ql/src/test/results/clientpositive/perf/tez/constraints/query36.q.out
f7bccf94f7
ql/src/test/results/clientpositive/perf/tez/constraints/query37.q.out
187ad5c5b5
ql/src/test/results/clientpositive/perf/tez/constraints/query4.q.out
68796c2274
ql/src/test/results/clientpositive/perf/tez/constraints/query40.q.out
070b5cb1f5
ql/src/test/results/clientpositive/perf/tez/constraints/query42.q.out
b2dc7b3a8a
ql/src/test/results/clientpositive/perf/tez/constraints/query43.q.out
b5a6c746d1
ql/src/test/results/clientpositive/perf/tez/constraints/query44.q.out
a243417159
ql/src/test/results/clientpositive/perf/tez/constraints/query45.q.out
3f5dbf4beb
ql/src/test/results/clientpositive/perf/tez/constraints/query46.q.out
804013fb80
ql/src/test/results/clientpositive/perf/tez/constraints/query47.q.out
701861a888
ql/src/test/results/clientpositive/perf/tez/constraints/query49.q.out
b384aea779
ql/src/test/results/clientpositive/perf/tez/constraints/query5.q.out
d3f79820f2
ql/src/test/results/clientpositive/perf/tez/constraints/query50.q.out
8c9754967f
ql/src/test/results/clientpositive/perf/tez/constraints/query51.q.out
4255d5cc33
ql/src/test/results/clientpositive/perf/tez/constraints/query52.q.out
56d73b8d46
ql/src/test/results/clientpositive/perf/tez/constraints/query53.q.out
f8f495d3a8
ql/src/test/results/clientpositive/perf/tez/constraints/query54.q.out
d85597af02
ql/src/test/results/clientpositive/perf/tez/constraints/query55.q.out
fb52de2aba
ql/src/test/results/clientpositive/perf/tez/constraints/query56.q.out
cd438d4ae3
ql/src/test/results/clientpositive/perf/tez/constraints/query57.q.out
7896f97fed
ql/src/test/results/clientpositive/perf/tez/constraints/query58.q.out
5fa126bf6e
ql/src/test/results/clientpositive/perf/tez/constraints/query59.q.out
82148531bc
ql/src/test/results/clientpositive/perf/tez/constraints/query6.q.out
d6031153b1
ql/src/test/results/clientpositive/perf/tez/constraints/query60.q.out
06a5689938
ql/src/test/results/clientpositive/perf/tez/constraints/query63.q.out
8d45d8260d
ql/src/test/results/clientpositive/perf/tez/constraints/query65.q.out
8b8a4146a6
ql/src/test/results/clientpositive/perf/tez/constraints/query66.q.out
be612609cf
ql/src/test/results/clientpositive/perf/tez/constraints/query67.q.out
ee1bf12068
ql/src/test/results/clientpositive/perf/tez/constraints/query68.q.out
97b97ce19d
ql/src/test/results/clientpositive/perf/tez/constraints/query69.q.out
d7469ae5a9
ql/src/test/results/clientpositive/perf/tez/constraints/query7.q.out
b2eccdbe90
ql/src/test/results/clientpositive/perf/tez/constraints/query70.q.out
d5efae9aeb
ql/src/test/results/clientpositive/perf/tez/constraints/query72.q.out
9d6a3f6a00
ql/src/test/results/clientpositive/perf/tez/constraints/query74.q.out
68ff9ebf26
ql/src/test/results/clientpositive/perf/tez/constraints/query75.q.out
b1341a2fdb
ql/src/test/results/clientpositive/perf/tez/constraints/query76.q.out
ce4f7cb061
ql/src/test/results/clientpositive/perf/tez/constraints/query77.q.out
95ab61bed2
ql/src/test/results/clientpositive/perf/tez/constraints/query78.q.out
ecb434d3cd
ql/src/test/results/clientpositive/perf/tez/constraints/query79.q.out
9e9bbec9e2
ql/src/test/results/clientpositive/perf/tez/constraints/query8.q.out
170bccf406
ql/src/test/results/clientpositive/perf/tez/constraints/query80.q.out
b18f89373c
ql/src/test/results/clientpositive/perf/tez/constraints/query81.q.out
2b859c230c
ql/src/test/results/clientpositive/perf/tez/constraints/query82.q.out
8dd6ae9f0f
ql/src/test/results/clientpositive/perf/tez/constraints/query83.q.out
a2163802ad
ql/src/test/results/clientpositive/perf/tez/constraints/query84.q.out
89b11e66e2
ql/src/test/results/clientpositive/perf/tez/constraints/query85.q.out
6bd833fd12
ql/src/test/results/clientpositive/perf/tez/constraints/query86.q.out
0b6ee2c156
ql/src/test/results/clientpositive/perf/tez/constraints/query89.q.out
28d604a433
ql/src/test/results/clientpositive/perf/tez/constraints/query93.q.out
193a542b53
ql/src/test/results/clientpositive/perf/tez/constraints/query99.q.out
c77a73f4d5
ql/src/test/results/clientpositive/perf/tez/query1.q.out f7affa4f14
ql/src/test/results/clientpositive/perf/tez/query10.q.out b346a5c5fb
ql/src/test/results/clientpositive/perf/tez/query11.q.out 96bdd7b504
ql/src/test/results/clientpositive/perf/tez/query12.q.out 4183059b2d
ql/src/test/results/clientpositive/perf/tez/query14.q.out 069fad2b4a
ql/src/test/results/clientpositive/perf/tez/query15.q.out 3670a718b3
ql/src/test/results/clientpositive/perf/tez/query17.q.out df70fbc46e
ql/src/test/results/clientpositive/perf/tez/query18.q.out d61a57d469
ql/src/test/results/clientpositive/perf/tez/query19.q.out 5cdeab2bf0
ql/src/test/results/clientpositive/perf/tez/query1b.q.out b7975918c3
ql/src/test/results/clientpositive/perf/tez/query20.q.out 2553c1ed76
ql/src/test/results/clientpositive/perf/tez/query21.q.out 5e158834d7
ql/src/test/results/clientpositive/perf/tez/query22.q.out e721f09f82
ql/src/test/results/clientpositive/perf/tez/query25.q.out d006795c79
ql/src/test/results/clientpositive/perf/tez/query26.q.out a1bf3b099b
ql/src/test/results/clientpositive/perf/tez/query27.q.out 6f49de2344
ql/src/test/results/clientpositive/perf/tez/query29.q.out 5066893829
ql/src/test/results/clientpositive/perf/tez/query3.q.out fe97d402c4
ql/src/test/results/clientpositive/perf/tez/query30.q.out ac683c2515
ql/src/test/results/clientpositive/perf/tez/query33.q.out dc4f6b17c1
ql/src/test/results/clientpositive/perf/tez/query35.q.out 265c51bb72
ql/src/test/results/clientpositive/perf/tez/query36.q.out 544d883846
ql/src/test/results/clientpositive/perf/tez/query37.q.out 2724fd44dc
ql/src/test/results/clientpositive/perf/tez/query4.q.out 99552295f0
ql/src/test/results/clientpositive/perf/tez/query40.q.out 4b65c82e00
ql/src/test/results/clientpositive/perf/tez/query42.q.out 3c059f85f6
ql/src/test/results/clientpositive/perf/tez/query43.q.out eb19d41926
ql/src/test/results/clientpositive/perf/tez/query44.q.out 38a281c8f9
ql/src/test/results/clientpositive/perf/tez/query45.q.out 4538a6540d
ql/src/test/results/clientpositive/perf/tez/query46.q.out 3e6358d907
ql/src/test/results/clientpositive/perf/tez/query47.q.out e54c366848
ql/src/test/results/clientpositive/perf/tez/query49.q.out 9c34eccceb
ql/src/test/results/clientpositive/perf/tez/query5.q.out 38fba27a8e
ql/src/test/results/clientpositive/perf/tez/query50.q.out 6e34831de6
ql/src/test/results/clientpositive/perf/tez/query51.q.out 906d702207
ql/src/test/results/clientpositive/perf/tez/query52.q.out b579d5fbff
ql/src/test/results/clientpositive/perf/tez/query53.q.out 1bee5e099c
ql/src/test/results/clientpositive/perf/tez/query54.q.out a19bd171e5
ql/src/test/results/clientpositive/perf/tez/query55.q.out 5cee50ed50
ql/src/test/results/clientpositive/perf/tez/query56.q.out 2be886a3eb
ql/src/test/results/clientpositive/perf/tez/query57.q.out b732cc873f
ql/src/test/results/clientpositive/perf/tez/query58.q.out 57f3477e48
ql/src/test/results/clientpositive/perf/tez/query59.q.out 566f888d40
ql/src/test/results/clientpositive/perf/tez/query6.q.out 5c49ab4517
ql/src/test/results/clientpositive/perf/tez/query60.q.out e77c89ba69
ql/src/test/results/clientpositive/perf/tez/query63.q.out 3cdf37b734
ql/src/test/results/clientpositive/perf/tez/query65.q.out dbc5c79a87
ql/src/test/results/clientpositive/perf/tez/query66.q.out 7ddcc21f92
ql/src/test/results/clientpositive/perf/tez/query67.q.out cbacaae150
ql/src/test/results/clientpositive/perf/tez/query68.q.out b9321679a9
ql/src/test/results/clientpositive/perf/tez/query69.q.out d11b5494e0
ql/src/test/results/clientpositive/perf/tez/query7.q.out c17ec8aeb9
ql/src/test/results/clientpositive/perf/tez/query70.q.out 7fe3a62399
ql/src/test/results/clientpositive/perf/tez/query72.q.out bebc3852b7
ql/src/test/results/clientpositive/perf/tez/query74.q.out f98b9c8815
ql/src/test/results/clientpositive/perf/tez/query75.q.out b426bf2d1a
ql/src/test/results/clientpositive/perf/tez/query76.q.out c0d60e88cc
ql/src/test/results/clientpositive/perf/tez/query77.q.out ab2b3dc570
ql/src/test/results/clientpositive/perf/tez/query78.q.out 4d090af463
ql/src/test/results/clientpositive/perf/tez/query79.q.out e1ef37ad38
ql/src/test/results/clientpositive/perf/tez/query8.q.out 0af8fdf3df
ql/src/test/results/clientpositive/perf/tez/query80.q.out 47844158fa
ql/src/test/results/clientpositive/perf/tez/query81.q.out 7feb6175fe
ql/src/test/results/clientpositive/perf/tez/query82.q.out c7721acffe
ql/src/test/results/clientpositive/perf/tez/query83.q.out 54c1ba6161
ql/src/test/results/clientpositive/perf/tez/query84.q.out 4fd546bc07
ql/src/test/results/clientpositive/perf/tez/query85.q.out c411a92145
ql/src/test/results/clientpositive/perf/tez/query86.q.out 37414f697f
ql/src/test/results/clientpositive/perf/tez/query89.q.out 927da252da
ql/src/test/results/clientpositive/perf/tez/query93.q.out 0df41d120b
ql/src/test/results/clientpositive/perf/tez/query99.q.out c01122f435
ql/src/test/results/clientpositive/tez/topnkey.q.out cf2ecf7133
ql/src/test/results/clientpositive/tez/vector_topnkey.q.out d179013e28
ql/src/test/results/clientpositive/topnkey.q.out cecbe89b1c
Diff: https://reviews.apache.org/r/71820/diff/10/
Changes: https://reviews.apache.org/r/71820/diff/9-10/
Testing
-------
Run q tests using TestMiniLlapLocalCliDriver
topnkey.q
vector_topnkey.q
Thanks,
Krisztian Kasa
