----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65174/ -----------------------------------------------------------
Review request for hive. Bugs: HIVE-17896 https://issues.apache.org/jira/browse/HIVE-17896 Repository: hive-git Description ------- For TPC-DS Query27, the TopN operation is delayed by the group-by - the group-by operator buffers up all the rows before discarding the 99% of the rows in the TopN Hash within the ReduceSink Operator. The RS TopN operator is very restrictive as it only supports doing the filtering on the shuffle keys, but it is better to do this before breaking the vectors into rows and losing the isRepeating properties. Adding a TopN Key operator in the physical operator tree allows the following to happen. GBY->RS(Top=1) can become TNK(1)->GBY->RS(Top=1) So that, the TopNKey can remove rows before they are buffered into the GBY and consume memory. Here's the equivalent implementation in Presto https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 Adding this as a sub-feature of GroupBy prevents further optimizations if the GBY is on keys "a,b,c" and the TopNKey is on just "a". Diffs ----- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 631c83644a itests/src/test/resources/testconfiguration.properties bd2416f4eb ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java a002348013 ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapperFactory.java 775c737bba ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java e665064133 ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdown.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 6f289703a7 ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java be33f38003 ql/src/java/org/apache/hadoop/hive/ql/plan/TopNKeyDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/VectorTopNKeyDesc.java PRE-CREATION ql/src/test/queries/clientpositive/topnkey.q PRE-CREATION ql/src/test/queries/clientpositive/vector_topnkey.q PRE-CREATION ql/src/test/results/clientpositive/llap/topnkey.q.out PRE-CREATION ql/src/test/results/clientpositive/llap/vector_topnkey.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/topnkey.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/vector_topnkey.q.out PRE-CREATION ql/src/test/results/clientpositive/topnkey.q.out PRE-CREATION ql/src/test/results/clientpositive/vector_topnkey.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 698ebe1ffe Diff: https://reviews.apache.org/r/65174/diff/1/ Testing ------- Thanks, Teddy Choi