-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65174/
-----------------------------------------------------------

Review request for hive.


Bugs: HIVE-17896
    https://issues.apache.org/jira/browse/HIVE-17896


Repository: hive-git


Description
-------

For TPC-DS Query27, the TopN operation is delayed by the group-by - the 
group-by operator buffers up all the rows before discarding the 99% of the rows 
in the TopN Hash within the ReduceSink Operator.
The RS TopN operator is very restrictive as it only supports doing the 
filtering on the shuffle keys, but it is better to do this before breaking the 
vectors into rows and losing the isRepeating properties.
Adding a TopN Key operator in the physical operator tree allows the following 
to happen.
GBY->RS(Top=1)
can become
TNK(1)->GBY->RS(Top=1)
So that, the TopNKey can remove rows before they are buffered into the GBY and 
consume memory.
Here's the equivalent implementation in Presto
https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35
Adding this as a sub-feature of GroupBy prevents further optimizations if the 
GBY is on keys "a,b,c" and the TopNKey is on just "a".


Diffs
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 631c83644a 
  itests/src/test/resources/testconfiguration.properties bd2416f4eb 
  
ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java
 a002348013 
  ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapperFactory.java 775c737bba 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java e665064133 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyPushdown.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
6f289703a7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java be33f38003 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TopNKeyDesc.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/VectorTopNKeyDesc.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey.q PRE-CREATION 
  ql/src/test/queries/clientpositive/vector_topnkey.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/topnkey.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/vector_topnkey.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/topnkey.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/vector_topnkey.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/topnkey.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/vector_topnkey.q.out PRE-CREATION 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
 698ebe1ffe 


Diff: https://reviews.apache.org/r/65174/diff/1/


Testing
-------


Thanks,

Teddy Choi

Reply via email to