Alex Behm has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10444
Change subject: IMPALA-110: Planner support for multiple distinct aggregations. ...................................................................... IMPALA-110: Planner support for multiple distinct aggregations. Adds planner support for multiple distinct aggregations in a single SELECT block. Design summary: - The existing tree-based plan shape with a two-phased aggregation is maintained. - Existing plans are not changed. - A single query block may contain multiple distinct aggregates. - Aggregates are grouped into "aggregation classes" based on their expressions in the distinct portion which may be empty for non-distinct aggregates. - The aggregation framework is generalized to simultaneously process multiple aggregation classes within the tree-based plan. This process splits the results of different aggregation classes into separate rows, so a final aggregation is needed to transpose the results into the desired form. - Main challenge: Each aggregation class consumes and produces different tuples, so conceptually a union-type of tuples flows through the runtime. The tuple union is represented by a TupleRow with one tuple per aggregation class. Only one tuple in such a TupleRow is non-NULL. - Backend exec nodes in the aggregation plan will be aware of this tuple-union either explicitly in their implementation or by relying on expressions that distinguish the aggregation classes. - To distinguish the aggregation classes, e.g. in hash exchanges, CASE expressions are crafted to hash/group on the appropriate slots. Deferred FE work: - Beautify/condense the long CASE exprs - Push applicable conjuncts into individual aggregators before the transposition step - Added a few testing TODOs to reduce the size of this patch - Decide whether we want to change existing plans to the new model Testing: - Added analyzer and planner tests - Ran end-to-end queries based on a prototype BE implementation - Ran hdfs/core tests Change-Id: I4c5cb348f9431350d2e5bf2c84325dcc44d38d2f --- M be/src/exec/partitioned-aggregation-node.cc M be/src/exec/partitioned-aggregation-node.h M be/src/exprs/CMakeLists.txt M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/aggregate-functions.h M be/src/exprs/scalar-expr.cc A be/src/exprs/valid-tuple-id.cc A be/src/exprs/valid-tuple-id.h M common/thrift/Exprs.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/AggregateInfoBase.java M fe/src/main/java/org/apache/impala/analysis/Expr.java A fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java A fe/src/main/java/org/apache/impala/analysis/ValidTupleIdExpr.java M fe/src/main/java/org/apache/impala/catalog/AggregateFunction.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/workloads/functional-planner/queries/PlannerTest/distinct.test A testdata/workloads/functional-planner/queries/PlannerTest/multiple-distinct-limit.test A testdata/workloads/functional-planner/queries/PlannerTest/multiple-distinct-materialization.test A testdata/workloads/functional-planner/queries/PlannerTest/multiple-distinct-predicates.test A testdata/workloads/functional-planner/queries/PlannerTest/multiple-distinct.test 37 files changed, 4,809 insertions(+), 591 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/10444/1 -- To view, visit http://gerrit.cloudera.org:8080/10444 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4c5cb348f9431350d2e5bf2c84325dcc44d38d2f Gerrit-Change-Number: 10444 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Behm <[email protected]>
