Hello Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24297
to look at the new patch set (#10).
Change subject: IMPALA-14600: Support HBO for AggregationNode cardinality
......................................................................
IMPALA-14600: Support HBO for AggregationNode cardinality
This extends HBO to support tracking and using cardinality of
AggregationNodes.
Frontend Changes
----------------
The HBO key string of AggregationNode consists of
- Logical AggPhase: FIRST, SECOND, TRANSPOSE.
- isPreagg, isGroupingSet: true/false.
- For each AggClass: canonicalized grouping exprs.
- AggClasses are sorted by their key strings.
- Conjuncts in HAVING clause.
- The HBO Key string of the real child (explained below).
Some fields are ignored since they decide how the output should be
processed and are unrelated to the output cardinality of current node,
e.g. isDistinctAgg, needsFinalize.
Also adds a concept of cardinality-preserving nodes for nodes that
always have inputCardinality == outputCardinality and won't impact
cardinality of ancestor nodes, e.g. SortNode, ExchangeNode,
TupleCacheNode. These nodes can be ignored in generating the HBO key
strings. "Ignored" here means using the key string of their child which
acts like they don't exist. Note that AnalyticEvalNode isn't kind of
such nodes. Though its outputCardinality is always the same as its
inputCardinality, the analytic function results it adds can be used as
group-by or join keys and impact ancestor nodes.
When adding the child key of AggregationNode, intermediate agg nodes
that belong to the same logical aggregation and cardinality preserving
nodes are ignored. We use the "real" child of the aggregation which
currently can only be a scan node or another aggregation (from different
multiAggInfo_ instance). Note that HBO doesn't support other node types
like JoinNode yet.
This is not just an optimization to simplify (and shorten) the HBO key
string, but also a correctness requirement since intermediate agg nodes
and ExchangeNodes could be added after the cardinality is computed. E.g.
After SingleNodePlanner computes the cardinality, DistributedPlanner
adds intermediate agg nodes and computes the cardinality again. The HBO
key string should be consistent in both the SingleNodePlanner and
DistributedPlanner.
Backend Changes
---------------
Most of the backend logic remains unchanged except for checking nodes
that have effective external runtime filters under the subtree.
"Effective" means the filter skips some rows on the target node.
"External" here means the runtime filter is generated outside the
subtree.
If a scan node has effective runtime filters, we travel through its
parents recursively until the source node of the runtime filter. These
nodes except the source node are marked as having effective external
runtime filters. Their cardinalities are unstable and won't be stored
into the HBO stats.
To help this traveling, FE adds the parent node id to TPlanNode.
Testing
- Added Java unit tests to verify the HBO key strings.
- Added e2e tests in test_hbo.py
- Updated golden test files of scan cardinality due to some agg nodes
now also have HBO stats.
Assisted-by: Claude Code 4.6
Assisted-by: Composer 2
Change-Id: Ie0fafaf9d827f3bf533b1af7e62fdb2303c126ce
---
M be/src/service/impala-server.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/ExprCanonicalizer.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
A fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-collection-scan.test
M
testdata/workloads/functional-query/queries/QueryTest/hbo-disjunct-selectivity.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-distinct-agg.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-grouping-set.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-iceberg-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-single-agg.test
M
testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-nonpartitioned-no-stats.test
M
testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-nonpartitioned-stats.test
M
testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-no-stats.test
M
testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
22 files changed, 1,107 insertions(+), 109 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/24297/10
--
To view, visit http://gerrit.cloudera.org:8080/24297
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0fafaf9d827f3bf533b1af7e62fdb2303c126ce
Gerrit-Change-Number: 24297
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
