Hello Aleksandr Efimov, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24592
to look at the new patch set (#12).
Change subject: IMPALA-15189: Support HBO for SortNode cardinality
......................................................................
IMPALA-15189: Support HBO for SortNode cardinality
By default, a SortNode leaves the total row count unchanged. However,
when combined with a limit (TopN, Partitioned TopN, or a total sort with
a LIMIT), the output row count decreases. This extends HBO to support
tracking and using the cardinality of these limit-applying SortNodes.
The HBO key string of a SortNode consists of
- Type: TOTAL, PARTIAL, TOPN, PARTITIONED_TOPN
- IncludeTies: true/false for Top-N
-- Only true for RANK() with LIMIT pushdown.
- OrderKind: LEXICAL, ZORDER
- Limit and Offset:
-- LIMIT, OFFSET for non-PARTITIONED_TOPN
-- PerPartitionLimit for PARTITIONED_TOPN where OFFSET is always 0.
- Partition Exprs for PARTITIONED_TOPN
- Order Exprs
- Child key
Local sort nodes in a distributed sort plan are skipped in the HBO key
string for two reasons:
- to ensure SingleNodePlanner and DistributedPlanner use the same key
string for the merge/final sort node, either a final Top-N or a
merging-exchange.
- Backend tracks the summed cross-instance cardinality for local sort
nodes but the planner needs single instance cardinality which is the
same as the cardinality of the final sort node.
To be specific, local sort node and final sort node shares the same HBO
key but only final sort node writes the HBO stats. The final sort node
can be a merging-exchange with SortInfo, LIMIT and OFFSET. Such
ExchangeNodes are not cardinality-preserving so we should generate HBO
key strings for them. To make all kinds of ExchangeNode delegate the HBO
key string generation to its child, i.e. regardless of whether it's
cardinality-preserving, this patch renames isCardinalityPreserving() to
ignoredInHboCardKey() so ExchangeNode can always return true in it.
To verify the cardinality of MERGING-EXCHANGE which only shown in
EXPLAIN output if explain_level >= 2, this patch extends
_run_hbo_explains() to support DISTRIBUTEDPLAN sections in the golden
test files.
Testing
- Added FE tests on HBO key strings
- Added e2e tests
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ib829887a91593bee124d56e661e714575fe3be97
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A
testdata/workloads/functional-query/queries/QueryTest/hbo-agg-on-topn-sort.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-analytic-topn.test
A
testdata/workloads/functional-query/queries/QueryTest/hbo-offset-only-sort.test
M tests/query_test/test_hbo.py
12 files changed, 775 insertions(+), 42 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/24592/12
--
To view, visit http://gerrit.cloudera.org:8080/24592
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib829887a91593bee124d56e661e714575fe3be97
Gerrit-Change-Number: 24592
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
