Hello Quanlong Huang, Aman Sinha, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#10).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, partition constants ignored)". An EXPR_REWRITE match keeps
its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
The ratio goes through a new PrintUtils.printTwoDecimalsRatio, which
formats under Locale.ROOT. It is there for a tool to read rather than
for the eye, and a coordinator whose JVM default is, say, de_DE would
write 0,05 instead. The cardinalities beside it use the default locale
like every other estimate in a plan, so this is about the one field
that claims to be machine-readable, not about the line as a whole.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- PrintUtilsTest, HboKeyStringTest, HistoricalStatsTest: 28 tests, 0
failures. Formatting the ratio under the default locale instead fails
testPrintTwoDecimalsRatio alone, with "expected:<0[.]05> but
was:<0[,]05>"
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M
testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
9 files changed, 233 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/10
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 10
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
