Aleksandr Efimov created IMPALA-15312:
-----------------------------------------
Summary: Show when an HBO cardinality changed a plan decision
Key: IMPALA-15312
URL: https://issues.apache.org/jira/browse/IMPALA-15312
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Aleksandr Efimov
h3. Problem
IMPALA-15236 puts the provenance of an HBO cardinality next to the number: the
canonicalization strategy that matched, the hash key, the estimate the planner
had computed on its own, and the ratio between the two. That answers where the
number came from. It does not answer whether the number changed the plan.
The two questions come apart. Decisions that consume cardinalities are cost
based, so a corrected estimate can flip one while leaving the shape of the plan
alone. Join inversion is the clearest case: {{Planner.invertJoins()}} inverts a
join when {{isInvertedJoinCheaper()}} says so, and that check is a function of
the two children's cardinality and average row size. Correct the build side and
the verdict can flip, so a join that was planned inverted goes back. The number
of joins does not change, the broadcast/partitioned mix does not change, and
the operator histogram of the plan does not change - only the side each join
builds on.
This is not hypothetical. On an outer-join-heavy query, HBO correcting a
build-side aggregate from a few thousand rows down to fourteen turned sixty
joins back from RIGHT OUTER to LEFT OUTER, and a comparison of the two plans by
operator counts reported them as the same plan.
Answering "what did HBO do to this plan" today means running the query twice,
once with {{use_hbo_stats=false}}, and diffing the two plans. On a report that
runs for minutes that is a measurement exercise rather than debugging.
h3. Proposed change
Mark the nodes where the HBO substitution changed a planner decision, not only
the estimate, in the plan and in the profile.
Two directions, each with a cost worth weighing:
* Plan twice under a debug query option and diff the results. Exact, and only
paid for when someone asks for it.
* Carry the pre-HBO estimate through the cardinality computation and
re-evaluate the cost based decisions with it. No second planning pass, but it
has to propagate: a node whose own key did not match can still get a different
cardinality because a descendant matched, and
{{PlanNode.cardinalityBeforeHbo_}} stays -1 on such a node, so a check that
reads only that field reports no change where there was one.
h3. Note for the comparison itself
Plan node ids are handed out as nodes are constructed, before {{invertJoins()}}
runs, and exchange nodes are numbered afterwards by the distributed planner.
Two plans of the same query built from different estimates need not agree on
ids, so a diff keyed on node id can report differences that are only numbering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
