deniskuzZ opened a new pull request, #6596:
URL: https://github.com/apache/hive/pull/6596
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA:
https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a
faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section
is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster
reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class
hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other
DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
The HEP planner's cumulative row-count cost is data-movement-blind
(cpu=io=0). The extended cost model (hive.cbo.costmodel.extended) does
model CPU and IO, but it only rescores plans after the fact: the greedy
ordering in HiveLoptOptimizeJoinRule is driven by factor weights with a
max-NDV tie-break and never consults it, so the shuffle-heavy ordering
is generated either way (runtime-validated on TPC-DS query24, where
enabling the extended model brought no improvement). The rule can thus
emit orderings that shuffle a large fact table raw before the selective
dimension joins that would have reduced it, inflating the bytes crossing
the network by orders of magnitude on queries like query24, query59 and
query72.
This adds a shuffle-cost model to the join reordering, enabled by
hive.cbo.join.reorder.shuffle.cost (default true):
* Generation: on a factor-weight tie in getBestNextFactor, prefer a
factor whose build side fits the map-join size budget
(hive.auto.convert.join.noconditionaltask.size), producing a
fact-reducing ordering the legacy max-NDV tie-break never yields.
* Selection: from the pool of N legacy orderings plus N
broadcast-preferring variants, emit only the plan with the lowest
estimated shuffle cost. A broadcast-eligible join (INNER, hash-joinable
key, build side under the threshold) is charged its build side;
anything else is charged both inputs. Outer joins are deliberately
never modelled as broadcasts: letting the model move them on the
strength of estimates alone was runtime-validated as harmful (an
estimate-better outer-join placement in query72 ran 14x slower).
* Fail-safe: unknown statistics make a candidate uncostable (NaN) and
any estimation failure logs a warning and falls back to emitting the
legacy N orderings unchanged.
The reordering only applies when map-join conversion is enabled, since
the cost model ranks orderings against broadcast joins the physical
planner must be able to materialize.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Performance regression in TPC-DS 1Tb after adding constraints
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as
the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes
- provide the console output, description, screenshot and/or a reproducable
example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to
the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some
test cases that check the changes thoroughly including negative and positive
cases if possible.
If it was tested in a way different from regular unit tests, please clarify
how you tested step by step, ideally copy and paste-able, so that other
reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why
it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
