ekoifman commented on pull request #30829:
URL: https://github.com/apache/spark/pull/30829#issuecomment-758358291
I agree that automatically computing the cost of the new shuffle is hard.
That is why this first step uses an option to enable this behavior - based on
previous runs of a query or report, for example. I think the mechanics of
adding a new shuffle have to be built first before it can become a cost based
decision.
I'm not sure what you mean by "skip counting"
`OptimizeSkewedJoin` runs on the part of the plan bounded by `Exchange`
above (call it E) and completed `QueryStageExec` leaves below.
So if `OptimizeSkewedJoin` makes a change that needs a new shuffle, it
happens in the Nth recursive invocation of `createQueryStages`. So it figures
out where the new `Exchange` should be and throws `ShuffleAddedException` to
pop back out to the top, add the new Exchange in the right place and start
`createQueryStages` again from the top. Now that there is a new Exchange (call
it E') is in the plan and below E, and the same `QueryStageExec` leaves,
`newQueryStage(E')` is run.
This seems like the easiest way to preserve the flow of the existing
algorithm.
The
```
var continue = true
while (continue) {
```
in `getFinalPhysicalPlan` is because in a complex plan there may be several
new stages created during one iteration of
```while (!result.allChildStagesMaterialized) {``` and each may potentially
introduce a new shuffle.
The processing the final stage (before any of my changes) is slightly
different - `finalStageOptimizerRules` to preserve properties of the output, so
`handleFinalStageShuffle` does the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
