ulysses-you commented on PR #36483:
URL: https://github.com/apache/spark/pull/36483#issuecomment-1120638033
We do not need a new shuffle origin and the current
`OptimizeShuffleWithLocalRead` almost supports this case except one thing:
The `OptimizeShuffleWithLocalRead` only match the root plan who is a shuffle
or shuffle reader, but the case you point is a Project which is from the
original output of the Join (join is converted to project). So the physical
plan after optimized looks like:
```
Project
AQEShuffleReader
...
```
It seems we can enhance `OptimizeShuffleWithLocalRead` that support optimzie
the shuffle which is under the Project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
