cloud-fan commented on pull request #30829: URL: https://github.com/apache/spark/pull/30829#issuecomment-762836967
This seems like a big change to the AQE framework. If it's rare to force apply the skew join handling, how about we start with a sub-optimal solution: we just add the extra shuffle and do not create a query stage for this shuffle. Under the hood, `ShuffleQueryStageExec` submits a map stage, which can contain shuffle as well. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
