cloud-fan commented on pull request #32210: URL: https://github.com/apache/spark/pull/32210#issuecomment-823919561
> We enabled shuffled hash join by default with this feature. In our environment, roughly 25% of sort merge join queries are now running with shuffled hash join after enabling shuffled hash join by default. Roughly 20% of shuffled hash join queries has tasks fall-backed. Roughly 10% of shuffled hash join tasks has fall-backed. TLDR is it saves 20% of shuffled hash join queries failure, and 10% of tasks failure. This looks great, let's add the fallback then. Picking SHJ in AQE is orthogonal to it and we can have both. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
