HeartSaVioR commented on pull request #35574: URL: https://github.com/apache/spark/pull/35574#issuecomment-1047383695
Another rough idea in parallel: let's consider the physical nodes in the same stage. Unless the stage contains the source node, the stage should have triggered shuffle, based on the required child distribution of the first physical node. If all nodes in the same stage require `ClusteredDistribution`s but having different group keys, it might be sensible expectation that picking up most number of grouping keys to the represent the required child distribution for "this stage" would bring less skews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
