zhengruifeng opened a new pull request #33893: URL: https://github.com/apache/spark/pull/33893
### What changes were proposed in this pull request? This PR aims to generalize `OptimizeSkewedJoin` to support all patterns that can be handled by current _split-duplicate_ method: 1, select the _splitttable_ shuffle query stages by the semantic of internal nodes; 2, for each splitttable shuffle query stages, check whether skew exists, if true, split the partitions; 3, handle _Combinatorial Explosion_: for each skew partition, check whether the combination number is too large, if so, re-split the stages to keep a reasonable number of combinations. For example, for partition 0, stage A/B/C are split to 100/100/100 specs, respectively. Then there are 1M combinations, which is too large, and will cause performance regression. 4, attach new spec to query stages; ### Why are the changes needed? to Generalize OptimizeSkewedJoin ### Does this PR introduce _any_ user-facing change? two additional configs are added ### How was this patch tested? existing testsuites and added testsuites -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
