xuyu-co opened a new pull request, #52052: URL: https://github.com/apache/spark/pull/52052
### What changes were proposed in this pull request? add identifiers and make an isolation between normal shuffle and shuffle converted from `REBALANCE` hint to control Shuffle partitionSizeInBytes converted from `REBALANCE` hint ### Why are the changes needed? As we all known, The `REBALANCE` hint can be used to rebalance the query result output partitions, which is only effective when AQE is enabled and will be converted to a ShuffleExchangeExec. When both normal shuffles，which are probably provided by SQL itself or extra `REPARTITION` hint and `REBALANCE` hint exist simultaneously, all of them will be converted to ShuffleExchangeExec and then AQE's ShuffleQueryStageExec. Even more exaggerated is that if we change "spark.sql.adaptive.advisoryPartitionSizeInBytes", ALL the Shuffle partitions num and size will be changed!!! In this scenario，the roles of `REPARTITION` hint and `REBALANCE` hint are similar. It is common for us to modify this parameter to control the number of final output files' num and reduce the issue of small files. So we want to make a config isolation on "spark.sql.adaptive.advisoryPartitionSizeInBytes", which is extremely useful when we use `REBALANCE` hint to control output partitions，**AND only effects the Shuffle converted from `REBALANCE` hint, will not effect the other Shuffle** ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Test in spark environment and UT ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
