sunchao opened a new pull request, #57128: URL: https://github.com/apache/spark/pull/57128
### What changes were proposed in this pull request? This is a follow-up to #57087. `OptimizeShuffleWithLocalRead` now checks the mapper-oriented partition specs it proposes against `spark.sql.adaptive.coalescePartitions.maxReducerPartitionsPerTask`. If any `PartialMapperPartitionSpec` spans too many reducer partitions, or any `CoalescedMapperPartitionSpec` reads too many reducers, the rule preserves the existing bounded shuffle read. Safe local-read rewrites are unchanged. The patch also adds an AQE regression test that exercises both mapper-oriented local-read spec forms and verifies that the bounded coalesced reads remain in the final plan. ### Why are the changes needed? SPARK-58004 added a hard limit on the number of reducer partitions that AQE can combine into one task. However, the local shuffle reader runs after shuffle partition coalescing and can replace the bounded `CoalescedPartitionSpec` values with mapper-oriented specs whose reducer spans exceed that limit. Without this follow-up, enabling local shuffle reads can therefore bypass the new fan-in bound and recreate the excessive per-task fetch pressure that #57087 is intended to prevent. ### Does this PR introduce _any_ user-facing change? Yes, compared with the current unreleased `master` branch. When a proposed local shuffle read would exceed `spark.sql.adaptive.coalescePartitions.maxReducerPartitionsPerTask`, Spark now keeps the existing coalesced shuffle read instead. Released Spark versions are not affected. ### How was this patch tested? Added and ran the focused regression test: ``` build/sbt 'sql/testOnly org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite -- -z "Preserve the reducer partition limit through local shuffle read optimization"' ``` Ran the adjacent positive-path local shuffle reader test: ``` build/sbt 'sql/testOnly org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite -- -z "Reuse the parallelism of coalesced shuffle in local shuffle read"' ``` Ran style checks: ``` build/sbt sql/scalastyle sql/Test/scalastyle ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
