manuzhang opened a new pull request #29540: URL: https://github.com/apache/spark/pull/29540
### What changes were proposed in this pull request? When the minimum number of coalesced partitions is not set in AQE, we do not fall back to default parallelism but coalesce partitions towards the `advisoryTargetSize`. ### Why are the changes needed? The default parallelism is usually the number of total executors cores which might be different across different runs of the same job when dynamic allocation is enabled. It results in uncertainty of the number of tasks after coalescing and leads to many small output files in some cases. The behavior is complex and hard to reason about. Hence, I'm proposing not falling back to the default parallelism but coalescing towards the `advisoryTargetSize` when the minimum number of coalesced partitions is not set. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing Tests. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
