ulysses-you commented on pull request #32468:
URL: https://github.com/apache/spark/pull/32468#issuecomment-840227088
Thank you for introducing this idea. But that's not only about
`spark.sql.adaptive.coalescePartitions.minPartitionNum ` but also other
adaptive configs, such as `spark.sql.adaptive.coalescePartitions.enabled`,
`spark.sql.adaptive.advisoryPartitionSizeInBytes`.
Follow this thought, we can split stages into two parts to make configs
isolated with each other:
* final stage (for write)
* previous stage (for query)
And the code may look like:
```
withSQLConf("final stage config") {
applyPhysicalRules(
result.newPlan,
finalStageOptimizerRules,
Some((planChangeLogger, "AQE Final Query Stage Optimization")))
}
```
The thought is not very mature but it's quite useful in our scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
