ulysses-you commented on a change in pull request #30138:
URL: https://github.com/apache/spark/pull/30138#discussion_r628711132
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanHelper.scala
##########
@@ -139,15 +138,21 @@ trait AdaptiveSparkPlanHelper {
}
/**
- * Returns a cloned [[SparkSession]] with adaptive execution disabled, or
the original
- * [[SparkSession]] if its adaptive execution is already disabled.
+ * Returns a cloned [[SparkSession]] with all specified configurations
disabled, or
+ * the original [[SparkSession]] if all configurations are already disabled.
*/
- def getOrCloneSessionWithAqeOff[T](session: SparkSession): SparkSession = {
- if (!session.sessionState.conf.adaptiveExecutionEnabled) {
+ def getOrCloneSessionWithConfigsOff(
Review comment:
@c21 I have such thought at first, but cann't find a negative case. Can
you point out a case that can cause correctness bug ?
If I don's miss something, it just affect the perfermance about extra
shuffle. For correctness, let's assuming a cache plan with AQE enabled:
* For lazy cache. the AQE framework will ensure the correctness of the new
query with the cached plan .
* For force cache. if the output paritioning or ordering of cached plan has
been affected by AQE then Spark will use `EnsureRequirements` to promise the
correctness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
