This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 1dcefc6709c3 [SPARK-59023][SQL] Support none mode in
spark.sql.ui.explainMode to skip plan description in SQL events
1dcefc6709c3 is described below
commit 1dcefc6709c3f220988f268399c47c30b044de11
Author: Cheng Pan <[email protected]>
AuthorDate: Fri Aug 28 10:26:03 2026 +0800
[SPARK-59023][SQL] Support none mode in spark.sql.ui.explainMode to skip
plan description in SQL events
### What changes were proposed in this pull request?
Add a `none` option to `spark.sql.ui.explainMode`. When it is set,
`SparkListenerSQLExecutionStart` and `SparkListenerSQLAdaptiveExecutionUpdate`
carry a placeholder string instead of the rendered plan description:
```
No plan description because spark.sql.ui.explainMode=none
```
This skips the explain string generation for the SQL UI entirely. The
change touches:
- `SQLConf.UI_EXPLAIN_MODE`: accepts `none` in addition to `simple`,
`extended`, `codegen`, `cost`, `formatted` (doc and validation updated)
- `SQLExecution`: hosts `planDescription`, which returns the placeholder
instead of calling `QueryExecution.explainString` when the mode is `none`
- `AdaptiveSparkPlanExec`: uses the same helper for the plan description
posted with adaptive execution updates
### Why are the changes needed?
Rendering the plan description is done solely for the SQL UI. For workloads
with large plans, or with adaptive execution enabled (where the plan
description is re-rendered on every adaptive update), this generation is
non-trivial overhead that users who do not use the SQL UI cannot avoid today,
since every valid mode renders something. `none` lets them opt out of the cost.
For example, we have a customer job that constructs a huge plan whose
`treeString` exceeds 280,000 lines. Rendering the plan description takes more
than 3 minutes per iteration, and with AQE enabled, assembling the plan tree
strings takes more than 40 minutes across the query execution. A string this
large also pressures driver memory (we have observed driver OOM), and the
browser becomes unresponsive when the SQL UI tries to render it.
`spark.sql.maxPlanStringLength` does not address this.
`StringConcat.append` only skips storing text past the limit, and
`TreeNode.generateTreeString` has no early exit, so every node's string is
still materialized. It bounds the final string's memory, not the CPU spent
building it.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.sql.ui.explainMode` now accepts `none`. When set, the SQL UI
shows the placeholder text above instead of the query plan description for SQL
executions. The default value (`formatted`) is unchanged.
### How was this patch tested?
The existing `control a plan explain mode in listeners via SQLConf` tests
are extended with a `none` case, asserting the placeholder is carried by the
events:
- `AdaptiveQueryExecSuite` (covers
`SparkListenerSQLAdaptiveExecutionUpdate`)
- `SQLAppStatusListenerSuite` (covers `SparkListenerSQLExecutionStart`)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Qwen3.8 Max
Closes #58313 from pan3793/SPARK-59023.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit d8429a174ec699a7e9ddff1ee9b7c82b60900ba7)
Signed-off-by: Cheng Pan <[email protected]>
---
.../org/apache/spark/sql/internal/SQLConf.scala | 11 ++++-----
.../apache/spark/sql/execution/SQLExecution.scala | 26 +++++++++++++++++++---
.../execution/adaptive/AdaptiveSparkPlanExec.scala | 3 +--
.../adaptive/AdaptiveQueryExecSuite.scala | 3 ++-
.../execution/ui/SQLAppStatusListenerSuite.scala | 3 ++-
5 files changed, 34 insertions(+), 12 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index fbbad38a8a5d..a9d79df84497 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -6664,14 +6664,15 @@ object SQLConf {
.createWithDefault(false)
val UI_EXPLAIN_MODE = buildConf("spark.sql.ui.explainMode")
- .doc("Configures the query explain mode used in the Spark SQL UI. The
value can be 'simple', " +
- "'extended', 'codegen', 'cost', or 'formatted'. The default value is
'formatted'.")
+ .doc("Configures the query explain mode used in the Spark SQL UI. The
value can be 'none', " +
+ "'simple', 'extended', 'codegen', 'cost', or 'formatted'. The default
value is 'formatted'.")
.version("3.1.0")
.stringConf
.transform(_.toUpperCase(Locale.ROOT))
- .checkValue(mode => Set("SIMPLE", "EXTENDED", "CODEGEN", "COST",
"FORMATTED").contains(mode),
- "Invalid value for 'spark.sql.ui.explainMode'. Valid values are
'simple', 'extended', " +
- "'codegen', 'cost' and 'formatted'.")
+ .checkValue(mode =>
+ Set("NONE", "SIMPLE", "EXTENDED", "CODEGEN", "COST",
"FORMATTED").contains(mode),
+ "Invalid value for 'spark.sql.ui.explainMode'. Valid values are 'none',
'simple', " +
+ "'extended', 'codegen', 'cost' and 'formatted'.")
.createWithDefault("formatted")
val SOURCES_BINARY_FILE_MAX_LENGTH =
buildConf("spark.sql.sources.binaryFile.maxLength")
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
index f25e908a9cdb..e296e00ba48d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
@@ -120,6 +120,27 @@ object SQLExecution extends Logging {
}
}
+ private[sql] val NONE_EXPLAIN_MODE = "none"
+
+ private[sql] val NO_PLAN_DESCRIPTION =
+ s"No plan description because
${SQLConf.UI_EXPLAIN_MODE.key}=$NONE_EXPLAIN_MODE"
+
+ /**
+ * Returns the plan description carried by the SQL UI events, or a
placeholder when the UI
+ * explain mode is `none`, in which case the explain string, which is
expensive to build for
+ * large plans, is not generated at all.
+ *
+ * `none` is intentionally not an [[ExplainMode]]: it only makes sense for
the UI events, and
+ * keeping it here avoids making `df.explain("none")` a valid public API.
+ */
+ private[sql] def planDescription(qe: QueryExecution, uiExplainMode: String):
String = {
+ if (uiExplainMode.equalsIgnoreCase(NONE_EXPLAIN_MODE)) {
+ NO_PLAN_DESCRIPTION
+ } else {
+ qe.explainString(ExplainMode.fromString(uiExplainMode))
+ }
+ }
+
/**
* Wrap an action that will execute "queryExecution" to track all Spark jobs
in the body so that
* we can connect them with an execution.
@@ -209,9 +230,8 @@ object SQLExecution extends Logging {
sc.listenerBus.post(startEvent)
throw e
case Right(f) =>
- val planDescriptionMode =
-
ExplainMode.fromString(sparkSession.sessionState.conf.uiExplainMode)
- val planDesc =
queryExecution.explainString(planDescriptionMode)
+ val planDesc = planDescription(
+ queryExecution,
sparkSession.sessionState.conf.uiExplainMode)
val planInfo = try {
SparkPlanInfo.fromSparkPlan(queryExecution.executedPlan)
} catch {
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
index a12b202b5b00..0ef9b02d32d0 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
@@ -1145,10 +1145,9 @@ case class AdaptiveSparkPlanExec(
context.session.sparkContext.listenerBus.post(SparkListenerSQLAdaptiveSQLMetricUpdates(
executionId, newMetrics))
} else {
- val planDescriptionMode = ExplainMode.fromString(conf.uiExplainMode)
context.session.sparkContext.listenerBus.post(SparkListenerSQLAdaptiveExecutionUpdate(
executionId,
- context.qe.explainString(planDescriptionMode),
+ SQLExecution.planDescription(context.qe, conf.uiExplainMode),
SparkPlanInfo.fromSparkPlan(context.qe.executedPlan)))
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
index 8e98d7785f6c..3ac16235333c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
@@ -2541,7 +2541,8 @@ class AdaptiveQueryExecSuite
"== Optimized Logical Plan ==", "== Physical Plan ==")),
("codegen", Seq("WholeStageCodegen subtrees")),
("cost", Seq("== Optimized Logical Plan ==",
"Statistics(sizeInBytes")),
- ("formatted", Seq("== Physical Plan ==", "Output",
"Arguments"))).foreach {
+ ("formatted", Seq("== Physical Plan ==", "Output", "Arguments")),
+ ("none", Seq(SQLExecution.NO_PLAN_DESCRIPTION))).foreach {
case (mode, expected) =>
checkPlanDescription(mode, expected)
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
index 9230eb48dcca..7d9c79ab9236 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
@@ -369,7 +369,8 @@ abstract class SQLAppStatusListenerSuite extends
SharedSparkSession with JsonTes
"== Optimized Logical Plan ==", "== Physical Plan ==")),
("codegen", Seq("WholeStageCodegen subtrees")),
("cost", Seq("== Optimized Logical Plan ==",
"Statistics(sizeInBytes")),
- ("formatted", Seq("== Physical Plan ==", "Output",
"Arguments"))).foreach {
+ ("formatted", Seq("== Physical Plan ==", "Output", "Arguments")),
+ ("none", Seq(SQLExecution.NO_PLAN_DESCRIPTION))).foreach {
case (mode, expected) =>
checkPlanDescription(mode, expected)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
