This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new dda69d2c52fa [SPARK-59936][DOC] Use short plugin class names in
`spark.plugins` examples and config docs
dda69d2c52fa is described below
commit dda69d2c52fa2954382285280097a2cdc4b5cc38
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 1 17:15:57 2026 -0700
[SPARK-59936][DOC] Use short plugin class names in `spark.plugins` examples
and config docs
### What changes were proposed in this pull request?
Use the short class names of built-in plugins (e.g. `DriverTimeoutPlugin`,
`ExecutorResizePlugin`) instead of fully-qualified names wherever
`spark.plugins` is set in `docs/configuration.md`,
`docs/running-on-kubernetes.md`, `connector/profiler/README.md`, and the
matching config doc strings in `package.scala`. The fully-qualified name is
kept at the first mention of each resize plugin in
`docs/running-on-kubernetes.md`.
### Why are the changes needed?
SPARK-59412 made `spark.plugins` accept the short names of built-in
plugins, but the documentation still shows the fully-qualified names.
- https://github.com/apache/spark/pull/58711
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change.
### How was this patch tested?
Not applicable. Only documentation and config doc strings are changed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #59193 from dongjoon-hyun/SPARK-59936.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 5f65d968c0b804819e75bdf1851a2f2b185ba43c)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
connector/profiler/README.md | 4 ++--
.../org/apache/spark/internal/config/package.scala | 6 +++---
docs/configuration.md | 7 +++----
docs/running-on-kubernetes.md | 22 +++++++++++-----------
4 files changed, 19 insertions(+), 20 deletions(-)
diff --git a/connector/profiler/README.md b/connector/profiler/README.md
index 9e74e589220e..ff6fef20bcbd 100644
--- a/connector/profiler/README.md
+++ b/connector/profiler/README.md
@@ -39,7 +39,7 @@ For more information on async_profiler see the [Async
Profiler Manual](https://k
To enable code profiling, first enable the code profiling plugin via
```
-spark.plugins=org.apache.spark.profiler.ProfilerPlugin
+spark.plugins=ProfilerPlugin
```
Then enable the profiling in the configuration.
@@ -150,7 +150,7 @@ On Kubernetes, spark will try to shut down the executor
pods while the profiler
--master <master-url> \
--deploy-mode <deploy-mode> \
-c spark.executor.extraJavaOptions="-XX:+UnlockDiagnosticVMOptions
-XX:+DebugNonSafepoints -XX:+PreserveFramePointer" \
- -c spark.plugins=org.apache.spark.profiler.ProfilerPlugin \
+ -c spark.plugins=ProfilerPlugin \
-c spark.profiler.executor.enabled=true \
-c spark.profiler.executor.fraction=0.10 \
-c spark.profiler.dfsDir=s3a://my-bucket/spark/profiles/ \
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 1a0c583d9b53..0d57cfaf9452 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -1251,7 +1251,7 @@ package object config {
private[spark] val DRIVER_TIMEOUT = ConfigBuilder("spark.driver.timeout")
.doc("A timeout for Spark driver in minutes. 0 means infinite. For the
positive time value, " +
"terminate the driver with the exit code 124 if it runs after timeout
duration. To use, " +
- "it's required to set
`spark.plugins=org.apache.spark.deploy.DriverTimeoutPlugin`.")
+ "it's required to set `spark.plugins=DriverTimeoutPlugin`.")
.version("4.0.0")
.timeConf(TimeUnit.MINUTES)
.checkValue(v => v >= 0, "The value should be a non-negative time value.")
@@ -3325,7 +3325,7 @@ package object config {
ConfigBuilder("spark.driver.log.redirectConsoleOutputs")
.doc("Comma-separated list of the console output kind for driver that
needs to redirect " +
"to logging system. Supported values are `stdout`, `stderr`. It only
takes affect when " +
- s"`${PLUGINS.key}` is configured with
`org.apache.spark.deploy.RedirectConsolePlugin`.")
+ s"`${PLUGINS.key}` is configured with `RedirectConsolePlugin`.")
.version("4.1.0")
.stringConf
.transform(_.toLowerCase(Locale.ROOT))
@@ -3338,7 +3338,7 @@ package object config {
ConfigBuilder("spark.executor.logs.redirectConsoleOutputs")
.doc("Comma-separated list of the console output kind for executor that
needs to redirect " +
"to logging system. Supported values are `stdout`, `stderr`. It only
takes affect when " +
- s"`${PLUGINS.key}` is configured with
`org.apache.spark.deploy.RedirectConsolePlugin`.")
+ s"`${PLUGINS.key}` is configured with `RedirectConsolePlugin`.")
.version("4.1.0")
.stringConf
.transform(_.toLowerCase(Locale.ROOT))
diff --git a/docs/configuration.md b/docs/configuration.md
index c85e1f58f1aa..1dc12a9e5791 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -464,8 +464,7 @@ of the most common options to set are:
<td>
A timeout for Spark driver in minutes. 0 means infinite. For the positive
time value,
terminate the driver with the exit code 124 if it runs after timeout
duration. To use,
- it's required to set <code>spark.plugins</code> with
- <code>org.apache.spark.deploy.DriverTimeoutPlugin</code>.
+ it's required to set <code>spark.plugins</code> with
<code>DriverTimeoutPlugin</code>.
</td>
<td>4.0.0</td>
</tr>
@@ -530,7 +529,7 @@ of the most common options to set are:
<td>
Comma-separated list of the console output kind for driver that needs to
redirect
to logging system. Supported values are `stdout`, `stderr`. It only takes
affect when
- `spark.plugins` is configured with
`org.apache.spark.deploy.RedirectConsolePlugin`.
+ `spark.plugins` is configured with `RedirectConsolePlugin`.
</td>
<td>4.1.0</td>
</tr>
@@ -789,7 +788,7 @@ Apart from these, the following properties are also
available, and may be useful
<td>
Comma-separated list of the console output kind for executor that needs to
redirect
to logging system. Supported values are `stdout`, `stderr`. It only takes
affect when
- `spark.plugins` is configured with
`org.apache.spark.deploy.RedirectConsolePlugin`.
+ `spark.plugins` is configured with `RedirectConsolePlugin`.
</td>
<td>4.1.0</td>
</tr>
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index ace9686ba876..322b299b38de 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -594,7 +594,7 @@ request are increased by
`spark.kubernetes.executor.resizeFactor`, up to
the container is not restarted.
```
---conf
spark.plugins=org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin
+--conf spark.plugins=ExecutorResizePlugin
--conf spark.kubernetes.executor.resizeInterval=1m
```
@@ -625,7 +625,7 @@ the filesystem usage of its local directories and reports
the highest usage rati
up to `spark.kubernetes.executor.pvc.resizeMaxStorage`.
```
---conf
spark.plugins=org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin
+--conf spark.plugins=ExecutorPVCResizePlugin
--conf
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName=OnDemand
--conf
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass=gp3
--conf
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit=100Gi
@@ -684,7 +684,7 @@ the replacement executors behave rather than the resources
of the existing ones.
registered together:
```
---conf
spark.plugins=org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin,org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin
+--conf spark.plugins=ExecutorResizePlugin,ExecutorPVCResizePlugin
```
All of these features operate at the level of the driver, not of an individual
job or session. In a
@@ -2126,7 +2126,7 @@ See the [configuration page](configuration.html) for
information on Spark config
<td><code>1min</code></td>
<td>
Interval between executor resize operations. To disable, set 0.
- Takes effect only when
<code>org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin</code>
+ Takes effect only when <code>ExecutorResizePlugin</code>
is registered via <code>spark.plugins</code>.
</td>
<td>4.2.0</td>
@@ -2136,7 +2136,7 @@ See the [configuration page](configuration.html) for
information on Spark config
<td><code>0.9</code></td>
<td>
The threshold to resize. It should be in (0, 1).
- Takes effect only when
<code>org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin</code>
+ Takes effect only when <code>ExecutorResizePlugin</code>
is registered via <code>spark.plugins</code>.
</td>
<td>4.2.0</td>
@@ -2146,7 +2146,7 @@ See the [configuration page](configuration.html) for
information on Spark config
<td><code>0.1</code></td>
<td>
The factor to resize. It should be in (0, 1].
- Takes effect only when
<code>org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin</code>
+ Takes effect only when <code>ExecutorResizePlugin</code>
is registered via <code>spark.plugins</code>.
</td>
<td>4.2.0</td>
@@ -2157,7 +2157,7 @@ See the [configuration page](configuration.html) for
information on Spark config
<td>
The upper bound of the executor container memory limit that the resize
plugin can grow to.
By default, it is <code>Long.MaxValue</code>, which means no upper bound.
- Takes effect only when
<code>org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin</code>
+ Takes effect only when <code>ExecutorResizePlugin</code>
is registered via <code>spark.plugins</code>.
</td>
<td>4.4.0</td>
@@ -2168,7 +2168,7 @@ See the [configuration page](configuration.html) for
information on Spark config
<td>
Interval between executor PVC resize operations, in minutes. Defaults to 5
minutes.
Set to 0 to disable. Must be 0 or a positive multiple of 5 minutes.
- Takes effect only when
<code>org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin</code>
+ Takes effect only when <code>ExecutorPVCResizePlugin</code>
is registered via <code>spark.plugins</code>.
</td>
<td>4.2.0</td>
@@ -2178,7 +2178,7 @@ See the [configuration page](configuration.html) for
information on Spark config
<td><code>0.5</code></td>
<td>
The PVC usage ratio (used / capacity) above which the driver triggers a
resize.
- Takes effect only when
<code>org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin</code>
+ Takes effect only when <code>ExecutorPVCResizePlugin</code>
is registered via <code>spark.plugins</code>.
</td>
<td>4.2.0</td>
@@ -2188,7 +2188,7 @@ See the [configuration page](configuration.html) for
information on Spark config
<td><code>1.0</code></td>
<td>
The factor to grow PVC storage by, relative to the current request.
- Takes effect only when
<code>org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin</code>
+ Takes effect only when <code>ExecutorPVCResizePlugin</code>
is registered via <code>spark.plugins</code>.
</td>
<td>4.2.0</td>
@@ -2199,7 +2199,7 @@ See the [configuration page](configuration.html) for
information on Spark config
<td>
The upper bound of the PVC storage request that the resize plugin can grow
to.
By default, it is <code>Long.MaxValue</code>, which means no upper bound.
- Takes effect only when
<code>org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin</code>
+ Takes effect only when <code>ExecutorPVCResizePlugin</code>
is registered via <code>spark.plugins</code>.
</td>
<td>4.4.0</td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
