This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 6c9f76d234d6 [SPARK-59924][4.3][K8S][DOC] Fix stale "Valid values"
list for `spark.kubernetes.executor.rollPolicy`
6c9f76d234d6 is described below
commit 6c9f76d234d699242cfdec7cf6a1282d670e0122
Author: Kousuke Saruta <[email protected]>
AuthorDate: Tue Oct 6 11:14:46 2026 -0700
[SPARK-59924][4.3][K8S][DOC] Fix stale "Valid values" list for
`spark.kubernetes.executor.rollPolicy`
### What changes were proposed in this pull request?
This backports #59187 (commit 25a9aa7) to branch-4.3.
This PR fixes the "Valid values" list for
`spark.kubernetes.executor.rollPolicy`, which was out of sync with the
`ExecutorRollPolicy` enum in both `Config.scala` and `running-on-kubernetes.md`.
- `Config.scala`: added the missing `TOTAL_SHUFFLE_WRITE` and `DISK_USED`.
- `running-on-kubernetes.md`: added the missing `AVERAGE_DURATION`,
`PEAK_JVM_ONHEAP_MEMORY`, `PEAK_JVM_OFFHEAP_MEMORY`, `TOTAL_SHUFFLE_WRITE`,
`DISK_USED` and `OUTLIER_NO_FALLBACK`, including their per-policy descriptions.
Both lists now follow the enum definition order.
### Why are the changes needed?
The documented list of valid values listed only a subset of the supported
policies, which is misleading.
### Does this PR introduce _any_ user-facing change?
No. Docs only. No behavior change.
### How was this patch tested?
Verified by inspection that the lists in both files match the branch-4.3
`ExecutorRollPolicy` enum, and that the `OUTLIER` dimensions, their checking
order, and the `DISK_USED` wording match `ExecutorRollPlugin` and
`AppStatusListener`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Kiro CLI / Claude
Closes #59253 from sarutak/SPARK-59924-branch-4.3.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/running-on-kubernetes.md | 16 ++++++++++++++--
.../main/scala/org/apache/spark/deploy/k8s/Config.scala | 15 ++++++++++-----
2 files changed, 24 insertions(+), 7 deletions(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index a306fdb1d545..98702a788b3a 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1958,7 +1958,9 @@ See the [configuration page](configuration.html) for
information on Spark config
<td><code>OUTLIER</code></td>
<td>
Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME,
- TOTAL_DURATION, FAILED_TASKS, and OUTLIER (default).
+ TOTAL_DURATION, AVERAGE_DURATION, FAILED_TASKS, PEAK_JVM_ONHEAP_MEMORY,
+ PEAK_JVM_OFFHEAP_MEMORY, TOTAL_SHUFFLE_WRITE, DISK_USED,
+ OUTLIER (default), and OUTLIER_NO_FALLBACK.
When executor roll happens, Spark uses this policy to choose
an executor and decommission it. The built-in policies are based on
executor summary
and newly started executors are protected by
spark.kubernetes.executor.minTasksPerExecutorBeforeRolling.
@@ -1968,10 +1970,20 @@ See the [configuration page](configuration.html) for
information on Spark config
TOTAL_DURATION policy chooses an executor with the biggest total task time.
AVERAGE_DURATION policy chooses an executor with the biggest average task
time.
FAILED_TASKS policy chooses an executor with the most number of failed
tasks.
+ PEAK_JVM_ONHEAP_MEMORY policy chooses an executor with the biggest peak
JVM on-heap memory.
+ PEAK_JVM_OFFHEAP_MEMORY policy chooses an executor with the biggest peak
JVM off-heap memory.
+ TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total
shuffle write.
+ DISK_USED policy chooses an executor with the biggest disk size used by its
+ stored blocks (e.g., disk-persisted RDD blocks).
OUTLIER policy chooses an executor with outstanding statistics which is
bigger than
at least two standard deviation from the mean in average task time,
- total task time, total task GC time, and the number of failed tasks if
exists.
+ total task time, total task GC time, the number of failed tasks,
+ peak JVM on-heap memory, peak JVM off-heap memory, total shuffle write,
+ and disk used if exists.
+ The dimensions are checked in this order and the first outlier found is
chosen.
If there is no outlier, it works like TOTAL_DURATION policy.
+ OUTLIER_NO_FALLBACK policy picks an outlier using the OUTLIER policy above.
+ If there is no outlier then no executor will be rolled.
</td>
<td>3.3.0</td>
</tr>
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
index cdc82ea4ac93..f4bc9cc1b4af 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
@@ -246,9 +246,10 @@ private[spark] object Config extends Logging {
ConfigBuilder("spark.kubernetes.executor.rollPolicy")
.doc("Executor roll policy: Valid values are ID, ADD_TIME,
TOTAL_GC_TIME, " +
"TOTAL_DURATION, AVERAGE_DURATION, FAILED_TASKS,
PEAK_JVM_ONHEAP_MEMORY, " +
- "PEAK_JVM_OFFHEAP_MEMORY, OUTLIER (default), and OUTLIER_NO_FALLBACK.
" +
+ "PEAK_JVM_OFFHEAP_MEMORY, TOTAL_SHUFFLE_WRITE, DISK_USED, " +
+ "OUTLIER (default), and OUTLIER_NO_FALLBACK. " +
"When executor roll happens, Spark uses this policy to choose " +
- "an executor and decommission it. The built-in policies are based on
executor summary." +
+ "an executor and decommission it. The built-in policies are based on
executor summary. " +
"ID policy chooses an executor with the smallest executor ID. " +
"ADD_TIME policy chooses an executor with the smallest add-time. " +
"TOTAL_GC_TIME policy chooses an executor with the biggest total task
GC time. " +
@@ -259,10 +260,14 @@ private[spark] object Config extends Logging {
"memory. PEAK_JVM_OFFHEAP_MEMORY policy chooses an executor with the
biggest peak JVM " +
"off-heap memory. " +
"TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total
shuffle write. " +
- "DISK_USED policy chooses an executor with the biggest used disk size.
" +
- "OUTLIER policy chooses an executor with outstanding statistics which
is bigger than" +
+ "DISK_USED policy chooses an executor with the biggest disk size used
by its " +
+ "stored blocks (e.g., disk-persisted RDD blocks). " +
+ "OUTLIER policy chooses an executor with outstanding statistics which
is bigger than " +
"at least two standard deviation from the mean in average task time, "
+
- "total task time, total task GC time, and the number of failed tasks
if exists. " +
+ "total task time, total task GC time, the number of failed tasks, " +
+ "peak JVM on-heap memory, peak JVM off-heap memory, total shuffle
write, " +
+ "and disk used if exists. " +
+ "The dimensions are checked in this order and the first outlier found
is chosen. " +
"If there is no outlier it works like TOTAL_DURATION policy. " +
"OUTLIER_NO_FALLBACK policy picks an outlier using the OUTLIER policy
above. " +
"If there is no outlier then no executor will be rolled.")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
