This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 39f1c3aaade7 [SPARK-59924][K8S][DOC] Fix stale "Valid values" list for
`spark.kubernetes.executor.rollPolicy`
39f1c3aaade7 is described below
commit 39f1c3aaade77ac9cfb7fcce4fba0ccb2aa1fff4
Author: Kousuke Saruta <[email protected]>
AuthorDate: Fri Oct 2 12:59:24 2026 -0700
[SPARK-59924][K8S][DOC] Fix stale "Valid values" list for
`spark.kubernetes.executor.rollPolicy`
### What changes were proposed in this pull request?
This PR fixes the "Valid values" list for
`spark.kubernetes.executor.rollPolicy`, which was out of sync with the
`ExecutorRollPolicy` enum in both `Config.scala` and `running-on-kubernetes.md`.
- `Config.scala`: added the missing `TOTAL_SHUFFLE_WRITE` and `DISK_USED`.
- `running-on-kubernetes.md`: added the missing `AVERAGE_DURATION`,
`PEAK_JVM_ONHEAP_MEMORY`, `PEAK_JVM_OFFHEAP_MEMORY`, `TOTAL_SHUFFLE_WRITE`,
`DISK_USED`, `ACTIVE_TASKS` and `OUTLIER_NO_FALLBACK`, including their
per-policy descriptions.
Both lists now follow the enum definition order.
### Why are the changes needed?
The documented list of valid values listed only a subset of the supported
policies, which is misleading.
### Does this PR introduce _any_ user-facing change?
No. Docs only. No behavior change.
### How was this patch tested?
Verified by inspection that the lists in both files match the
`ExecutorRollPolicy` enum, and that the `OUTLIER` dimensions, their checking
order, and the `DISK_USED` wording match `ExecutorRollPlugin` and
`AppStatusListener`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Kiro CLI / Claude
Closes #59187 from sarutak/rollpolicy-valid-values-docfix.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 25a9aa7861642a834af8e045de6416626b12bb15)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/running-on-kubernetes.md | 16 ++++++++++++++--
.../main/scala/org/apache/spark/deploy/k8s/Config.scala | 15 ++++++++++-----
2 files changed, 24 insertions(+), 7 deletions(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 322b299b38de..07bf8294a097 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -2100,7 +2100,9 @@ See the [configuration page](configuration.html) for
information on Spark config
<td><code>OUTLIER</code></td>
<td>
Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME,
- TOTAL_DURATION, FAILED_TASKS, and OUTLIER (default).
+ TOTAL_DURATION, AVERAGE_DURATION, FAILED_TASKS, PEAK_JVM_ONHEAP_MEMORY,
+ PEAK_JVM_OFFHEAP_MEMORY, TOTAL_SHUFFLE_WRITE, DISK_USED, ACTIVE_TASKS,
+ OUTLIER (default), and OUTLIER_NO_FALLBACK.
When executor roll happens, Spark uses this policy to choose
an executor and decommission it. The built-in policies are based on
executor summary
and newly started executors are protected by
spark.kubernetes.executor.minTasksPerExecutorBeforeRolling.
@@ -2110,14 +2112,24 @@ See the [configuration page](configuration.html) for
information on Spark config
TOTAL_DURATION policy chooses an executor with the biggest total task time.
AVERAGE_DURATION policy chooses an executor with the biggest average task
time.
FAILED_TASKS policy chooses an executor with the most number of failed
tasks.
+ PEAK_JVM_ONHEAP_MEMORY policy chooses an executor with the biggest peak
JVM on-heap memory.
+ PEAK_JVM_OFFHEAP_MEMORY policy chooses an executor with the biggest peak
JVM off-heap memory.
+ TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total
shuffle write.
+ DISK_USED policy chooses an executor with the biggest disk size used by its
+ stored blocks (e.g., disk-persisted RDD blocks).
ACTIVE_TASKS policy chooses an executor with the smallest number of active
tasks.
If there is a tie, it chooses an executor with the smallest add-time.
It is recommended to use it with
spark.kubernetes.executor.minTasksPerExecutorBeforeRolling
because newly started executors usually have no active tasks.
OUTLIER policy chooses an executor with outstanding statistics which is
bigger than
at least two standard deviation from the mean in average task time,
- total task time, total task GC time, and the number of failed tasks if
exists.
+ total task time, total task GC time, the number of failed tasks,
+ peak JVM on-heap memory, peak JVM off-heap memory, total shuffle write,
+ and disk used if exists.
+ The dimensions are checked in this order and the first outlier found is
chosen.
If there is no outlier, it works like TOTAL_DURATION policy.
+ OUTLIER_NO_FALLBACK policy picks an outlier using the OUTLIER policy above.
+ If there is no outlier then no executor will be rolled.
</td>
<td>3.3.0</td>
</tr>
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
index 705bc475d310..5162640de39a 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
@@ -247,9 +247,10 @@ private[spark] object Config extends Logging {
ConfigBuilder("spark.kubernetes.executor.rollPolicy")
.doc("Executor roll policy: Valid values are ID, ADD_TIME,
TOTAL_GC_TIME, " +
"TOTAL_DURATION, AVERAGE_DURATION, FAILED_TASKS,
PEAK_JVM_ONHEAP_MEMORY, " +
- "PEAK_JVM_OFFHEAP_MEMORY, ACTIVE_TASKS, OUTLIER (default), and
OUTLIER_NO_FALLBACK. " +
+ "PEAK_JVM_OFFHEAP_MEMORY, TOTAL_SHUFFLE_WRITE, DISK_USED,
ACTIVE_TASKS, " +
+ "OUTLIER (default), and OUTLIER_NO_FALLBACK. " +
"When executor roll happens, Spark uses this policy to choose " +
- "an executor and decommission it. The built-in policies are based on
executor summary." +
+ "an executor and decommission it. The built-in policies are based on
executor summary. " +
"ID policy chooses an executor with the smallest executor ID. " +
"ADD_TIME policy chooses an executor with the smallest add-time. " +
"TOTAL_GC_TIME policy chooses an executor with the biggest total task
GC time. " +
@@ -260,15 +261,19 @@ private[spark] object Config extends Logging {
"memory. PEAK_JVM_OFFHEAP_MEMORY policy chooses an executor with the
biggest peak JVM " +
"off-heap memory. " +
"TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total
shuffle write. " +
- "DISK_USED policy chooses an executor with the biggest used disk size.
" +
+ "DISK_USED policy chooses an executor with the biggest disk size used
by its " +
+ "stored blocks (e.g., disk-persisted RDD blocks). " +
"ACTIVE_TASKS policy chooses an executor with the smallest number of
active tasks. " +
"If there is a tie, it chooses an executor with the smallest add-time.
" +
"It is recommended to use it with " +
"spark.kubernetes.executor.minTasksPerExecutorBeforeRolling " +
"because newly started executors usually have no active tasks. " +
- "OUTLIER policy chooses an executor with outstanding statistics which
is bigger than" +
+ "OUTLIER policy chooses an executor with outstanding statistics which
is bigger than " +
"at least two standard deviation from the mean in average task time, "
+
- "total task time, total task GC time, and the number of failed tasks
if exists. " +
+ "total task time, total task GC time, the number of failed tasks, " +
+ "peak JVM on-heap memory, peak JVM off-heap memory, total shuffle
write, " +
+ "and disk used if exists. " +
+ "The dimensions are checked in this order and the first outlier found
is chosen. " +
"If there is no outlier it works like TOTAL_DURATION policy. " +
"OUTLIER_NO_FALLBACK policy picks an outlier using the OUTLIER policy
above. " +
"If there is no outlier then no executor will be rolled.")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
