This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 476214b [SPARK-37824][DOCS] Document K8s executor rolling
configurations
476214b is described below
commit 476214bc1cc813f0a2332bee53dfc7248ebd2a66
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Jan 5 23:06:29 2022 -0800
[SPARK-37824][DOCS] Document K8s executor rolling configurations
### What changes were proposed in this pull request?
This PR aims to document newly added K8s executor rolling configurations.
### Why are the changes needed?
To make doc up-to-date.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually review.
<img width="1393" alt="Screen Shot 2022-01-05 at 10 04 19 PM"
src="https://user-images.githubusercontent.com/9700541/148336572-125f2a78-5131-4a04-9fc9-f1457a381d4e.png";>
Closes #35114 from dongjoon-hyun/SPARK-37824.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/running-on-kubernetes.md | 41 ++++++++++++++++++++++++++++++++++++++++-
1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index dbc0783..b869951 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1486,7 +1486,46 @@ See the [configuration page](configuration.html) for
information on Spark config
</td>
<td>3.3.0</td>
</tr>
-
+<tr>
+ <td><code>spark.kubernetes.executor.rollInterval</code></td>
+ <td><code>0s</code></td>
+ <td>
+ Interval between executor roll operations. It's disabled by default with
`0s`.
+ </td>
+ <td>3.3.0</td>
+</tr>
+<tr>
+
<td><code>spark.kubernetes.executor.minTasksPerExecutorBeforeRolling</code></td>
+ <td><code>0</code></td>
+ <td>
+ The minimum number of tasks per executor before rolling.
+ Spark will not roll executors whose total number of tasks is smaller
+ than this configuration. The default value is zero.
+ </td>
+ <td>3.3.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.executor.rollPolicy</code></td>
+ <td><code>OUTLIER</code></td>
+ <td>
+ Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME,
+ TOTAL_DURATION, FAILED_TASKS, and OUTLIER (default).
+ When executor roll happens, Spark uses this policy to choose
+ an executor and decommission it. The built-in policies are based on
executor summary
+ and newly started executors are protected by
spark.kubernetes.executor.minTasksPerExecutorBeforeRolling.
+ ID policy chooses an executor with the smallest executor ID.
+ ADD_TIME policy chooses an executor with the smallest add-time.
+ TOTAL_GC_TIME policy chooses an executor with the biggest total task GC
time.
+ TOTAL_DURATION policy chooses an executor with the biggest total task time.
+ AVERAGE_DURATION policy chooses an executor with the biggest average task
time.
+ FAILED_TASKS policy chooses an executor with the most number of failed
tasks.
+ OUTLIER policy chooses an executor with outstanding statistics which is
bigger than
+ at least two standard deviation from the mean in average task time,
+ total task time, total task GC time, and the number of failed tasks if
exists.
+ If there is no outlier, it works like TOTAL_DURATION policy.
+ </td>
+ <td>3.3.0</td>
+</tr>
</table>
#### Pod template properties
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/9700541/148336572-125f2a78-5131-4a04-9fc9-f1457a381d4e.png"](https://user-images.githubusercontent.com/9700541/148336572-125f2a78-5131-4a04-9fc9-f1457a381d4e.png"){kind=link}