[spark] branch master updated: [SPARK-37824][DOCS] Document K8s executor rolling configurations

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 476214b  [SPARK-37824][DOCS] Document K8s executor rolling 
configurations
476214b is described below

commit 476214bc1cc813f0a2332bee53dfc7248ebd2a66
Author: Dongjoon Hyun <dongj...@apache.org>
AuthorDate: Wed Jan 5 23:06:29 2022 -0800

    [SPARK-37824][DOCS] Document K8s executor rolling configurations
    
    ### What changes were proposed in this pull request?
    
    This PR aims to document newly added K8s executor rolling configurations.
    
    ### Why are the changes needed?
    
    To make doc up-to-date.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Manually review.
    
    <img width="1393" alt="Screen Shot 2022-01-05 at 10 04 19 PM" 
src="https://user-images.githubusercontent.com/9700541/148336572-125f2a78-5131-4a04-9fc9-f1457a381d4e.png";>
    
    Closes #35114 from dongjoon-hyun/SPARK-37824.
    
    Authored-by: Dongjoon Hyun <dongj...@apache.org>
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
 docs/running-on-kubernetes.md | 41 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index dbc0783..b869951 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1486,7 +1486,46 @@ See the [configuration page](configuration.html) for 
information on Spark config
   </td>
   <td>3.3.0</td>
 </tr>
-
+<tr>
+  <td><code>spark.kubernetes.executor.rollInterval</code></td>
+  <td><code>0s</code></td>
+  <td>
+    Interval between executor roll operations. It's disabled by default with 
`0s`.
+  </td>
+  <td>3.3.0</td>
+</tr>
+<tr>
+  
<td><code>spark.kubernetes.executor.minTasksPerExecutorBeforeRolling</code></td>
+  <td><code>0</code></td>
+  <td>
+    The minimum number of tasks per executor before rolling.
+    Spark will not roll executors whose total number of tasks is smaller
+    than this configuration. The default value is zero.
+  </td>
+  <td>3.3.0</td>
+</tr>
+<tr>
+  <td><code>spark.kubernetes.executor.rollPolicy</code></td>
+  <td><code>OUTLIER</code></td>
+  <td>
+    Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME, 
+    TOTAL_DURATION, FAILED_TASKS, and OUTLIER (default).
+    When executor roll happens, Spark uses this policy to choose
+    an executor and decommission it. The built-in policies are based on 
executor summary
+    and newly started executors are protected by 
spark.kubernetes.executor.minTasksPerExecutorBeforeRolling.
+    ID policy chooses an executor with the smallest executor ID.
+    ADD_TIME policy chooses an executor with the smallest add-time.
+    TOTAL_GC_TIME policy chooses an executor with the biggest total task GC 
time.
+    TOTAL_DURATION policy chooses an executor with the biggest total task time.
+    AVERAGE_DURATION policy chooses an executor with the biggest average task 
time.
+    FAILED_TASKS policy chooses an executor with the most number of failed 
tasks.
+    OUTLIER policy chooses an executor with outstanding statistics which is 
bigger than
+    at least two standard deviation from the mean in average task time,
+    total task time, total task GC time, and the number of failed tasks if 
exists.
+    If there is no outlier, it works like TOTAL_DURATION policy.
+  </td>
+  <td>3.3.0</td>
+</tr>
 </table>
 
 #### Pod template properties

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org