Github user foxish commented on a diff in the pull request:
https://github.com/apache/spark/pull/20059#discussion_r158564775
--- Diff: docs/running-on-kubernetes.md ---
@@ -528,51 +545,94 @@ specific to Spark on Kubernetes.
</td>
</tr>
<tr>
- <td><code>spark.kubernetes.driver.limit.cores</code></td>
- <td>(none)</td>
- <td>
- Specify the hard CPU
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
for the driver pod.
- </td>
- </tr>
- <tr>
- <td><code>spark.kubernetes.executor.limit.cores</code></td>
- <td>(none)</td>
- <td>
- Specify the hard CPU
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
for each executor pod launched for the Spark Application.
- </td>
- </tr>
- <tr>
- <td><code>spark.kubernetes.node.selector.[labelKey]</code></td>
- <td>(none)</td>
- <td>
- Adds to the node selector of the driver pod and executor pods, with
key <code>labelKey</code> and the value as the
- configuration's value. For example, setting
<code>spark.kubernetes.node.selector.identifier</code> to
<code>myIdentifier</code>
- will result in the driver pod and executors having a node selector
with key <code>identifier</code> and value
- <code>myIdentifier</code>. Multiple node selector keys can be added
by setting multiple configurations with this prefix.
- </td>
- </tr>
- <tr>
-
<td><code>spark.kubernetes.driverEnv.[EnvironmentVariableName]</code></td>
- <td>(none)</td>
- <td>
- Add the environment variable specified by
<code>EnvironmentVariableName</code> to
- the Driver process. The user can specify multiple of these to set
multiple environment variables.
- </td>
- </tr>
- <tr>
-
<td><code>spark.kubernetes.mountDependencies.jarsDownloadDir</code></td>
- <td><code>/var/spark-data/spark-jars</code></td>
- <td>
- Location to download jars to in the driver and executors.
- This directory must be empty and will be mounted as an empty
directory volume on the driver and executor pods.
- </td>
- </tr>
- <tr>
-
<td><code>spark.kubernetes.mountDependencies.filesDownloadDir</code></td>
- <td><code>/var/spark-data/spark-files</code></td>
- <td>
- Location to download jars to in the driver and executors.
- This directory must be empty and will be mounted as an empty
directory volume on the driver and executor pods.
- </td>
- </tr>
+ <td><code>spark.kubernetes.driver.limit.cores</code></td>
+ <td>(none)</td>
+ <td>
+ Specify the hard CPU
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
for the driver pod.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.executor.limit.cores</code></td>
+ <td>(none)</td>
+ <td>
+ Specify the hard CPU
[limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container)
for each executor pod launched for the Spark Application.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.node.selector.[labelKey]</code></td>
+ <td>(none)</td>
+ <td>
+ Adds to the node selector of the driver pod and executor pods, with
key <code>labelKey</code> and the value as the
+ configuration's value. For example, setting
<code>spark.kubernetes.node.selector.identifier</code> to
<code>myIdentifier</code>
+ will result in the driver pod and executors having a node selector
with key <code>identifier</code> and value
+ <code>myIdentifier</code>. Multiple node selector keys can be added
by setting multiple configurations with this prefix.
+ </td>
+</tr>
+<tr>
+
<td><code>spark.kubernetes.driverEnv.[EnvironmentVariableName]</code></td>
+ <td>(none)</td>
+ <td>
+ Add the environment variable specified by
<code>EnvironmentVariableName</code> to
+ the Driver process. The user can specify multiple of these to set
multiple environment variables.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.mountDependencies.jarsDownloadDir</code></td>
+ <td><code>/var/spark-data/spark-jars</code></td>
+ <td>
+ Location to download jars to in the driver and executors.
+ This directory must be empty and will be mounted as an empty directory
volume on the driver and executor pods.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.mountDependencies.filesDownloadDir</code></td>
+ <td><code>/var/spark-data/spark-files</code></td>
+ <td>
+ Location to download jars to in the driver and executors.
+ This directory must be empty and will be mounted as an empty directory
volume on the driver and executor pods.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.mountDependencies.mountTimeout</code></td>
+ <td>5 minutes</td>
+ <td>
+ Timeout before aborting the attempt to download and unpack dependencies
from remote locations when initializing
+ the driver and executor pods.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.initContainer.image</code></td>
+ <td>(none)</td>
+ <td>
+ Container image for the init-container of the driver and executors for
downloading dependencies.
+ This is usually of the form
<code>example.com/repo/spark-init:v1.0.0</code>.
+ This configuration is optional and must be provided by the user if any
non-container local dependency is used and
+ must be downloaded remotely.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.initContainer.maxThreadPoolSize</code></td>
--- End diff --
I feel like this option name needs fixing. Maybe
`spark.kubernetes.mountDependencies.maxThreadPoolSize`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
