Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/22323#discussion_r215799370
--- Diff: docs/running-on-kubernetes.md ---
@@ -215,6 +215,19 @@
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.clai
The configuration properties for mounting volumes into the executor pods
use prefix `spark.kubernetes.executor.` instead of `spark.kubernetes.driver.`.
For a complete list of available options for each supported type of volumes,
please refer to the [Spark Properties](#spark-properties) section below.
+## Local Storage
+
+Spark uses temporary scratch space to spill data to disk during shuffles
and other operations. When using Kubernetes as the resource manager the pods
will be created with an
[emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir)
volume mounted for each directory listed in `SPARK_LOCAL_DIRS`. If no
directories are explicitly specified then a default directory is created and
configured appropriately.
--- End diff --
Looking at this again, I'm wondering if we're getting too detailed in our
documentation about all the features we support. Particularly with the pod
template files pull request coming in adding more documentation, and it's
possible that we have an information overload on this particular page.
Would it be reasonable to only include this information in the
configuration options below, under the `spark.kubernetes.local.dirs.tmpfs`
setting?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
