Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/22323#discussion_r215806032
--- Diff: docs/running-on-kubernetes.md ---
@@ -215,6 +215,19 @@
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.clai
The configuration properties for mounting volumes into the executor pods
use prefix `spark.kubernetes.executor.` instead of `spark.kubernetes.driver.`.
For a complete list of available options for each supported type of volumes,
please refer to the [Spark Properties](#spark-properties) section below.
+## Local Storage
+
+Spark uses temporary scratch space to spill data to disk during shuffles
and other operations. When using Kubernetes as the resource manager the pods
will be created with an
[emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir)
volume mounted for each directory listed in `SPARK_LOCAL_DIRS`. If no
directories are explicitly specified then a default directory is created and
configured appropriately.
--- End diff --
Don't really want to block on this though so we can merge regardless I
think. This might be something to consider for the next release, to go back
over this document as a whole and clean it up if we think it's too verbose.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
