GitHub user mccheah opened a pull request:
https://github.com/apache/spark/pull/21238
[SPARK-24137] Mount local directories as empty dir volumes.
## What changes were proposed in this pull request?
Dramatically improves performance and won't cause Spark applications to
fail because they write too much data to the Docker image's specific file
system. The file system's directories that back emptydir volumes are generally
larger and more performant.
## How was this patch tested?
Has been in use via the prototype version of Kubernetes support, but lost
in the transition to here.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/palantir/spark mount-local-dirs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21238.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21238
----
commit 736674bd2baef34e8d55fb7a7cd684e9f467f4ea
Author: mcheah <mcheah@...>
Date: 2018-05-01T20:59:29Z
[SPARK-24137] Mount local directories as empty dir volumes.
Dramatically improves performance and won't cause Spark applications to
fail because they write too much data to the Docker image's specific
file system. The file system's directories that back emptydir volumes
are generally larger and more performant.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
