[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...
Github user madanadit closed the pull request at: https://github.com/apache/spark/pull/21032 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...
Github user madanadit commented on a diff in the pull request: https://github.com/apache/spark/pull/21032#discussion_r181197979 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -117,6 +117,13 @@ private[spark] object Config extends Logging { .stringConf .createWithDefault("spark") + val KUBERNETES_EXECUTOR_VOLUMES = --- End diff -- Yes. I'll add the driver option after I rebase on [#20910](https://github.com/apache/spark/pull/20910) then. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21032#discussion_r181176057 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -117,6 +117,13 @@ private[spark] object Config extends Logging { .stringConf .createWithDefault("spark") + val KUBERNETES_EXECUTOR_VOLUMES = --- End diff -- Also, do we need a symmetric option for the drivers also? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21032#discussion_r181172885 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -117,6 +117,13 @@ private[spark] object Config extends Logging { .stringConf .createWithDefault("spark") + val KUBERNETES_EXECUTOR_VOLUMES = --- End diff -- If we are adding this option, it should be a more general API - allowing PVs, hostpath volumes and emptyDir volumes to be mounted. In the near future, hostpath volumes will be superseded by local PVs - https://github.com/kubernetes/kubernetes/issues/7562 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...
GitHub user madanadit opened a pull request: https://github.com/apache/spark/pull/21032 [SPARK-23529][K8s] Support mounting hostPath volumes for executors ## What changes were proposed in this pull request? This PR introduces a new config `spark.kubernetes.executor.volumes` taking a values of the form `hostPath:containerPath[:ro|rw][,...]`; where `hostPath` is the path for the executor pod volume, `containerPath` is the mount path and `ro` is read-only mode. The use case is to enable short-circuit writes to distributed storage on k8s. The Alluxio File System uses domain sockets to enable short-circuit writes from the client to worker memory when co-located on the same host machine. A directory, lets say /tmp/domain on the host, is mounted on the Alluxio worker container as well as the Alluxio client ( = Spark executor) container. The worker creates a domain socket /tmp/domain/d and if the client container mounts the same directory, it can write directly to the Alluxio worker w/o passing through network stack. The end result is faster data access when data is local. ## How was this patch tested? Manual testing on a k8s v1.8 cluster. Unit tests added to `ExecutorPodFactorySuite`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/madanadit/spark k8s-vols Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21032.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21032 commit a7edf6cf71e66caaec117335bb569f9ab93967f7 Author: madanadit Date: 2018-04-05T01:02:25Z Support mounting hostPath volumes for executors commit 52f0d1cdcf125b3ba327a40360481eb4568991e2 Author: madanadit Date: 2018-04-05T20:57:45Z Read mode for mounted volumes commit 70d050ea922d51f40384b7e473fe15dcfb5f4447 Author: madanadit Date: 2018-04-05T22:05:07Z Refactor commit 463e9b5c227a42c4ae02f6e25475606ed66b50df Author: madanadit Date: 2018-04-06T18:18:09Z Fix style commit 0a21a19aca92a9409df0c03e24a8aa99d4dcd5b6 Author: madanadit Date: 2018-04-06T19:57:45Z Add unit tests commit 87bcc9f638cc0720e00de86f46a3e677a8fc54c8 Author: madanadit Date: 2018-04-06T20:00:18Z Update comment commit c72cbc1bd8a5043f5dbc55fa7ba0631c694a9317 Author: madanadit Date: 2018-04-06T23:56:31Z Fix unit tests --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org