GitHub user madanadit opened a pull request:

    [SPARK-23529][K8s] Support mounting hostPath volumes for executors

    ## What changes were proposed in this pull request?
    This PR introduces a new config `spark.kubernetes.executor.volumes` taking 
a values of the form  `hostPath:containerPath[:ro|rw][,...]`; where  `hostPath` 
is the path for the executor pod volume, `containerPath` is the mount path and 
`ro` is read-only mode.
    The use case is to enable short-circuit writes to distributed storage on 
k8s. The Alluxio File System uses domain sockets to enable short-circuit writes 
from the client to worker memory when co-located on the same host machine. A 
directory, lets say /tmp/domain on the host, is mounted on the Alluxio worker 
container as well as the Alluxio client ( = Spark executor) container. The 
worker creates a domain socket /tmp/domain/d and if the client container mounts 
the same directory, it can write directly to the Alluxio worker w/o passing 
through network stack. The end result is faster data access when data is local.
    ## How was this patch tested?
    Manual testing on a k8s v1.8 cluster. Unit tests added to 

You can merge this pull request into a Git repository by running:

    $ git pull k8s-vols

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21032
commit a7edf6cf71e66caaec117335bb569f9ab93967f7
Author: madanadit <adit@...>
Date:   2018-04-05T01:02:25Z

    Support mounting hostPath volumes for executors

commit 52f0d1cdcf125b3ba327a40360481eb4568991e2
Author: madanadit <adit@...>
Date:   2018-04-05T20:57:45Z

    Read mode for mounted volumes

commit 70d050ea922d51f40384b7e473fe15dcfb5f4447
Author: madanadit <adit@...>
Date:   2018-04-05T22:05:07Z


commit 463e9b5c227a42c4ae02f6e25475606ed66b50df
Author: madanadit <adit@...>
Date:   2018-04-06T18:18:09Z

    Fix style

commit 0a21a19aca92a9409df0c03e24a8aa99d4dcd5b6
Author: madanadit <adit@...>
Date:   2018-04-06T19:57:45Z

    Add unit tests

commit 87bcc9f638cc0720e00de86f46a3e677a8fc54c8
Author: madanadit <adit@...>
Date:   2018-04-06T20:00:18Z

    Update comment

commit c72cbc1bd8a5043f5dbc55fa7ba0631c694a9317
Author: madanadit <adit@...>
Date:   2018-04-06T23:56:31Z

    Fix unit tests



To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to