GitHub user andrusha opened a pull request:
https://github.com/apache/spark/pull/21260
[SPARK-23529][K8s] Support mounting hostPath volumes
This PR continues #21095 and intersects with #21238. I've added volume
mounts as a separate step and added PersistantVolumeClaim support.
There is a fundamental problem with how we pass the options through spark
conf to fabric8. For each volume type and all possible volume options we would
have to implement some custom code to map config values to fabric8 calls. This
will result in big body of code we would have to support and means that Spark
will always be somehow out of sync with k8s.
I think there needs to be a discussion how to proceed correctly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrusha/spark k8s-vol
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21260.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21260
----
commit 51a877a7d86b6c0f18d381ba9300de18ccef9189
Author: madanadit <adit@...>
Date: 2018-04-05T01:02:25Z
Support mounting hostPath volumes for executors
commit 4ada724ab2f6e79553fa05475b216884f9ba127a
Author: madanadit <adit@...>
Date: 2018-04-05T20:57:45Z
Read mode for mounted volumes
commit 17258a3967b6d8b2170c3c7d7186449c3155fef1
Author: madanadit <adit@...>
Date: 2018-04-05T22:05:07Z
Refactor
commit 6fff716cbe42e184059660cf2009f594048a6420
Author: madanadit <adit@...>
Date: 2018-04-06T18:18:09Z
Fix style
commit f961b33aa34ad7908ac6d77ded251e18bf6e835a
Author: madanadit <adit@...>
Date: 2018-04-06T19:57:45Z
Add unit tests
commit af4d9baae8d637215d2be645b377fa5dbd714e94
Author: madanadit <adit@...>
Date: 2018-04-06T20:00:18Z
Update comment
commit 13e10493fa6f3b40c3eb406d0dc2f88d09ce20b8
Author: madanadit <adit@...>
Date: 2018-04-06T23:56:31Z
Fix unit tests
commit 7a25d76e386c0d8157eba0ccc0517bea3b7f7f0e
Author: madanadit <adit@...>
Date: 2018-04-16T22:41:26Z
Fix typo
commit a5a277a065dec156e976478de7767062bcf1da13
Author: madanadit <adit@...>
Date: 2018-04-17T21:51:58Z
Change configuration format
commit 772128e13895d6f313efa3cbc38947db51c3e493
Author: madanadit <adit@...>
Date: 2018-04-17T23:34:37Z
Fix build
commit a393b92c93d60bdcf96025ae24d6c1fbecf17f9b
Author: madanadit <adit@...>
Date: 2018-04-17T23:44:08Z
Fix test
commit b9b3dcb22ded628f0abe1caa0beb2b15da6ccc49
Author: madanadit <adit@...>
Date: 2018-04-18T00:39:14Z
Fetch properties correctly
commit 81a7811b06a0195e84ffeddc135875da6c500a7e
Author: madanadit <adit@...>
Date: 2018-04-18T05:21:11Z
Fix test cases
commit 95d1b0d8c681cd46f47c8ab1692172d0b3b0aba8
Author: madanadit <adit@...>
Date: 2018-04-18T06:42:56Z
Abstract tests
commit facde97b365a7acb02c41e5ef076a9ea0f1edff9
Author: madanadit <adit@...>
Date: 2018-04-18T06:59:43Z
Add readOnly option
commit ccdc7990ca8995ff86f46647f8a2949848f06380
Author: madanadit <adit@...>
Date: 2018-04-18T07:06:17Z
Fix test
commit 7c1be8aff51a462bf96012fafbbfec765424de53
Author: madanadit <adit@...>
Date: 2018-04-18T07:24:55Z
Driver hostPath volumes with tests
commit f482dfc370bace0c5315a2514604be2965bcbbaf
Author: madanadit <adit@...>
Date: 2018-04-19T23:56:58Z
Refactor
commit 136f5f65481f05cf10bd0c6968bf84c4363df53d
Author: Andrew Korzhuev <andrew.korzhuev@...>
Date: 2018-05-07T17:03:05Z
Address comments, rewrite volume mount to step
commit f1ada4c402a7f5ca5fd5428a28a457cd2400c6f2
Author: Andrew Korzhuev <andrew.korzhuev@...>
Date: 2018-05-07T17:08:12Z
Add persistant volume claim
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
