[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21260 @er0sin PVCs can be mounted similarly to the example below: ``` spark.kubernetes.driver.volumes.persistentVolumeClaim.pv1.mount.path=/mnt spark.kubernetes.driver.volumes.persistentVolumeClaim.pv1.mount.readOnly=false spark.kubernetes.driver.volumes.persistentVolumeClaim.pv1.options.claimName=clm-1 ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user er0sin commented on the issue: https://github.com/apache/spark/pull/21260 How does one configure a PV/PVC with this change? `spark.kubernetes.executor.volumes=pvName:containerPath` ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user andrusha commented on the issue: https://github.com/apache/spark/pull/21260 @liyinan926 sounds fair, it also pending tests, I'll add those in todo list for this PR and ping you once its done --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21260 @andrusha Can you remove coverage of https://github.com/apache/spark/pull/21238 and update the title? I will take a look once it's ready. You also need to rebase the PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21260 > There is a fundamental problem with how we pass the options through spark conf to fabric8. For each volume type and all possible volume options we would have to implement some custom code to map config values to fabric8 calls. This will result in big body of code we would have to support and means that Spark will always be somehow out of sync with k8s. This is indeed concerning, given that we don't yet support a lot of pod customization options yet, e.g., affinity and anti-affinity, security context, etc. Ideally, pod specs should be specified declaratively like in Deployment and StatefulSet, but Spark is configuration property based. The [Spark Operator](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) attempted to address this using initializers. But initializers are alpha and dangerous. Admission webhooks are an option, but again, they pose risks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21260 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2951/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21260 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2951/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21260 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3035/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21260 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21260 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90363/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21260 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21260 **[Test build #90363 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90363/testReport)** for PR 21260 at commit [`f1ada4c`](https://github.com/apache/spark/commit/f1ada4c402a7f5ca5fd5428a28a457cd2400c6f2). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21260 **[Test build #90363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90363/testReport)** for PR 21260 at commit [`f1ada4c`](https://github.com/apache/spark/commit/f1ada4c402a7f5ca5fd5428a28a457cd2400c6f2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21260 jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21260 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21260 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
