[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21238 Makes sense to me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21238 SG. @liyinan926, let's revisit this if we hear from 2.3 users. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21238 What would make this difficult to backport is the fact that this patch was built on top of the big refactor PR that only went in after 2.3. So we'd need to rewrite this with the old architecture which is a non-trivial effort. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21238 I think we can afford to hold off here. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21238 @mccheah, wdyt? I just haven't heard from any users here of 2.3 - if you think it's useful for 2.3.1 and low risk, then please feel free to propose a cherrypick. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21238 Maintenance releases most often have fixes for stability. We could maybe backport this since it's not a new feature but an omission from before. If it is going to be some effort, thanks to all the refactors that went in so far, I think we should think twice about whether we need to. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/21238 @foxish @mccheah should we also merge this to branch-2.3? I think we should target this for 2.3.1. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21238 LGTM. Merging to master. Thanks @mccheah --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21238 Requesting sign off and merge from @foxish --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90432/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 **[Test build #90432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90432/testReport)** for PR 21238 at commit [`fa095cd`](https://github.com/apache/spark/commit/fa095cd9faceb1247f3704a1a4949be834b05746). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90431/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 **[Test build #90431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90431/testReport)** for PR 21238 at commit [`aebdb68`](https://github.com/apache/spark/commit/aebdb6885237163b55a90fb739bcbbdcb00d7890). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3002/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3088/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3002/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 **[Test build #90432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90432/testReport)** for PR 21238 at commit [`fa095cd`](https://github.com/apache/spark/commit/fa095cd9faceb1247f3704a1a4949be834b05746). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3087/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 **[Test build #90431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90431/testReport)** for PR 21238 at commit [`aebdb68`](https://github.com/apache/spark/commit/aebdb6885237163b55a90fb739bcbbdcb00d7890). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21238 @rxin @liyinan926 @foxish addressed comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user erikerlandson commented on the issue: https://github.com/apache/spark/pull/21238 I agree with @mcheah that the potential code reuse is small. Keeping this as a separate pod construction step, decoupled from the user-exposed step, is cleaner. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21238 Also #21260 currently only supports hostPath and PVCs but you definitely want emptyDir for isolation (though that looks like a trivial enough change). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21238 @andrusha I don't think it's entirely analogous - for the simple reason that the hostPath volumes PR doesn't take into account `SPARK_LOCAL_DIRS`. That environment variable is used to determine the right place for shuffle files. We want to strictly tie that setting to emptydir volumes - and moreover this should be done transparently for the user, i.e. the user shouldn't have to know to provision these volumes to get better performance. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user andrusha commented on the issue: https://github.com/apache/spark/pull/21238 Seems like it addresses similar problem to https://github.com/apache/spark/pull/21095. It might be worth investigating how to unify both. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90225/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 **[Test build #90225 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90225/testReport)** for PR 21238 at commit [`f68fcd8`](https://github.com/apache/spark/commit/f68fcd80dcd93d6a10bb5f30493a4ecc167e5d28). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2854/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2854/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2931/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21238 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21238 **[Test build #90225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90225/testReport)** for PR 21238 at commit [`f68fcd8`](https://github.com/apache/spark/commit/f68fcd80dcd93d6a10bb5f30493a4ecc167e5d28). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
