[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user jomach commented on the issue: https://github.com/apache/spark/pull/14644 We have some servers running 8 GPUs on mesos. I would like to run Spark on it but I need to be able from spark to allocate a GPU only per map phase. On Hadoop 3.0 you can do spark.yarn.executor.resource.yarn.io/gpu. I have a Spark job that receives a list of files to process, each map on spark should call a c script that reads a chunk of the list and process it on the gpu. For this I need that Spark recognizes the allocated gpu from Mesos like GPU0 is yours and of course mesos needs to mark that gpu as used. with this gpu.max this is not possible --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14644 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66613/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #66613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66613/consoleFull)** for PR 14644 at commit [`98377b5`](https://github.com/apache/spark/commit/98377b5e65a5a1517edc8701f26183020e3a32ea). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user tnachen commented on the issue: https://github.com/apache/spark/pull/14644 I just tested it with a GPU instance and it works. @mgummelt @klueska any more comments? otherwise @srowen I think we should merge as we no longer have any comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #66613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66613/consoleFull)** for PR 14644 at commit [`98377b5`](https://github.com/apache/spark/commit/98377b5e65a5a1517edc8701f26183020e3a32ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66160/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #66160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66160/consoleFull)** for PR 14644 at commit [`de6d201`](https://github.com/apache/spark/commit/de6d201497eef2e73843af8cbef806b2f8b1786f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #66160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66160/consoleFull)** for PR 14644 at commit [`de6d201`](https://github.com/apache/spark/commit/de6d201497eef2e73843af8cbef806b2f8b1786f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user tnachen commented on the issue: https://github.com/apache/spark/pull/14644 Ya, the default GPU requirement I have is 0 (cores per executor/node is 1). I'm still gathering feedback what's the more sensible thing to do for GPUs. We can either set a configurable amount that each executor has to use, or have a max. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14644 ...so if you ask for 1 GPU you may only get 0? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user tnachen commented on the issue: https://github.com/apache/spark/pull/14644 1. Good catch, my old patch had docs but I rebased and it didn't apply for some reason. Let me add it. 2,3: we don't fail if you ask for more GPUs since it's not a hard requirement but simply a max, just like how CPUs.max work. I didn't add a required amount setting but we can certainly add it in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14644 1. Any plans to add documentation? 2. What happens if you ask for (any, more) GPUs than there are? 2. If it fails, that could be a good test: ask for a very large number and expect scheduling to fail. (that said, I don't know how scheduling will fail here...YARN varies depending on whether you ask for out of range resources vs. unsatisfied ones) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user tnachen commented on the issue: https://github.com/apache/spark/pull/14644 @mgummelt @srowen Please review as well --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user tnachen commented on the issue: https://github.com/apache/spark/pull/14644 @klueska Just updated the patch and I think it's using the right semantics now, where it has a global gpus max just like cores. Can you try it out? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65775/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #65775 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65775/consoleFull)** for PR 14644 at commit [`6d34632`](https://github.com/apache/spark/commit/6d3463233b16bf23f8a8d0f33d2bb6bf79965168). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #65775 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65775/consoleFull)** for PR 14644 at commit [`6d34632`](https://github.com/apache/spark/commit/6d3463233b16bf23f8a8d0f33d2bb6bf79965168). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #65708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65708/consoleFull)** for PR 14644 at commit [`5e65a51`](https://github.com/apache/spark/commit/5e65a5168b0c8b9757b4e17b8a903a1abc6be108). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65708/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #65708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65708/consoleFull)** for PR 14644 at commit [`5e65a51`](https://github.com/apache/spark/commit/5e65a5168b0c8b9757b4e17b8a903a1abc6be108). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #65700 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65700/consoleFull)** for PR 14644 at commit [`3031915`](https://github.com/apache/spark/commit/30319156a9936df4da0249a1e38011ff90c2a3a6). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65700/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14644 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14644: [SPARK-14082][MESOS] Enable GPU support with Mesos
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14644 **[Test build #65700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65700/consoleFull)** for PR 14644 at commit [`3031915`](https://github.com/apache/spark/commit/30319156a9936df4da0249a1e38011ff90c2a3a6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
