GitHub user yanji84 opened a pull request:
https://github.com/apache/spark/pull/17979
[SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus
required in each spark executor when running on mesos
## What changes were proposed in this pull request?
Currently, Spark only allows specifying overall gpu resources as an upper
limit, this adds a new conf parameter to allow specifying a hard limit on the
number of gpu cores for each executor while still respecting the overall gpu
resource constraint
## How was this patch tested?
Unit testing
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanji84/spark ji/set_allow_set_docker_user
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17979.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17979
----
commit 5f8ccd5789137363e035d1dfb9a05d3b9bf3ce6b
Author: Ji Yan <[email protected]>
Date: 2017-03-10T05:30:11Z
respect both gpu and maxgpu
commit 33ebff693d9b78a15221f931dbbca777cba944e0
Author: Ji Yan <[email protected]>
Date: 2017-03-10T05:43:21Z
Merge branch 'master' into ji/hard_limit_on_gpu
commit c2c1c5b66436a439e1d7342b7a2c58c502e26d6b
Author: Ji Yan <[email protected]>
Date: 2017-03-10T05:30:11Z
respect both gpu and maxgpu
commit c5c5c379fc27f579952700fdf2d15dae9eba104a
Author: Ji Yan <[email protected]>
Date: 2017-05-13T16:25:48Z
Merge branch 'ji/hard_limit_on_gpu' of https://github.com/yanji84/spark
into ji/hard_limit_on_gpu
commit ba87b35817a7288b9b6aa41f4ac2244e235f2efd
Author: Ji Yan <[email protected]>
Date: 2017-05-13T16:53:59Z
fix syntax
commit 5ef2881a2b1e1180b73d532988bab72c5fdab64c
Author: Ji Yan <[email protected]>
Date: 2017-05-14T20:02:16Z
fix gpu offer
commit c301f3d1e05cc7359142a6cfb8222ad65cbdd9eb
Author: Ji Yan <[email protected]>
Date: 2017-05-14T20:15:55Z
syntax fix
commit 7a07742f4e004e0e88aa2b3bc5143adab3689644
Author: Ji Yan <[email protected]>
Date: 2017-05-15T00:30:50Z
pass all tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
