GitHub user sebastienrainville opened a pull request:
https://github.com/apache/spark/pull/10924
[SPARK-13001] [CORE] [MESOS] Prevent getting offers when reached max cores
Similar to https://github.com/apache/spark/pull/8639
This change rejects offers for 120s when reached `spark.cores.max` in
coarse-grained mode to mitigate offer starvation. This prevents Mesos to send
us offers again and again, starving other frameworks. This is especially
problematic when running many small frameworks on the same Mesos cluster, e.g.
many small Sparks streaming jobs, and cause the bigger spark jobs to stop
receiving offers. By rejecting the offers for a long period of time, they
become available to those other frameworks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sebastienrainville/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10924.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10924
----
commit 181a6efbd8c3c2d620107b65db163015b4f35b39
Author: Sebastien Rainville <[email protected]>
Date: 2016-01-23T04:11:32Z
[SPARK-13001] [CORE] [MESOS] Prevent getting offers in coarse-grained mode
when reached max cores
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
