We have struggled with this problem many times, and our solution was to reduce the size of jobs. Reducing job size has many advantages:
- Big jobs have less chance of getting starved out - Frameworks have less incentive to hold onto offers - The DRF algorithm assumes that all frameworks want offers equally (unless you use weighting). This means that if a framework with large jobs receives a lot of small offers it can't use, and it is therefore not only starved itself, but also starving out the smaller jobs. This leads to more under-utilization of the cluster. There are other ways to solve this problem, besides reducing job size. These are probably more complicated, and I'd love if others have good solutions or comments here: - Statically partition the cluster such that some machines serve large jobs (and perhaps some small jobs with uniqueness constraints that guarantee small jobs won't all cluster on one host). - Write your own variant of the Allocator module (not just the Sorter module) that gives preference to large offers. I think you might have to modify the sort() api call <https://github.com/apache/mesos/blob/2ef849a7ca91a2785d950141a6bc1819d40a75d3/src/master/allocator/mesos/hierarchical.hpp#L897> to include the size of an offer. This generally seems hard. - Wait for optimistic offers or some other planned Mesos feature and hope the community will somehow figure this out? Does anyone have other ideas? As I mentioned above, my solution was to make all frameworks generate small jobs, where small is relative to the amount of resources on a given slave. Thanks, Alex Gaudio On Wed, Jun 24, 2015 at 10:44 AM Brian Candler <[email protected]> wrote: > On 24/06/2015 15:35, David Greenberg wrote: > > I'm not aware of any minimum offer size option in mesos. What I've > > seen success with is holding onto small offers and waiting until I > > accumulate enough to launch the large task. This way, the need for > > large offers doesn't affect the cluster, but the framework that needs > > it gets it. > This means that the execution of a task doesn't need to be linked to a > single resource offer, but they can be aggregated? > > Also, it seems to me this would break down if there were two frameworks > both which required 20GB of RAM. Let's say each of them grabs 16 x 1GB > offers. At this point all 32GB of one node has been committed, and there > is deadlock. > > Conversely, ISTM that a 20GB minimum offer size would lead to big > wastage in the case where frameworks only made 1GB allocations. After > 13GB of RAM had been used (19GB free), no further offers would be made. > > Regards, > > Brian. > >

