er to offer 20 GB to the large framework.
>>>>>>
>>>>>> I understand this idea has some forward looking expectations on how
>>>>>> dynamic reservations would/could work. Caveat: I haven't involved myself
>>>>>> closely with that feat
ld be wrong with my
>>>>> expectations.
>>>>>
>>>>> Until something like that lands, the existing static reservations, of
>>>>> course, should work. But, that reduces utilization drastically if the
>>>>> large
>>>&
.
>>>>>
>>>>> Until something like that lands, the existing static reservations, of
>>>>> course, should work. But, that reduces utilization drastically if the
>>>>> large
>>>>> framework runs jobs sporadically.
>>>
.
>>>>
>>>> Until something like that lands, the existing static reservations, of
>>>> course, should work. But, that reduces utilization drastically if the large
>>>> framework runs jobs sporadically.
>>>>
>>>> Another idea is to have
hedule both the 20GB jobs and
>>> 1GB jobs. Within the framework, it can bin pack the 1GB jobs on to as small
>>> a number of slaves as possible. This increases the likelihood of finding
>>> 20GB on a slave. Combining that with preemptions from within the framework
>>&
o as small a
>> number of slaves as possible. This increases the likelihood of finding 20GB
>> on a slave. Combining that with preemptions from within the framework (a
>> simple kill of certain number of 1GB jobs) should satisfy the 20 GB jobs.
>>
>>
>>
>
.
>
>
>
> On Wed, Jun 24, 2015 at 9:26 AM, Tim St Clair wrote:
>
>>
>>
>> ----- Original Message -
>> > From: "Brian Candler"
>> > To: user@mesos.apache.org
>> > Sent: Wednesday, June 24, 2015 10:50:43 AM
>> > Subjec
sage -
> > From: "Brian Candler"
> > To: user@mesos.apache.org
> > Sent: Wednesday, June 24, 2015 10:50:43 AM
> > Subject: Re: Setting minimum offer size
> >
> > On 24/06/2015 16:31, Alex Gaudio wrote:
> > > Does anyone have other ideas?
&g
- Original Message -
> From: "Brian Candler"
> To: user@mesos.apache.org
> Sent: Wednesday, June 24, 2015 10:50:43 AM
> Subject: Re: Setting minimum offer size
>
> On 24/06/2015 16:31, Alex Gaudio wrote:
> > Does anyone have other ideas?
> HTCondo
On 24/06/2015 16:31, Alex Gaudio wrote:
Does anyone have other ideas?
HTCondor deals with this by having a "defrag" demon, which periodically
stops hosts accepting small jobs, so that it can coalesce small slots
into larger ones.
http://research.cs.wisc.edu/htcondor/manual/latest/3_5Policy_Co
We have struggled with this problem many times, and our solution was to
reduce the size of jobs. Reducing job size has many advantages:
- Big jobs have less chance of getting starved out
- Frameworks have less incentive to hold onto offers
- The DRF algorithm assumes that all frameworks want offe
On 24/06/2015 15:35, David Greenberg wrote:
I'm not aware of any minimum offer size option in mesos. What I've
seen success with is holding onto small offers and waiting until I
accumulate enough to launch the large task. This way, the need for
large offers doesn't affect the cluster, but the f
I'm not aware of any minimum offer size option in mesos. What I've seen
success with is holding onto small offers and waiting until I accumulate
enough to launch the large task. This way, the need for large offers
doesn't affect the cluster, but the framework that needs it gets it.
On Wed, Jun 24,
The following problem is mentioned in the Mesos technical paper at
http://mesos.berkeley.edu/mesos_tech_report.pdf
" when a cluster is filled by tasks with small resource requirements, a
framework f with large resource requirements may starve, because
whenever a small task finishes, f cannot a
14 matches
Mail list logo
