Sunil G commented on YARN-3510:

Hi [~cwelch] 
Thanks for taking this optimization up. I have few doubts here.

Here an evenly distributed preempting policy across applications are tried. But 
each application internally has containers from different priorities, and least 
priority container is selected first from an application for preemption.

Now consider a scenario where we have 2 applications (assuming map reduce). 
App1 has containers <10 containers:Priority 10, 5 containers:Priority 20>   Old 
App2 has containers <10 containers:Priority 10, 2 containers:Priority 20>   New 
As per new implementation, after 2 rounds, some containers of priority 10(maps) 
may  get preempted if I am not wrong. Is this intentional, because killing maps 
is costlier.

I feel, we can group containers based on priority among all applications, and 
then can do this preemption at each container priority level. It may be more 
better but we may have more buckets of priorities. Please share your thoughts.

> Create an extension of ProportionalCapacityPreemptionPolicy which preempts a 
> number of containers from each application in a way which respects fairness
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: YARN-3510
>                 URL: https://issues.apache.org/jira/browse/YARN-3510
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>            Reporter: Craig Welch
>            Assignee: Craig Welch
>         Attachments: YARN-3510.2.patch, YARN-3510.3.patch, YARN-3510.5.patch, 
> YARN-3510.6.patch
> The ProportionalCapacityPreemptionPolicy preempts as many containers from 
> applications as it can during it's preemption run.  For fifo this makes 
> sense, as it is prempting in reverse order & therefore maintaining the 
> primacy of the "oldest".  For fair ordering this does not have the desired 
> effect - instead, it should preempt a number of containers from each 
> application which maintains a fair balance /close to a fair balance between 
> them

This message was sent by Atlassian JIRA

Reply via email to