Carlo Curino commented on YARN-4390:

[~bikassaha] if the containers to be preempted (remember we respect the 
priority in the queue, and priority of containers) are in one box, the 8GB 
freed should be bundled and given to the AM, however the preemptionpolicy does 
not at the moment try to free resources to satisfy an exact request. 

This is an important philosophical point: 
I am quite convinced that preemption should be used to fix "large imbalances" 
in fairness/capacity between queues/users (hence the dead-zone in which we do 
not trigger preemption even if we are off balance), and not to micro-manage 
allocations. Keep in mind that preemption will take a while to kick in (by 
design), as it allows the application to respond to a preemption signal etc. As 
such in many cases the 8GB container request will be already otherwise 
satisfied before this preemption kicks in.  The current implementation follows 
this philosophy and only looks at the overall demand of resources, not at 
exactly which pending requests exists. 

I think this is correct and sufficient for large clusters with batch mostly 
workloads (like the one we were focusing on when we started preemption a few 
years back) since cluster conditions mutate too quickly for us to try to chase 
and micromanage allocations with preemption... In very small clusters, or in 
cluster running several long-running services, things might be different, as we 
can have potentially small, but very persistent imbalances which we might want 
to address with more surgical preemption actions.

my 2 cents.. 

> Consider container request size during CS preemption
> ----------------------------------------------------
>                 Key: YARN-4390
>                 URL: https://issues.apache.org/jira/browse/YARN-4390
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>    Affects Versions: 3.0.0, 2.8.0, 2.7.3
>            Reporter: Eric Payne
>            Assignee: Eric Payne
> There are multiple reasons why preemption could unnecessarily preempt 
> containers. One is that an app could be requesting a large container (say 
> 8-GB), and the preemption monitor could conceivably preempt multiple 
> containers (say 8, 1-GB containers) in order to fill the large container 
> request. These smaller containers would then be rejected by the requesting AM 
> and potentially given right back to the preempted app.

This message was sent by Atlassian JIRA

Reply via email to