Eric Payne commented on YARN-4781:

[~sunilg], sorry for the churn on here, but I think we need to get this right. 
I think we need to take a step back and decide what exactly the FairOrdering 
Preemption policy should do.

Your previous suggestion that it should consider pending as well as used when 
that feature is enabled caused me to begin thinking that we wanted FairOrdering 
Preemption to order apps in the same exact way as the FairOrderingApp policy 
does for assigning containers. However, the {{FairComparator}} does not compare 
which users are most underserved. Another part of the code does that. So, it's 
not enough for FairOrdering Preemption to just call into the {{FairComparator}}

I suggest that rather than trying to re-use the {{FairComparator}}, 
FairOrdering Preemption should mimic the behavior of the Capacity Scheduler 
{{FairOrderingPolicy}}. I know that would require a bit of a maintenance risk 
since if the behavior of one changes, the other must follow. However, I think 
that would be the simplest option at this point.

So, I think that the {{TAFairOrderingComparator}} should do what it does in my 
latest patch with 2 additions:
- if {{fair.enable-size-based-weight}} is enabled for the queue, consider 
pending resources as well
- if 2 apps are from the same user, the younger one is defined to be more 
underserved (and therefore will be preempted after the older one)

> Support intra-queue preemption for fairness ordering policy.
> ------------------------------------------------------------
>                 Key: YARN-4781
>                 URL: https://issues.apache.org/jira/browse/YARN-4781
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Major
>         Attachments: YARN-4781.001.patch, YARN-4781.002.patch, 
> YARN-4781.003.patch
> We introduced fairness queue policy since YARN-3319, which will let large 
> applications make progresses and not starve small applications. However, if a 
> large application takes the queue’s resources, and containers of the large 
> app has long lifespan, small applications could still wait for resources for 
> long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to 
> preempt resources of queue with fairness policy enabled.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to