Arun Suresh updated YARN-2154:
    Attachment: YARN-2154.1.patch

Attaching a proof-of-concept patch.

The patch introduces an extra stage in the current preemption logic. Salient 
points to note :
# In the first stage, we iterate through starved LeafQueues and obtain an 
aggregate  of the {{ResourceDeficit}}, which has two fields:
## unmarked resources : The deficit for which the queue is starved for, 
essentially, no app can be allocated to the queue due to the deficit
## marked resources : These are app specific deficits, viz. node specific 
resources that an app is waiting on to launch a container.
# In the second stage, we try to match the Marked resources obtained in the 
first step with containers owned by apps that are consuming above their 
fair/min share. If we find such a container,
## we first see if any app is already reserved on the Node hosting the 
## If no, we Reserve the app originating the resource Request on the Node
## we then place the container in the {{warnedContainers}} list
## we return the totalResources that we reclaimed
# In the last stage, we call {{preemptResources}} as before.. with the unmarked 
resources + the reclaimed resources in the previous stage. At which time, the 
{{warnedContainers}} list will be iterated over and containers will be killed.

# The Matching can happen more efficiently. In the current patch, all first 
matching container that fits a resourcereq is targeted for preemption. This can 
probably be modified to a best fit algorithm 
# Fixing test cases.

> FairScheduler: Improve preemption to preempt only those containers that would 
> satisfy the incoming request
> ----------------------------------------------------------------------------------------------------------
>                 Key: YARN-2154
>                 URL: https://issues.apache.org/jira/browse/YARN-2154
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.4.0
>            Reporter: Karthik Kambatla
>            Assignee: Arun Suresh
>            Priority: Critical
>         Attachments: YARN-2154.1.patch
> Today, FairScheduler uses a spray-gun approach to preemption. Instead, it 
> should only preempt resources that would satisfy the incoming request. 

This message was sent by Atlassian JIRA

Reply via email to