[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742797#comment-14742797 ] Xianyin Xin commented on YARN-2154: --- The new logic would solve the issue raised in YARN-3405 and YARN-4134. link them for tracking. > FairScheduler: Improve preemption to preempt only those containers that would > satisfy the incoming request > -- > > Key: YARN-2154 > URL: https://issues.apache.org/jira/browse/YARN-2154 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Arun Suresh >Priority: Critical > Attachments: YARN-2154.1.patch > > > Today, FairScheduler uses a spray-gun approach to preemption. Instead, it > should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710324#comment-14710324 ] Arun Suresh commented on YARN-2154: --- Thanks for going thru the patch [~adhoot], [~kasha] and [~bpodgursky], bq. ..The previous ordering is better since if you happen to choose something just above its fairShare, after preemption it may go below and cause additional preemption, causing excessive thrashing of resources. This will not happen, as the current patch has a check to only preempt from an app, a container above its fair/min share. Am still working on the unit tests.. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Priority: Critical Attachments: YARN-2154.1.patch Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708215#comment-14708215 ] Karthik Kambatla commented on YARN-2154: bq. The previous ordering is better since if you happen to choose something just above its fairShare, after preemption it may go below and cause additional preemption, causing excessive thrashing of resources. Has been a while since I saw the patch, but would think that it honors the guarantees FairScheduler provides: # Resources are preempted from a Schedulable, only if it is above its fairshare. # Preempting resources from a Schedulable doesn't take it under its fairshare. In other words, it shouldn't lead to additional preemption. It is possible that its fairshare goes up right after a preemption and triggers additional preemption, but I guess we can't foresee it. Ordering shouldn't affect correctness of 1 or 2. Ordering does reduce the impact of preemption in case the fairshare increases because another queue goes inactive. [~asuresh] - could you confirm the patch conforms to the above guarantees? Do we have unittests that check these invariants? Once we identify all the apps we could preempt from, can we order them for efficiency purposes? What would be the cost of such ordering? FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Priority: Critical Attachments: YARN-2154.1.patch Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704209#comment-14704209 ] Anubhav Dhoot commented on YARN-2154: - Apart from this, the patch changes the order of choosing which apps to preempt from. Earlier it would prefer apps most over the limit and now there is no order. The previous ordering is better since if you happen to choose something just above its fairShare, after preemption it may go below and cause additional preemption, causing excessive thrashing of resources. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Priority: Critical Attachments: YARN-2154.1.patch Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698178#comment-14698178 ] Karthik Kambatla commented on YARN-2154: [~bpodgursky] - I was only observing what the patch seems to be doing. The patch prioritizes currently running apps over pending apps. The pending apps aren't necessarily starved - subsequent preemptions should handle them. [~asuresh] - could you please confirm the same. If this is not the case, I agree with [~bpodgursky] that we shouldn't starve small apps. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Priority: Critical Attachments: YARN-2154.1.patch Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679355#comment-14679355 ] Ben Podgursky commented on YARN-2154: - Hi, I've been watching this ticket since this has been a big problem on our cluster.Karthik, your last comment concerned me because it seems like it will violate one of the things FairScheduler is supposed to promise: Unlike the default Hadoop scheduler, which forms a queue of jobs, this lets short jobs finish in reasonable time while not starving long jobs.' and When other jobs are submitted, tasks slots that free up are assigned to the new jobs, so that each job gets roughly the same amount of CPU time. If a job can't even get an AM, it surely won't be able to get an equitable portion of slots for Maps and Reduces -- being able to get small jobs through quickly even when the pool has huge jobs running is the reason we use FairScheduler. Sorry if I'm misunderstanding that comment. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Priority: Critical Attachments: YARN-2154.1.patch Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14663177#comment-14663177 ] Karthik Kambatla commented on YARN-2154: Thanks for posting the approach here, Arun. Observation: We are prioritizing marked resources over unmarked resources. If a queue has some large jobs that have already started and a small job waiting for resources for its AM to start, we prioritize large jobs. Is this okay? I feel it might be. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Priority: Critical Attachments: YARN-2154.1.patch Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615228#comment-14615228 ] Arun Suresh commented on YARN-2154: --- This looks like it is required for a complete solution to YARN-3453 [~kasha], Thanks for proposing this. If are you working on this actively, do you mind if I take it up ? I have a couple of ideas. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615246#comment-14615246 ] Karthik Kambatla commented on YARN-2154: Would be nice to discuss approaches on JIRA before putting in the effort to code it up. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Priority: Critical Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615241#comment-14615241 ] Karthik Kambatla commented on YARN-2154: Please go ahead and take it. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125950#comment-14125950 ] Karthik Kambatla commented on YARN-2154: Just discussed this with [~ashwinshankar77] offline. He rightly pointed out the sort order should take usage into account. I ll post what the order should be, as soon as I get to consult my notes. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126005#comment-14126005 ] Sandy Ryza commented on YARN-2154: -- I'd like to add another constraint that I've been thinking about into the mix. We don't necessarily need to implement it in this JIRA, but I think it's worth considering how it would affect the approach. A queue should only be able to preempt a container from another queue if every queue between the starved queue and their least common ancestor is starved. This essentially means that we consider preemption and fairness hierarchically. If the marketing and engineering queues are square in terms of resources, starved teams in engineering shouldn't be able to take resources from queues in marketing - they should only be able to preempt from queues within engineering. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[
https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123380#comment-14123380
]
Karthik Kambatla commented on YARN-2154:
At an abstract level, I propose the following:
# For each queue, the child {{Schedulable}}s should be sorted in the ascending
order of (desiredShare - fairShare). In case of FSLeafQueue, this corresponds
to running applications.
# Add {{FSQueue#estimateStarvation(ListFSAppAttempt)}} to find enough
applications, considering minshare and fairshare starvation in that order. Or,
should we have two methods, one for each kind of starvation.
# For a starved queue, we do a depth-first traversal (in the order of
deprivation) to find these applications. If none of child queues/applications
are under their shares, we pick the application (if it exists) with a positive
demand and least over its share.
# Once the application list is computed, we traverse through the
ResourceRequests (ordered first by application, second by priority) and preempt
any matching containers currently assigned to an application over its share.
# As a follow up, we can consider looking at this application list first at
schedule time.
Thoughts?
FairScheduler: Improve preemption to preempt only those containers that would
satisfy the incoming request
--
Key: YARN-2154
URL: https://issues.apache.org/jira/browse/YARN-2154
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.4.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Today, FairScheduler uses a spray-gun approach to preemption. Instead, it
should only preempt resources that would satisfy the incoming request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[
https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123541#comment-14123541
]
Wei Yan commented on YARN-2154:
---
bq. 1. For each queue, the child {{Schedulable}}s should be sorted in the
ascending order of (desiredShare - fairShare). In case of FSLeafQueue, this
corresponds to running applications.
Do we still need to do the sorting? As we traverse from the root queue, and
take applications from the queues which are starvated for min/fair share. I
think we collect all applications from starvated queues.
bq. 4. Once the application list is computed, we traverse through the
ResourceRequests (ordered first by application, second by priority) and preempt
any matching containers currently assigned to an application over its share.
To sort the applications, do we still need to follow the policy? For example,
for two applications belong to the same FSLeafQueue, they should be sorted
according to the leafQueue's policy; for two applications belong to two
different leaf queues, may need to refer to parent queue's policy.
And, after preempting one container and assigning the resource to one
application in the list, we also need to check whether that application and its
parent queue is still starvated.
FairScheduler: Improve preemption to preempt only those containers that would
satisfy the incoming request
--
Key: YARN-2154
URL: https://issues.apache.org/jira/browse/YARN-2154
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.4.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Today, FairScheduler uses a spray-gun approach to preemption. Instead, it
should only preempt resources that would satisfy the incoming request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123608#comment-14123608 ] Karthik Kambatla commented on YARN-2154: bq. Do we still need to do the sorting? As we traverse from the root queue, and take applications from the queues which are starvated for min/fair share. I think we collect all applications from starvated queues. Queues might starve without their children starving. The sorting is for those cases. bq. To sort the applications, do we still need to follow the policy? We shouldn't need to sort the applications. The first application corresponds to the queue that is starved the most even though the application itself might not be the most starved one. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
[ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111724#comment-14111724 ] Karthik Kambatla commented on YARN-2154: Started looking into this. Today, we just look at the amount of resources to be preempted. Instead, we should collect a list of applications for which we are preempting containers. Iterate through these applications and their ResourceRequests to find potential matches in free resources and subsequently in resources assigned to another application that is over its fairshare. Will post a patch for this once YARN-2395 and YARN-2394 get committed. FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request -- Key: YARN-2154 URL: https://issues.apache.org/jira/browse/YARN-2154 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only preempt resources that would satisfy the incoming request. -- This message was sent by Atlassian JIRA (v6.2#6252)
