[ 
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351704#comment-16351704
 ] 

Steven Rand commented on YARN-7655:
-----------------------------------

Thanks [~yufeigu], new patch is attached. 

Unfortunately I'm still struggling to have the starved app be allocated the 
right number of containers in the test (though the preemption part happens 
correctly). The details of that are in my first comment above. It seems like 
the options are:

* What the current patch does, which is just leave a TODO above where we check 
for allocation.
* Only test that the preemption went as expected, and don't test allocation, 
i.e., don't call {{verifyPreemption}}.
* Find a way to have the allocation work out while still guaranteeing that the 
RR we consider for preemption is the {{NODE_LOCAL}} one. I thought I'd be able 
to figure this out, but have to admit I've been unsuccessful.

> avoid AM preemption caused by RRs for specific nodes or racks
> -------------------------------------------------------------
>
>                 Key: YARN-7655
>                 URL: https://issues.apache.org/jira/browse/YARN-7655
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 3.0.0
>            Reporter: Steven Rand
>            Assignee: Steven Rand
>            Priority: Major
>         Attachments: YARN-7655-001.patch, YARN-7655-002.patch
>
>
> We frequently see AM preemptions when 
> {{starvedApp.getStarvedResourceRequests()}} in 
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs 
> that request containers on a specific node. Since this causes us to only 
> consider one node to preempt containers on, the really good work that was 
> done in YARN-5830 doesn't save us from AM preemption. Even though there might 
> be multiple nodes on which we could preempt enough non-AM containers to 
> satisfy the app's starvation, we often wind up preempting one or more AM 
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM 
> containers for an RR that specifies a node or rack, then we should instead 
> expand the search space to consider all nodes. That way we take advantage of 
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a 
> patch with an initial implementation of this. We've been running it on a few 
> clusters, and have seen AM preemptions drop from double-digit occurrences on 
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is 
> less likely to be allocated resources at the most specific locality level 
> that it asked for. My opinion is that this tradeoff is worth it, but 
> interested to hear what others think as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to