[
https://issues.apache.org/jira/browse/SLIDER-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved SLIDER-828.
-----------------------------------
Resolution: Fixed
Assignee: Steve Loughran
> Redundant container request from slider causing high load on busy cluster
> -------------------------------------------------------------------------
>
> Key: SLIDER-828
> URL: https://issues.apache.org/jira/browse/SLIDER-828
> Project: Slider
> Issue Type: Bug
> Components: appmaster
> Affects Versions: Slider 0.61, Slider 0.70
> Reporter: Shrijeet Paliwal
> Assignee: Steve Loughran
> Priority: Blocker
> Fix For: Slider 0.80
>
>
> Context:
> We were seeing very aggressive preemption done by Fair Scheduler and 98% of
> preemption activity is triggered due to slider queue's needs. Slider queue is
> stable queue i.e its containers don't churn and it has been provided a fair
> share guarantee of more than it needs (high weight & min share double of its
> steady state needs). So it was puzzling to see it triggering preemption. When
> I turned on debug logging of fair scheduler I noticed scheduler demand update
> thread reporting unusually high demand from Slider queue.
> Initial thought was a bug in scheduler but later I concluded its Slider's
> problem but not due to its own code but due to AMRMClient code. I can
> deterministically reproduce the issue on my laptop running a pseudo
> yarn+slider setup. I traced it to an open issue
> https://issues.apache.org/jira/browse/YARN-3020.
> The problem:
> 1. A region server fails for the first time, slider notices it and registers
> a request to RM via AMRMClient for a new container. At this time AMRMClient
> caches this allocation request with the 'Resource' (a data structure with
> memory, cpu & priority) as key. (source: AMRMClientImpl.java, cache is
> remoteRequestsTable)
> 2. A region server fails again, slider notices it and registers a request to
> RM again via AMRMClient for a (one) new container. AMRMClient finds that
> similar Resource request (the memory, cpu and priority for RS doesn't change
> obviously) in its cache, add +1 to the container count before putting it over
> wire.NOTE: Slider didn't need 2 containers, but ends up receiving 2. When
> containers are allocated, slider keeps one and discards one.
> 3. As explained in YARN-3020, with subsequent failures we will keep asking
> for more and more containers when in reality we always need one.
> For fair scheduler this means demand keeps going up. It doesn't know that
> slider ends up discarding the surplus containers. In order to satisfy the
> demand it kills mercilessly. Needless to say this will not be just triggered
> by container failure, even flexing should trigger this.
> The fix:
> Rumor is that AMRMClient doesn't have a bug, its intended behaviour (source:
> comments in YARN-3020). The claim is that on receiving container client
> should clear the cache by calling a method called 'removeContainerRequest'.
> Slider isn't following the protocol correctly, in Slider's defense the
> protocol is not well defined.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
