If that floats the boat! Here: https://issues.apache.org/jira/browse/SLIDER-828
-- Shrijeet On Mon, Mar 23, 2015 at 3:15 PM, Ted Yu <[email protected]> wrote: > The JIRAs you mentioned are somewhat dormant. > > You can open a Slider JIRA to track this issue. > > Thanks > > On Mon, Mar 23, 2015 at 10:44 AM, Shrijeet Paliwal < > [email protected]> wrote: > > > More evidence: > > > > Spark is also affected: https://issues.apache.org/jira/browse/SPARK-2687 > > One more relevant yarn jira: > > https://issues.apache.org/jira/browse/YARN-1902 > > > > -- > > Shrijeet > > > > On Mon, Mar 23, 2015 at 10:15 AM, Shrijeet Paliwal < > > [email protected]> wrote: > > > > > Hello, > > > > > > *Context:* > > > > > > We were seeing very aggressive preemption done by Fair Scheduler and > 98% > > > of preemption activity is triggered due to slider queue's needs. Slider > > > queue is stable queue i.e its containers don't churn and it has been > > > provided a fair share guarantee of more than it needs (high weight & > min > > > share double of its steady state needs). So it was puzzling to see it > > > triggering preemption. When I turned on debug logging of fair > scheduler I > > > noticed scheduler demand update thread reporting unusually high demand > > from > > > Slider queue. > > > > > > Initial thought was a bug in scheduler but later I concluded its > Slider's > > > problem but not due to its own code but due to AMRMClient code. I can > > > deterministically reproduce the issue on my laptop running a pseudo > > > yarn+slider setup. I traced it to an open issue > > > https://issues.apache.org/jira/browse/YARN-3020. > > > > > > *The problem: * > > > > > > 1. A region server fails for the first time, slider notices it > > > and registers a request to RM via AMRMClient for a new container. At > this > > > time AMRMClient caches this allocation request with the 'Resource' (a > > data > > > structure with memory, cpu & priority) as key. > > > (source: AMRMClientImpl.java, cache is remoteRequestsTable) > > > 2. A region server fails again, slider notices it and registers a > request > > > to RM again via AMRMClient for a (one) new container. AMRMClient finds > > that > > > similar Resource request (the memory, cpu and priority for RS doesn't > > > change obviously) in its cache, add +1 to the container count before > > > putting it over wire.*NOTE*: Slider didn't need 2 containers, but ends > up > > > receiving 2. When containers are allocated, slider keeps one and > discards > > > one. > > > 3. As explained in YARN-3020, with subsequent failures we will keep > > asking > > > for more and more containers when in reality we always need one. > > > > > > For fair scheduler this means demand keeps going up. It doesn't know > that > > > slider ends up discarding the surplus containers. In order to satisfy > the > > > demand it kills mercilessly. Needless to say this will not be just > > > triggered by container failure, even flexing should trigger this. > > > > > > *The fix: * > > > > > > Rumor is that AMRMClient doesn't have a bug, its intended behaviour > > > (source: comments in YARN-3020). The claim is that on receiving > > > container client should clear the cache by calling a method called > > > 'removeContainerRequest'. Slider isn't following the protocol > correctly, > > in > > > Slider's defense the protocol is not well defined. > > > > > > Thoughts? > > > -- > > > Shrijeet > > > > > >
