Github user Tagar commented on the issue:
https://github.com/apache/spark/pull/19046
@tgravescs, here's quote from Wilfred Spiegelenburg - hope it answers both
of your questions.
> The behaviour I discussed earlier around the Spark AM reservations is not
optimal. It turns out that the AM is releasing and then acquiring the
reservations again and again until it has enough to run all tasks that it
needs. This seems to be triggered by getting a container assigned to the
application by the scheduler. Due to the sustained backlog trigger doubling the
request size each time it floods the scheduler with requests. This issue will
be logged as an internal jira at first. The next steps will be to discuss that
behaviour with the Spark team with the goal of making it behave better on the
cluster. The MR AM does behave better in this respect as it takes into account
the available resources for the application via what is called "headroom". The
Spark AM does not do this.
thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
