Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2746#issuecomment-60198472
Yep - that's exactly what I was wondering about. If YARN doesn't expose the
internal counter it seems like there is a race (actually even if it does
expose it, there still is a minor race where you could read it and then
reset it but it changes in the middle). I guess we just live with it...
On Wed, Oct 22, 2014 at 11:21 PM, Sandy Ryza <notificati...@github.com>
wrote:
> So yeah it internally decrements the pending number to 8. The app can and
> is expected to infer YARN has decremented the counter. Maybe TMI, but for
> getting a grasp on it, it might be helpful to understand the race
> conditions this approach exposes - i.e. there are situations where YARN
can
> overallocate. For example imagine you requested 10 and then you decide you
> want 11. YARN just got 2 for you and decremented its counter to 8. You
> might tell YARN you want 11 before finding out about the 2 YARN is giving
> to you, which means you would overwrite the 8 with 11. In the brief period
> before you can go back to YARN and tell it you only want 9 now, it could
> conceivably give you 11 containers, for a total of 13, which is more than
> you ever asked for. The app is expected to handle these situations and
> release allocated containers that it doesn't need.
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/2746#issuecomment-60197171>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
