Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/6082#issuecomment-102137968
@tgravescs if I understand the point you're trying to make, is that there's
a lower bound for the time it takes to allocate a container from YARN, and
polling faster than that won't help much?
I guess it would be nice if the YARN client was event-driven (i.e. YARN
could send a "hey there are new containers" message instead of you having to
poll), but that's not the case... but barring that, having a more aggressive
polling period seems to help, even if a little bit. I expect these "little
bits" to accumulate when you're using dynamic allocation, though. And if YARN
magically becomes faster, then the aggressive polling would help too.
As for removing the exponential backoff, I don't feel strongly one way or
another. It could potentially reduce latency, since you'd be sleeping less, but
at the cost of putting more load on the RM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
