Github user ehnalis commented on the pull request:
https://github.com/apache/spark/pull/6082#issuecomment-101959561
@tgravescs Of course it differs from when you have a constant HB interval.
In any case you wish to have an adaptive solution or the one they've
implemented in MR. I can easily set up a scenario for you, where a 1 second of
HB interval will stress the RM and every user will suffer. I really don't see
your point here. Multiplicative back-off is a very simple and adaptive
solution. What you are trying to do is not adaptive to the cluster, thus being
not adaptive to the user, because practically the user sits on the cluster.
> Also if the cluster is idle with lots of resources you should get them
the first time before the sleep even happens, right? We call allocate before
launching the reporter thread, which will do an allocate that should get
response from first one before doing the sleep.
No. There is not everything about free resources, but the RM has its own
logic that needs time to complete. 200ms could be tuned, but back-off is
essential here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
