Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/6082#issuecomment-101825308
Normally I would expect us to grab most of it up front and then run for a
while without needing more, then perhaps iterate if dynamic allocation is on
and we go to different stage. it seems like a person should know pretty
quickly after initial testing what their queue limits are, but a valid point.
It just seems this kind of limits the usefulness of it. If the cluster is
idle it does let me get stuff quicker, but if the cluster is idle then just
decrease your heartbeat anyway. I think having just the 2 configs works the
same in that case. if its idle with lots of free resources, I get them and
then go to the slower heartbeat.
If the cluster or queue is at all busy you might be delayed and then I'm
back to asking slower and this change does me no good. personally I would set
the heartbeat when need containers at like 2 or 3 seconds and leave the other
at 5 seconds. Making it 200ms for the entire time would be to much load on RM
atleast for our clusters.
Also if the cluster is idle with lots of resources you should get them the
first time before the sleep even happens, right? We call allocate before
launching the reporter thread, which will do an allocate that should get
response from first one before doing the sleep.
Have you run this on various conditions to see what actually happens?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
