Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/6082#issuecomment-103201086
I think this change makes a lot of sense and stands on its own without a
full workload to benchmark against. I think the "empty queue" situation that
you refer to @tgravescs is fairly common, especially when running benchmarks.
I don't think you need a small job to see the benefit. With continuous
scheduling turned on, the Fair Scheduler can assign tons of containers without
waiting for NodeManager heartbeats, so a Spark application should be able to
ramp up on executors very quickly after this change.
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
