Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/6430#issuecomment-108655236
@jerryshao Thanks for the update. The latest changes look fairly reasonable.
(1) I think you're correct. We don't actually need to wait for the initial
set of executors to register to set `initializing = false`. Before the first
stage is submitted, we can't actually ever ramp up our target. We can't cancel
either, however, because all executors have already registered. In other words,
setting `initializing = false` in this case does nothing.
(2) I was thinking we don't actually need to sync with the cluster manager
to get our executors while we're initializing. We don't do that currently, and
I don't see why we need to add that. This `initializing` flag is just a way for
us to delay canceling. Then we don't need to worry about spamming the AM with
sync messages while we initialize.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
