Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3779#issuecomment-72586368
> If we have tasks at some high locality level that cannot be scheduled at
their preferred locality and there are tasks at a lower locality level that can
be scheduled, then I guess we might be concerned about whether scheduling the
less-local tasks could rob tasks waiting at a higher locality level of their
opportunity to run. This shouldn't happen, though, since those tasks will
already have been offered those resources and turned them down.
Actually, on reflection, I guess this gets trickier when you have multiple
task sets: in this case, you want to ensure that _all_ unscheduled
process-local tasks have turned down a resource before you allow _any_
node-local task to obtain it.
I'm going to try to trace through the interaction between DAGScheduler and
TaskSetManager to try to understand how we handle this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
