Github user CodingCat commented on the pull request:
https://github.com/apache/spark/pull/1313#issuecomment-50260912
@mateiz for skipping the levels in computeValidLocalityLevels, that's
straightforward, however, the current TaskSetManager does not update the valid
locality after the completion of each task...I didn't do that because it's
unclear whether calling computeValidLocalityLevels for each time is a big
overhead
and....en....I think the current PR is just doing what you are describing
see the case branch in
(https://github.com/apache/spark/pull/1313/files#diff-bad3987c83bd22d46416d3dd9d208e76L440),
when the level returns no task, we tried to call NO_PREF
I think the code seems more complicate than you thought fall several reasons
1. we should call NO_PREF only after the level returns no task, otherwise,
we will over-complicate the login here, e.g. we have two PROCESS_LOCAL tasks,
if we call NO_PREF after the first successful launch, then it's actually
running the second PROCESS_LOCAL
2. when we call resourceOffer with NO_PREF, we should not allow the
preferredLocality to be modified due to the delay scheduling, so I added a
parameter allowAdjustPrefLocality: Boolean
3. we should avoid some duplicate traversing on the pendingTask list, e.g.
when you failed to find a task on PROCESS_LOCAL, you call resourceOffer
with NOPREF again, you should not search the task list for PROCESS_LOCAL again,
instead, we just skip that level
4. we should ensure NODE_LOCAL always be scheduled before NOPREF, it's
achieved by making NOPREF "farther" than NODE_LOCAL....
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
