Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1313#issuecomment-50294011
Maybe I misunderstood, but just to be clear, what I was suggesting is this:
- First call resourceOffers with PROCESS_LOCAL. It can only return a
process-local task.
- If this didn't return a task, call resourceOffers with NODE_LOCAL. It can
return a node-local task.
- If this didn't return a task, call resourceOffers with NO_PREFS. This
will always return a no-pref task, without doing any kind of delay scheduling,
but it also doesn't update the "last locality level" variable or anything like
that.
This is different from what you mentioned above because there's no 3 second
wait. If your TaskSet has only process-local and no-prefs tasks, it will
greedily launch all the process-local ones, then immediately launch all the
no-prefs ones.
I guess one case this doesn't cover is if the TaskSet has some tasks that
are process-local, some that are node-local, and some that are no-prefs. Then
it will launch the no-prefs ones before node-local during the 3 seconds where
it's waiting for node-local. I think this case is pretty uncommon, but if we do
want to fix it, probably the best way is to make resourceOffers on NO_PREFS
check if NODE_LOCAL is in myLocalityLevels, and if so, wait until its delay has
passed too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
