Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1313#issuecomment-49205621
I'm curious about one thing here: doesn't this change mean that we might
wait longer before launching a no-prefs or speculative task, due to delay
scheduling? This is because in TaskSetManager we take the preferredLocality and
make it no bigger than our allowed locality level based on delays. The unit
tests seem to add extra sleeps for this reason.
I don't really like this behavior, since with both kind of tasks you'd like
to launch them as soon as possible. No-prefs tasks can just run, and for
speculative tasks you want to launch them ASAP in situations like Spark
Streaming, where a 3-second delay could mess up your stream latency.
It might be better to add a special "preferredLocality" value for no
preference, and always call the TaskSetManager with that value after calling it
with the others, then have a code path in there that deals with that specially
(not taking the min of that with allowedLocality).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
