Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1313#issuecomment-50249494
Hey Nan, sorry for the delay in getting to this. IMO this design is still
too complicated -- we are passing so much state to resourceOffer and it's not
super clear how the different parameters are supposed to interact. What would
be wrong with the following solution?
- Add a NO_PREFS level that is after NODE_LOCAL
- Return this as a valid locality level from the TaskSet, and make offers
for this level as part of the same sequence we make offers in now
This would be a minimal change from the current code that fixes the bug. I
also wouldn't try to optimize stuff as part of this patch by adding new
parameters; first of all it's not clear that those things are a performance
problem (many of the findTask checks are O(1)) and second any reorganization
should be done as a different JIRA and PR, since the goal of this one is to
just fix the bug. It's always riskier to merge a change if it refactors other
stuff.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
