Github user lirui-intel commented on the pull request:
https://github.com/apache/spark/pull/892#issuecomment-45592572
@kayousterhout - I've fixed how we compute valid locality levels and added
some unit test.
Now computeValidLocalityLevels considers a level as valid only if some
executors are available to satisfy that level. And computeValidLocalityLevels
is called every time new executor is added. (I'm not sure if it should also be
called when executor is lost and when task finishes)
I also modified how the allowed locality is determined slightly. Now
allowedLocality is set to maxLocality if allowedLocality is greater and
maxLocality is a valid level to this TaskSetManager. This may help avoid the
problem you mentioned earlier (when RACK_LOCAL is the only valid level and we
tries from PROCESS_LOCAL).
Please help to review. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
