Github user CodingCat commented on the pull request:
https://github.com/apache/spark/pull/1313#issuecomment-49499397
@lirui-intel , you are right, we will not get a PROCESS_LOCAL + NOPREF
setup in any case, since PROCESS_LOCAL is also NODE_LOCAL. I also agree with
that the valid localities should be recalculated when the executor is lost
(since we do that when the executer is added, there is no point for us not to
do the same thing when the executer is gone...)
As @mateiz said, If we add a special locality between NODE_LOCAL and
RACK_LOCAL, and always pass that as the parameter of findTask, it seems that we
don't need to recalculate the valid locality upon the completion of each task;
because PROCESS_LOCAL and NODE_LOCAL will always be executed before noPref and
when there are only noPref tasks or when the process_local are all started, we
can also ensure that the noPref tasks are checked for each call of TaskManager
Hi, all, sorry for the delay of the patch update....a bit hectic week, will
update it within the weekend...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
