Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/8100#issuecomment-151013429
Hi @vanzin , this patch improves the container locality calculating
algorithm by considering the locality of pending requests.
1. If the locality of current pending requests cannot match the locality
needs of pending tasks, we will cancel these pending requests.
2. If the locality of current pending request can match the locality needs,
treating them as valid requests in locality calculation, avoid sending
unnecessary requests again.
3. Locality unmatched requests will be canceled and recalculated again with
locality preferences of pending tasks, to get the optimal locality distribution.
Note this is a best effort algorithm based on the status of Yarn and Spark
at that time, since the status will be changed, it cannot guarantee the global
optimal result.
Please help to review, thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
