Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/1313#issuecomment-49200080
I just noticed that pendingTasksWithNotReadyPrefs is not being used now ?
It is getting updated but never actually queried from ...
Do we need to maintain it ?
The way I initially thought about this problem was,
1) When a task has no preferred location by definition : schedule it on any
node when there are no NODE_LOCAL tasks available for that executor.
2) When a task has preferred location defined, but none available right
now, treat is as ANY task : so that other PROCESS/NODE/RACK local tasks have
precedence over it. If/when a node/rack local host pops in, it becomes eligible
for better schedule preference.
@CodingCat, @kayousterhout @lirui-intel any thoughts ? I might be missing
somethere here !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
