Jobs stall forever if mapred.task.cache.levels is set to 0.
-----------------------------------------------------------
Key: MAPREDUCE-2419
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2419
Project: Hadoop Map/Reduce
Issue Type: Task
Components: jobtracker
Affects Versions: 0.21.0, 0.20.1
Reporter: Gerald Guo
Priority: Minor
It seems that mapred.task.cache.levels is used by JobTracker to create task
caches for nodes at various levels. This makes data-locality scheduling
possible.
If I set mapred.task.cache.levels to 0 (to disable task caching) and use
default network topology, then mapreduce job will stall forever. The reason is
JobInProgress::findNewMapTask always returns -1. Field "nonRunningMapCache" is
empty and field "nonLocalMaps" is also empty.
I can think of two possible fixes.
1) If mapred.task.cache.levels is set 0, Hadoop should fall back to some
default caching strategy. E.g. put all tasks into JobInProgress::nonLocalMaps.
2) ignore values less than 1
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
