Reduce number of getFileStatus call made from every task(TaskDistributedCache)
setup
------------------------------------------------------------------------------------
Key: MAPREDUCE-2011
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2011
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: distributed-cache
Reporter: Koji Noguchi
On our cluster, we had jobs with 20 dist cache and very short-lived tasks
resulting in 500 map tasks launched per second resulting in 10,000
getFileStatus calls to the namenode. Namenode can handle this but asking to
see if we can reduce this somehow.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
