[
https://issues.apache.org/jira/browse/MAPREDUCE-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041505#comment-14041505
]
Diane Feddema commented on MAPREDUCE-5661:
------------------------------------------
The description for mapreduce.cluster.local.dir (below) is misleading and could
be improved by saying it is NOT a replacement for deprecated property
mapred.local.dir.
Unfortunatley, Link
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
lists mapreduce.cluster.local.dir as the replacement for mapred.local.dir.
Instead it should explain intermediate files will be written
to yarn.nodemanager.local-dirs.
<property>
<name>mapreduce.cluster.local.dir</name>
<value>${hadoop.tmp.dir}/mapred/local</value>
<description>
The local directory where MapReduce stores intermediate
data files. May be a comma-separated list of
directories on different devices in order to spread disk i/o.
Directories that do not exist are ignored.
</description>
</property>
> ShuffleHandler using yarn.nodemanager.local-dirs instead of
> mapreduce.cluster.local.dir
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5661
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5661
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.2.0
> Reporter: Eric Sirianni
> Priority: Trivial
>
> While debugging an issue where a MapReduce job is failing due to running out
> of disk space, I noticed that the {{ShuffleHandler}} uses
> {{yarn.nodemanager.local-dirs}} for its {{LocalDirAllocator}} whereas all of
> the other MapReduce classes use {{mapreduce.cluster.local.dir}}:
> {noformat}
> $ find hadoop-mapreduce-project/hadoop-mapreduce-client/*/src/main/java/
> -name "*.java" | xargs grep "new LocalDirAllocator("
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java:
> LocalDirAllocator lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnOutputFiles.java:
> new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java:
> new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java:
> this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java:
> new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java:
> new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java:
> this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
> *****hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java:
> new LocalDirAllocator(YarnConfiguration.NM_LOCAL_DIRS);
> {noformat}
> This inconsistency feels like something that is likely to confuse admins.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
