[
https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358994#comment-14358994
]
Allen Wittenauer commented on MAPREDUCE-5653:
---------------------------------------------
bq. You could make an argument that DistCp, as a Yarn application, knows
better than the defaults about how much memory it uses.
One could, but one pass through the code makes that argument kind of moot given
there is zero logic currently for distcp to have that level of intelligence.
bq. If the defaults in your cluster were to use a lot of memory, and DistCP
uses very little (after all, it's copying a buffer around), it's wasteful.
Part of the argument around MAPREDUCE-5785 revolves around the fact that one
doesn't actually want to set defaults anymore for MapReduce applications.
> DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-5653
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.9, 2.2.0
> Reporter: Mithun Radhakrishnan
> Assignee: Ratandeep Ratti
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-5653.branch-0.23.patch,
> MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch,
> MAPREDUCE-5653.trunk.patch
>
>
> When a DistCp job is run through Oozie (through a Java action that launches
> DistCp), one sees that mapred.child.java.opts as set from the caller is
> honoured by DistCp. But, DistCp doesn't seem to honour any overrides for
> configs mapreduce.[map,reduce].memory.mb.
> Problem has been identified. I'll post a patch shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
