[
https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938852#comment-15938852
]
Allen Wittenauer commented on MAPREDUCE-5653:
---------------------------------------------
Yeah, the whole situation is kind of screwy and I'm definitely sympathetic..
distcp directly setting memory parameters is clearly a problem but we also
can't suddenly change this after years of working this way by simply removing
those parameters. A "reasonable default" in code bases without MAPREDUCE-5785
and without making the situation worse (e.g., Philip's concerns, the proposed
"fix" in HADOOP-14176, us making assumptions about what JVM is in use, etc)
would need to be provided.
distcp, years ago, probably should have provided alternate ways to set it's
memory, etc, and then changed these params based upon either the provided
values or use it's own internal defaults when they weren't overriden. This
"hides" the fact that it's "just" a fancy MR job while still providing the
flexibility needed for advanced users. For branch-2, that's really the only
way I see out of this mess. But that ship has mostly sailed, with Hadoop 2.x
on it's deathbed and the problem mostly solved in 3.x by this
patch+MAPREDUCE-5785.
> DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-5653
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.23.9, 2.2.0
> Reporter: Mithun Radhakrishnan
> Assignee: Ratandeep Ratti
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5653.branch-0.23.patch,
> MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch,
> MAPREDUCE-5653.trunk.patch
>
>
> When a DistCp job is run through Oozie (through a Java action that launches
> DistCp), one sees that mapred.child.java.opts as set from the caller is
> honoured by DistCp. But, DistCp doesn't seem to honour any overrides for
> configs mapreduce.[map,reduce].memory.mb.
> Problem has been identified. I'll post a patch shortly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
