[
https://issues.apache.org/jira/browse/MAPREDUCE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761417#comment-16761417
]
BELUGA BEHR commented on MAPREDUCE-7180:
----------------------------------------
[~wilfreds] I wanted to further edit my previous comment, but I'll just write a
new one. Your last message was very clarifying. Thanks. I need more
explanation on my side.
I have witnessed this particular errors many times with default settings (no
Xmx defined manually):
bq. Container ... is running beyond physical memory limits
So it may be the case that an 80/20 split is too aggressive for general use.
However, how to determine exactly the right number to use here? I don't know.
Can anyone know? I do know however that if a few workloads require more
headroom (say, 75-25) they shouldn't fail. By increasing the container size,
the headroom expands nominally as well and this case is covered.
*In addition* this change would cover cases where a Mapper or Reducer fail
because of an OOM situation. Simply increasing the size of the container
increases the JVM heap size and provides tasks additional chances to succeed.
> Relaunching Failed Containers
> -----------------------------
>
> Key: MAPREDUCE-7180
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7180
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv1, mrv2
> Reporter: BELUGA BEHR
> Priority: Major
>
> In my experience, it is very common that a MR job completely fails because a
> single Mapper/Reducer container is using more memory than has been reserved
> in YARN. The following message is logging the the MapReduce
> ApplicationMaster:
> {code}
> Container [pid=46028,containerID=container_e54_1435155934213_16721_01_003666]
> is running beyond physical memory limits.
> Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual
> memory used. Killing container.
> {code}
> In this case, the container is re-launched on another node, and of course, it
> is killed again for the same reason. This process happens three (maybe
> four?) times before the entire MapReduce job fails. It's often said that the
> definition of insanity is doing the same thing over and over and expecting
> different results.
> For all intents and purposes, the amount of resources requested by Mappers
> and Reducers is a fixed amount; based on the default configuration values.
> Users can set the memory on a per-job basis, but it's a pain, not exact, and
> requires intimate knowledge of the MapReduce framework and its memory usage
> patterns.
> I propose that if the MR ApplicationMaster detects that a container is killed
> because of this specific memory resource constraint, that it requests a
> larger container for the subsequent task attempt.
> For example, increase the requested memory size by 50% each time the
> container fails and the task is retried. This will prevent many Job failures
> and allow for additional memory tuning, per-Job, after the fact, to get
> better performance (v.s. fail/succeed).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
