[
https://issues.apache.org/jira/browse/MAPREDUCE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758567#comment-16758567
]
BELUGA BEHR commented on MAPREDUCE-7180:
----------------------------------------
[~Jim_Brennan] Thank you for the feedback!
I could see a new configuration called growth-factor which specifies how large
to grow the container each time it is re-tried. This would be a percentage,
therefore, a growth-factor of 1.0f (100%) would preserve the current behavior.
I would just say, that, from my perspective, it is far more important that
MapReduce jobs complete than fail. Please consider the current workflow:
# User launches MapReduce job with 100 Mappers @ 1GB/container (for a total of
100GB)
# MapReduce job fails
# User alerts cluster administrator
# Cluster administrator diagnosis the issue (not exactly trivial)
# Administrator informs user that their job needs more memory
# User re-runs MapReduce job with 100 Mapper @ 2GB/container (for a total of
200GB)
# Job completes
This is a very tedious process and can add large lag times to resolution. Even
with all this, the fix is almost always the same: "throw more memory at the
problem." Also note, that if this failure was caused by a single Mapper that
required the 2GB container, then a single retry with a failed container would
require 102GB of memory on the cluster, whereas running the MapReduce job again
with all of the containers set to a larger size consumes many more resources
((100GB initial) + (3 container retries x 1GB) + (1 Job retry x 200GB) = 303GB).
For additional context, I work often with Hive-on-MapReduce. Users using a SQL
front-end do not want (and should not have to) know how to troubleshoot
MapReduce for failures. As long as there is some sort of logging or counter to
track the number of times a retry occurs due to OOM, configurations can be
tuned after the fact. Even if containers are failing sporadically, it may be
the case that users would rather fail a couple of containers (and retry with
larger container sizes) than to launch every container with a larger size.
> Relaunching Failed Containers
> -----------------------------
>
> Key: MAPREDUCE-7180
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7180
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv1, mrv2
> Reporter: BELUGA BEHR
> Priority: Major
>
> In my experience, it is very common that a MR job completely fails because a
> single Mapper/Reducer container is using more memory than has been reserved
> in YARN. The following message is logging the the MapReduce
> ApplicationMaster:
> {code}
> Container [pid=46028,containerID=container_e54_1435155934213_16721_01_003666]
> is running beyond physical memory limits.
> Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual
> memory used. Killing container.
> {code}
> In this case, the container is re-launched on another node, and of course, it
> is killed again for the same reason. This process happens three (maybe
> four?) times before the entire MapReduce job fails. It's often said that the
> definition of insanity is doing the same thing over and over and expecting
> different results.
> For all intents and purposes, the amount of resources requested by Mappers
> and Reducers is a fixed amount; based on the default configuration values.
> Users can set the memory on a per-job basis, but it's a pain, not exact, and
> requires intimate knowledge of the MapReduce framework and its memory usage
> patterns.
> I propose that if the MR ApplicationMaster detects that a container is killed
> because of this specific memory resource constraint, that it requests a
> larger container for the subsequent task attempt.
> For example, increase the requested memory size by 50% each time the
> container fails and the task is retried. This will prevent many Job failures
> and allow for additional memory tuning, per-Job, after the fact, to get
> better performance (v.s. fail/succeed).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
