[
https://issues.apache.org/jira/browse/MAPREDUCE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16759996#comment-16759996
]
Jim Brennan commented on MAPREDUCE-7180:
----------------------------------------
[~belugabehr] I agree that there are use cases where this would be useful. But
there are also cases where the automatic growth would not be desirable. For
example:
# On a large multi-tenant cluster that serves a large number of users
# Hourly MapReduce job with 100 Mappers @ 1GB/container (for a total of 100GB)
# All mappers fail first attempt
# All 100 mappers are retried with 2 GB containers
# Job completes successfully, still well within required time limits for the
job.
# Because the job did not fail and was not late, nobody complains.
In this scenario, we waste 100 GB of cluster capacity for some period of time
every hour. Possibly more, if the maps really only needed an additional 512MB,
for example. Ideally, the user responsible for this job would notice the
large number of map failures, and follow-up, but this does not always happen in
a timely fashion. If the job fails the first time it goes over its memory
limit, the problem will more likely be addressed sooner and avoid wasting
cluster resources.
{quote}I could see a new configuration called growth-factor which specifies how
large to grow the container each time it is re-tried. This would be a
percentage, therefore, a growth-factor of 1.0f (100%) would preserve the
current behavior.
{quote}
I think something like this would work.
> Relaunching Failed Containers
> -----------------------------
>
> Key: MAPREDUCE-7180
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7180
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv1, mrv2
> Reporter: BELUGA BEHR
> Priority: Major
>
> In my experience, it is very common that a MR job completely fails because a
> single Mapper/Reducer container is using more memory than has been reserved
> in YARN. The following message is logging the the MapReduce
> ApplicationMaster:
> {code}
> Container [pid=46028,containerID=container_e54_1435155934213_16721_01_003666]
> is running beyond physical memory limits.
> Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual
> memory used. Killing container.
> {code}
> In this case, the container is re-launched on another node, and of course, it
> is killed again for the same reason. This process happens three (maybe
> four?) times before the entire MapReduce job fails. It's often said that the
> definition of insanity is doing the same thing over and over and expecting
> different results.
> For all intents and purposes, the amount of resources requested by Mappers
> and Reducers is a fixed amount; based on the default configuration values.
> Users can set the memory on a per-job basis, but it's a pain, not exact, and
> requires intimate knowledge of the MapReduce framework and its memory usage
> patterns.
> I propose that if the MR ApplicationMaster detects that a container is killed
> because of this specific memory resource constraint, that it requests a
> larger container for the subsequent task attempt.
> For example, increase the requested memory size by 50% each time the
> container fails and the task is retried. This will prevent many Job failures
> and allow for additional memory tuning, per-Job, after the fact, to get
> better performance (v.s. fail/succeed).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
