[
https://issues.apache.org/jira/browse/MAPREDUCE-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032545#comment-14032545
]
Jason Lowe commented on MAPREDUCE-5928:
---------------------------------------
I'm pretty sure you're using the CapacityScheduler since that's been the
default in Apache Hadoop for some time now. I'm not positive about the HDP
release, but I suspect it, too, is configured to use the CapacityScheduler by
default.
After a quick examination of the AM log, it looks like a couple of things are
going on. The AM is blacklisting one of the nodes, and we can see that node
not being used in the cluster picture. There's a known issue with headroom
calculation not taking into account blacklisted nodes. See YARN-1680.
The node ends up being blacklisted because the NM shot a number of the tasks
for being over container limits. It looks like the containers are being
allocated as using 500MB but the JVM heap sizes are set to 512MB. Note that
the container size includes the size of the entire process tree for the task.
That's not just the heap, so it needs to also include thread stacks, JVM data,
JVM code, any subprocesses launched (e.g.: hadoop streaming) etc. If you
really need a 512MB heap then I'd allocate 768MB or maybe even 1024MB
containers, depending on what the tasks are doing.
It does look like some fractional memory shenanigans could be involved here, as
the picture shows most of the nodes having only 200MB free. It'd be
interesting to know if you still hit the deadlock after fixing the cause of the
blacklisting.
> Deadlock allocating containers for mappers and reducers
> -------------------------------------------------------
>
> Key: MAPREDUCE-5928
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5928
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: Hadoop 2.4.0 (as packaged by HortonWorks in HDP 2.1.2)
> Reporter: Niels Basjes
> Attachments: AM-MR-syslog - Cleaned.txt.gz, Cluster fully
> loaded.png.jpg, MR job stuck in deadlock.png.jpg
>
>
> I have a small cluster consisting of 8 desktop class systems (1 master + 7
> workers).
> Due to the small memory of these systems I configured yarn as follows:
> {quote}
> yarn.nodemanager.resource.memory-mb = 2200
> yarn.scheduler.minimum-allocation-mb = 250
> {quote}
> On my client I did
> {quote}
> mapreduce.map.memory.mb = 512
> mapreduce.reduce.memory.mb = 512
> {quote}
> Now I run a job with 27 mappers and 32 reducers.
> After a while I saw this deadlock occur:
> - All nodes had been filled to their maximum capacity with reducers.
> - 1 Mapper was waiting for a container slot to start in.
> I tried killing reducer attempts but that didn't help (new reducer attempts
> simply took the existing container).
> *Workaround*:
> I set this value from my job. The default value is 0.05 (= 5%)
> {quote}
> mapreduce.job.reduce.slowstart.completedmaps = 0.99f
> {quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
