[
https://issues.apache.org/jira/browse/MAPREDUCE-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833670#action_12833670
]
Matei Zaharia commented on MAPREDUCE-1436:
------------------------------------------
I'm also reposting the second deadlock with better formatting in case my
previous post was hard to read:
{code}
Found one Java-level deadlock:
=============================
"IPC Server handler 24 on 9001":
waiting to lock monitor 0x0000000040c91750 (object 0x00007fc0243e2c20, a
org.apache.hadoop.mapred.JobTracker),
which is held by "IPC Server handler 0 on 9001"
"IPC Server handler 0 on 9001":
waiting to lock monitor 0x0000000040bc0770 (object 0x00007fc0243e3080, a
org.apache.hadoop.mapred.FairScheduler),
which is held by "FairScheduler update thread"
"FairScheduler update thread":
waiting to lock monitor 0x000000004095dd98 (object 0x00007fc0258bc0d0, a
org.apache.hadoop.mapred.JobInProgress),
which is held by "IPC Server handler 0 on 9001"
Java stack information for the threads listed above:
===================================================
"IPC Server handler 24 on 9001":
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2487)
- waiting to lock <0x00007fc0243e2c20> (a
org.apache.hadoop.mapred.JobTracker)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
"IPC Server handler 0 on 9001":
at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2115)
- waiting to lock <0x00007fc0243e3080> (a
org.apache.hadoop.mapred.FairScheduler)
- locked <0x00007fc0243e3420> (a java.util.TreeMap)
- locked <0x00007fc0243e2c20> (a org.apache.hadoop.mapred.JobTracker)
at
org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2510)
- locked <0x00007fc0258bc0d0> (a org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2146)
at
org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2084)
- locked <0x00007fc0258bc0d0> (a org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:883)
- locked <0x00007fc0258bc0d0> (a org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3564)
at
org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:2758)
- locked <0x00007fc0243e2c20> (a org.apache.hadoop.mapred.JobTracker)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2553)
- locked <0x00007fc0243e2c20> (a org.apache.hadoop.mapred.JobTracker)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
"FairScheduler update thread":
at
org.apache.hadoop.mapred.JobInProgress.scheduleReduces(JobInProgress.java:1203)
- waiting to lock <0x00007fc0258bc0d0> (a
org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobSchedulable.updateDemand(JobSchedulable.java:53)
at
org.apache.hadoop.mapred.PoolSchedulable.updateDemand(PoolSchedulable.java:81)
at org.apache.hadoop.mapred.FairScheduler.update(FairScheduler.java:577)
- locked <0x00007fc0243e3080> (a org.apache.hadoop.mapred.FairScheduler)
at
org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:277)
Found 1 deadlock.
{code}
A review of the new patch would be appreciated!
> Deadlock in preemption code in fair scheduler
> ---------------------------------------------
>
> Key: MAPREDUCE-1436
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1436
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.21.0, 0.22.0
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Priority: Blocker
> Attachments: deadlock.png, mapreduce-1436-v2.patch,
> mapreduce-1436.patch
>
>
> In testing the fair scheduler with preemption, I found a deadlock between
> updatePreemptionVariables and some code in the JobTracker. This was found
> while testing a backport of the fair scheduler to Hadoop 0.20, but it looks
> like it could also happen in trunk and 0.21. Details are in a comment below.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
