[
https://issues.apache.org/jira/browse/HADOOP-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HADOOP-6269:
--------------------------------
Status: Open (was: Patch Available)
Under some more testing this patch ends up occasionally producing a deadlock:
{noformat}
Thread 30 (pool-2-thread-1):
State: BLOCKED
Blocked count: 4
Waited count: 0
Blocked on org.apache.hadoop.mapred.jobc...@30e3c624
Blocked by 1 (main)
Stack:
org.apache.hadoop.conf.Configuration.reloadConfiguration(Configuration.java:326)
<-- synch on object
org.apache.hadoop.conf.Configuration.addDefaultResource(Configuration.java:257)
<-- synch on class
org.apache.hadoop.thriftfs.ThriftPluginServer.<clinit>(ThriftPluginServer.java:64)
org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin.start(ThriftJobTrackerPlugin.java:415)
org.apache.hadoop.util.PluginDispatcher$2.run(PluginDispatcher.java:104)
org.apache.hadoop.util.PluginDispatcher$2.run(PluginDispatcher.java:101)
org.apache.hadoop.util.PluginDispatcher$1.run(PluginDispatcher.java:84)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
Thread 1 (main):
State: BLOCKED
Blocked count: 14
Waited count: 10
Blocked on java.lang.cl...@4987b287
Blocked by 30 (pool-2-thread-1)
Stack:
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1056)
<-- synch on class
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1005) <--
synch on object
org.apache.hadoop.conf.Configuration.get(Configuration.java:382)
org.apache.hadoop.conf.Configuration.getInt(Configuration.java:451)
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:180)
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:168)
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1363)
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1378)
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:192)
org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
org.apache.hadoop.mapred.JobTracker$RecoveryManager.updateRestartCount(JobTracker.java:1166)
org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:1822)
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3711)
{noformat}
So please don't commit :)
I'm now doing a more thorough audit of the synchronization in Configuration so
we can hope to solve the issue completely.
> Missing synchronization for defaultResources in Configuration.addResource
> -------------------------------------------------------------------------
>
> Key: HADOOP-6269
> URL: https://issues.apache.org/jira/browse/HADOOP-6269
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.20.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.20.2, 0.21.0
>
> Attachments: hadoop-6269.txt
>
>
> Configuration.defaultResources is a simple ArrayList. In two places in
> Configuration it is accessed without appropriate synchronization, which we've
> seen to occasionally result in ConcurrentModificationExceptions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
