Maciej Arciuch created HDFS-7272:
------------------------------------
Summary: Concurrency problem in Dfsck/URLConnectionFactory classes
Key: HDFS-7272
URL: https://issues.apache.org/jira/browse/HDFS-7272
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Maciej Arciuch
Priority: Minor
Hi,
I think I've found a concurrency issue in the URLConnectionFactory class used
by the Dfsck class.
The problem is that when multiple instances of new Dfsck run in the same JVM,
401 errors occur. I know that this class in usually run in different
command-line processes, but if I call programatically Dfsck.doWork() on
multiple instances in parallel I get strange, random auth errors. Earlier
versions worked fine.
So, I took a look at the code and that's I found:
In the earlier versions of the Dfsck the connection was instatiated using
SecurityUtil.openSecureHttpConnection(path), which worked just fine. New
versions use URLConnectionFactory.
Quick links to grepcode:
old, correct code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.0.0-cdh4.2.1/org/apache/hadoop/hdfs/tools/DFSck.java/#161
new code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.3.0-cdh5.1.2/org/apache/hadoop/hdfs/tools/DFSck.java/#174
The difference is that the old SecurityUtil always passed null Authenticator to
the AuthenticatedURL constructor, which, if case of null argument, instantiated
a new Authenticator instance using the default instantiator class. *So, there
was new Authenticator instance created for each call. The URLConnectorFactory
passes a non-null, static Authenticator, which is stateful and not thread-safe.*
old code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-common/2.0.0-cdh4.2.1/org/apache/hadoop/security/SecurityUtil.java#508
new code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.3.0-cdh5.1.2/org/apache/hadoop/hdfs/web/URLConnectionFactory.java#164
Can anyone confirm and perhaps fix this? Best regards, Maciej
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
