[
https://issues.apache.org/jira/browse/HADOOP-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594758#action_12594758
]
lohit edited comment on HADOOP-3348 at 5/6/08 8:26 PM:
------------------------------------------------------------------
After looking a bit with hints from Raghu, looks like this is what is causing
the problem.
once setUrlStreamHandle is set in JVM, opening a device file /dev/random is
opened as ChecksumFileSystem.
SecureRandom wraps up the stream returned by opening /dev/random into
BufferedInputStream and calls read to get 20 byte DIGEST to generate the seed.
This read when passed through FSInputChecker.read with a buffer size of 8K,
which used to loop until we read 8K bytes. We invoke readFully() which loops
calling multiple reads until 8K buffer is filled up. /dev/random was unable to
produce random bytes so fast and hence registration of Datanode used to take
forever.
was (Author: lohit):
After looking a bit with hints from Raghu, looks like this is what is
causing the problem.
once setUrlStreamHandle is set in JVM, opening a device file /dev/random is
readFully opened as ChecksumFileSystem.
SecureRandom wraps up the stream returned by opening /dev/random into
BufferedInputStream and invokes read to get 20 byte DIGEST to generate the
seed. This read when passed through FSInputChecker.read with a buffer size of
8K used to loop until we read 8K bytes. We invoke readFully() which loops
calling multiple reads until 8K buffer is filled up. /dev/random was unable to
produce random bytes so fast and hence registration of Datanode used to take
forever.
> TestUrlStreamHandler hangs on LINUX
> -----------------------------------
>
> Key: HADOOP-3348
> URL: https://issues.apache.org/jira/browse/HADOOP-3348
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0
> Environment: LINUX 2.6.9
> Reporter: lohit vijayarenu
> Attachments: Datanode_jstack.txt
>
>
> TestUrlStreamHandler sets setURLStreamHandlerFactory as
> {noformat}
> FsUrlStreamHandlerFactory factory =
> new org.apache.hadoop.fs.FsUrlStreamHandlerFactory();
> java.net.URL.setURLStreamHandlerFactory(factory);
> {noformat}
> After this, MiniDFSCluster seems to hang while Datanodes tries to register in
> setNewStorageID, specifically at
> {noformat}
> rand = SecureRandom.getInstance("SHA1PRNG").nextInt(Integer.MAX_VALUE);
> {noformat}
> jstack output shows that the main thread is stuck in
> RawLocalFileSystem$LocalFSFileInputStream.read
> (Attaching the jstack)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
