ShellCommand, in particular 'df -k', sometimes hang
---------------------------------------------------
Key: HADOOP-2231
URL: https://issues.apache.org/jira/browse/HADOOP-2231
Project: Hadoop
Issue Type: Bug
Components: fs
Affects Versions: 0.15.1
Reporter: Christian Kunz
We noticed that some pipes applications writing to dfs using libhdfs have about
6% chance of hanging when executing 'df -k' to find out whether there is enough
space available on the local filesystem before opening a file for write.
Why not using File.getFreeSpace() or File.GetUsableSpace()?
The call stack is:
Exception in thread "main" java.io.IOException
at org.apache.hadoop.fs.ShellCommand.runCommand
(ShellCommand.java:52)
at org.apache.hadoop.fs.ShellCommand.run(ShellCommand.java:42)
at org.apache.hadoop.fs.DF.getAvailable(DF.java:72)
at org.apache.hadoop.fs.LocalDirAllocator
$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:264)
at org.apache.hadoop.fs.LocalDirAllocator
$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:294)
at
org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite
(LocalDirAllocator.java:155)
at org.apache.hadoop.dfs.DFSClient
$DFSOutputStream.newBackupFile(DFSClient.java:1470)
at org.apache.hadoop.dfs.DFSClient
$DFSOutputStream.openBackupStream(DFSClient.java:1437)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk
(DFSClient.java:1579)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk
(FSOutputSummer.java:140)
at org.apache.hadoop.fs.FSOutputSummer.write1
(FSOutputSummer.java:100)
at org.apache.hadoop.fs.FSOutputSummer.write
(FSOutputSummer.java:86)
at org.apache.hadoop.fs.FSDataOutputStream
$PositionCache.write(FSDataOutputStream.java:39)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
