[
https://issues.apache.org/jira/browse/HADOOP-6963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255122#comment-13255122
]
Ravi Prakash commented on HADOOP-6963:
--------------------------------------
I just wanted to come back and say this was a very peculiar case too.
- The cycle has to be nested within another cycle. Inside the inner cycle an
ELOOP is thrown by the kernel, which (I don't understand how) causes it to
abandon that branch and come back to the outer loop. This probably also clears
the stack, so that the JVM stack doesn't exhaust the memory either. (In our
case, the process had been running for 9 days without getting an OOME).
> Fix FileUtil.getDU. It should not include the size of the directory or follow
> symbolic links
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-6963
> URL: https://issues.apache.org/jira/browse/HADOOP-6963
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.205.0, 0.23.1
> Reporter: Owen O'Malley
> Assignee: Ravi Prakash
> Priority: Critical
> Fix For: 1.0.3, 0.23.3
>
> Attachments: HADOOP-6963.branch-1.0.2.patch,
> HADOOP-6963.branch-1.0.2.patch, HADOOP-6963.branch-1.patch,
> HADOOP-6963.branch-23.patch, HADOOP-6963.branch-23.patch,
> HADOOP-6963.branch-23.patch
>
>
> The getDU method should not include the size of the directory. The Java
> interface says that the value is undefined and in Linux/Sun it gets the 4096
> for the inode. Clearly this isn't useful.
> It also recursively calls itself. In case the directory has a symbolic link
> forming a cycle, getDU keeps spinning in the cycle. In our case, we saw this
> in the org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects
> call. This prevented other tasks on the same node from committing, causing
> the TT to become effectively useless (because the JT thinks it already has
> enough tasks running)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
