[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17690389#comment-17690389
]
Daniel Carl Jones commented on HADOOP-18636:
--------------------------------------------
{quote}when you ask actually for a file, it will look for the parent dir, but
it calls {{{}mkdir(){}}}, rather than {{mkdirs()}}
{quote}
This is the suspect line in question?
[https://github.com/apache/hadoop/blob/7e19bc31b65f86be91451a0ec7590023b6b57a12/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java#L190]
Sneaky one.
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
> Affects Versions: 3.3.4
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, but it
> calls {{mkdir()}}, rather than {{mkdirs()}}
> This means it will recreate a missing parent file but cannot recover from a
> missing grandparent. If during the life of an application the temp directory
> is cleaned up, it can result in the failure of the application.
> Fix add an "s" to the right place in the production code, plus a new test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
