[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692120#comment-17692120
]
ASF GitHub Bot commented on HADOOP-18636:
-----------------------------------------
steveloughran commented on PR #5412:
URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1439805659
to completely clear: that temp dir creation already happens on the first
invocation of getLocalPathForWrite, this just adds it to every call (this is
cheap if the path exists()).
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
> Affects Versions: 3.3.4
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will
> again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it
> is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So
> they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
