[
https://issues.apache.org/jira/browse/HADOOP-16090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761089#comment-16761089
]
Dmitri Chmelev commented on HADOOP-16090:
-----------------------------------------
{quote}So do it when createFile() is kicked off? Possibly, though as the PUT
doesn't manifest data until the write completes, that's not ideal. But I'm
thinking about what we could maybe do there, especially given we do have a pool
of threads to play with.
{quote}
Not just createFile(). I was thinking of adding a pre-check in the paths that
can end up creating objects in S3 (create, close, mkdir, etc.) to call
getFileStatus() on the parent directory before initiating the rest of the
operation. If the result is either a fake dir or nothing is found, then we know
that we are "likely" about to create the first dirent if the operation
succeeds. We can then cache the hint as part of the operation context and
conditionally invoke the cleanup callback if the hint is present. Note that
false positives are fine here from the point of view of correctness, because
they will lead to multiple attempts to cleanup.
> deleteUnnecessaryFakeDirectories() creates unnecessary delete markers in a
> versioned S3 bucket
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-16090
> URL: https://issues.apache.org/jira/browse/HADOOP-16090
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.1
> Reporter: Dmitri Chmelev
> Priority: Minor
>
> The fix to avoid calls to getFileStatus() for each path component in
> deleteUnnecessaryFakeDirectories() (HADOOP-13164) results in accumulation of
> delete markers in versioned S3 buckets. The above patch replaced
> getFileStatus() checks with a single batch delete request formed by
> generating all ancestor keys formed from a given path. Since the delete
> request is not checking for existence of fake directories, it will create a
> delete marker for every path component that did not exist (or was previously
> deleted). Note that issuing a DELETE request without specifying a version ID
> will always create a new delete marker, even if one already exists ([AWS S3
> Developer
> Guide|https://docs.aws.amazon.com/AmazonS3/latest/dev/RemDelMarker.html])
> Since deleteUnnecessaryFakeDirectories() is called as a callback on
> successful writes and on renames, delete markers accumulate rather quickly
> and their rate of accumulation is inversely proportional to the depth of the
> path. In other words, directories closer to the root will have more delete
> markers than the leaves.
> This behavior negatively impacts performance of getFileStatus() operation
> when it has to issue listObjects() request (especially v1) as the delete
> markers have to be examined when the request searches for first current
> non-deleted version of an object following a given prefix.
> I did a quick comparison against 3.x and the issue is still present:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2947|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2947]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
