[
https://issues.apache.org/jira/browse/HADOOP-16090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795964#comment-16795964
]
Steve Loughran commented on HADOOP-16090:
-----------------------------------------
I'm dependent on other people's reviews, so a couple-of-days was going to be
the bare minimum anyway
* love to see what you've been doing
* assume that I'll be backporting the invoker stuff to branch-2 (but not the
current use throughout branch-3.1+)
* refactoring getFileStatus. Hmm. Popular entry point. Slow enough over
long-haul you can see the logs pausing.
I'm actually sketching out (but yet to publish) something about incremental
refactoring of the S3A code itself: the S3AFileSystem class has got over
complex.
Something like
* {{org.apache.hadoop.fs.s3a.impl.S3Core}}: S3-api level, with state coming in
as context, link to executor pool
* {{org.apache.hadoop.fs.s3a.impl.S3AModel}}: the filesystem model atop the
core: S3Guard, directory trees etc.
* {{S3AFileSytem}}: Hadoop API
Core doesn't call model; model doesn't call FS.
At the same time, no grand "break all backporting" patch.
Anyhow, assume getObjectMetadata is the lowest-level, but in the S3AModel it'd
be going through S3Guard to, in future, pick up version info from there as/when
collected. If you've not been keeping an eye on things. input streams are now
version aware too.
> S3A Client to add explicit support for versioned stores
> -------------------------------------------------------
>
> Key: HADOOP-16090
> URL: https://issues.apache.org/jira/browse/HADOOP-16090
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.1
> Reporter: Dmitri Chmelev
> Assignee: Steve Loughran
> Priority: Minor
>
> The fix to avoid calls to getFileStatus() for each path component in
> deleteUnnecessaryFakeDirectories() (HADOOP-13164) results in accumulation of
> delete markers in versioned S3 buckets. The above patch replaced
> getFileStatus() checks with a single batch delete request formed by
> generating all ancestor keys formed from a given path. Since the delete
> request is not checking for existence of fake directories, it will create a
> delete marker for every path component that did not exist (or was previously
> deleted). Note that issuing a DELETE request without specifying a version ID
> will always create a new delete marker, even if one already exists ([AWS S3
> Developer
> Guide|https://docs.aws.amazon.com/AmazonS3/latest/dev/RemDelMarker.html])
> Since deleteUnnecessaryFakeDirectories() is called as a callback on
> successful writes and on renames, delete markers accumulate rather quickly
> and their rate of accumulation is inversely proportional to the depth of the
> path. In other words, directories closer to the root will have more delete
> markers than the leaves.
> This behavior negatively impacts performance of getFileStatus() operation
> when it has to issue listObjects() request (especially v1) as the delete
> markers have to be examined when the request searches for first current
> non-deleted version of an object following a given prefix.
> I did a quick comparison against 3.x and the issue is still present:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2947|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2947]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
