[
https://issues.apache.org/jira/browse/HADOOP-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188423#comment-17188423
]
Steve Loughran commented on HADOOP-13230:
-----------------------------------------
bq. S3 does not have directories at all. So you cannot have an 'empty
directory' in s3. In order to signify an 'empty directory' in S3 an object is
created with a trailing / character to indicate an empty directory. It's more
like a 'file' than a 'directory', it's just a file that has the '/' character
as it's last character. Once ANY object is added to this 'directory' a list
operation via the S3 api will find that object when searching at that path.
Once that state is reached the 'emptyDir' object is no longer necessary, and
deleted.
Correct. But we add a complication that we blindly issue the DELETE calls all
the way up the path list as it is faster than probing ... but this generates
(throttled) IO load especially on deep hive trees and adds tombstones on
versioned buckets (So slows down subsequent listings)
bq. I do not know if there any implications (with the S3 API) with leaving the
emptyDir object in place though. If there are not, it seems the easiest
solution here would be to just leave the object there until deletion of the
folder. Such that the emptyDir object becomes optional.
We have added a CLI tool to audit/purge markers, and we make sure that on a
rename none are copied. Not much else we can do.
FWIW, prestp deals with this by always having list(path) perform LIST; return
an empty list if there are no children. There are no empty directories, there
are no paths which don't "exist". So mkdirs is a no-op. They can get away with
this as they know exactly how their query engine works.
> S3A to optionally retain directory markers
> ------------------------------------------
>
> Key: HADOOP-13230
> URL: https://issues.apache.org/jira/browse/HADOOP-13230
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Aaron Fabbri
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Attachments: 2020-02-Fixing the S3A directory marker problem.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Users of s3a may not realize that, in some cases, it does not interoperate
> well with other s3 tools, such as the AWS CLI. (See HIVE-13778, IMPALA-3558).
> Specifically, if a user:
> - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path
> - Copies data into that directory via another tool, i.e. aws cli.
> - Tries to access the data in that directory with any Hadoop software.
> Then the last step fails because the fake empty directory blob that s3a wrote
> in the first step, causes s3a (listStatus() etc.) to continue to treat that
> directory as empty, even though the second step was supposed to populate the
> directory with data.
> I wanted to document this fact for users. We may mark this as not-fix, "by
> design".. May also be interesting to brainstorm solutions and/or a config
> option to change the behavior if folks care.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
