[
https://issues.apache.org/jira/browse/HADOOP-13230?focusedWorklogId=497205&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497205
]
ASF GitHub Bot logged work on HADOOP-13230:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 08/Oct/20 09:40
Start Date: 08/Oct/20 09:40
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2149:
URL: https://github.com/apache/hadoop/pull/2149#issuecomment-705455586
@liuml07 Looking forward to a Hadoop 2 backport!
I'll do a hadoop-2 fixup of list time changes, but if you look #2310 delete
gets complex, especially handling partial failures of deletes. I'm going to
have to say that any branch without that (major) change will be able to read
data in a path which keeps markers, but it must not delete things -otherwise
S3Guard can get out of sync.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 497205)
Time Spent: 0.5h (was: 20m)
> S3A to optionally retain directory markers
> ------------------------------------------
>
> Key: HADOOP-13230
> URL: https://issues.apache.org/jira/browse/HADOOP-13230
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Aaron Fabbri
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Attachments: 2020-02-Fixing the S3A directory marker problem.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Users of s3a may not realize that, in some cases, it does not interoperate
> well with other s3 tools, such as the AWS CLI. (See HIVE-13778, IMPALA-3558).
> Specifically, if a user:
> - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path
> - Copies data into that directory via another tool, i.e. aws cli.
> - Tries to access the data in that directory with any Hadoop software.
> Then the last step fails because the fake empty directory blob that s3a wrote
> in the first step, causes s3a (listStatus() etc.) to continue to treat that
> directory as empty, even though the second step was supposed to populate the
> directory with data.
> I wanted to document this fact for users. We may mark this as not-fix, "by
> design".. May also be interesting to brainstorm solutions and/or a config
> option to change the behavior if folks care.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
