[
https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354870#comment-16354870
]
Aaron Fabbri commented on HADOOP-15183:
---------------------------------------
Looking at rename path for HADOOP-13761. Rename is already fairly robust to
these sorts of things just because the MetadataStore is updated *after* the
rename succeeds in S3. In this case, I'm guessing:
* Step 2 adds a delete marker to MetadataStore (tombstone in dynamo table)
* Step 3 does *not* update the MetadataStore at all: since the delete() fails,
rename() bails out before this happens.
Basically, rename() updates MetadataStore as all-or-nothing: If any parts of
rename failed, no update. If it all succeeds only then does MetadataStore get
updated.
First glance, seems like we should make the logic smarter so it actually
reports what changed to MetadataStore. In your Step 3, the failed delete path
should be omitted from the {{pathsToDelete}} argument to
{{MetadataStore#move()}}
> S3Guard store becomes inconsistent after partial failure of rename
> ------------------------------------------------------------------
>
> Key: HADOOP-15183
> URL: https://issues.apache.org/jira/browse/HADOOP-15183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Priority: Major
> Attachments: org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
>
>
> If an S3A rename() operation fails partway through, such as when the user
> doesn't have permissions to delete the source files after copying to the
> destination, then the s3guard view of the world ends up inconsistent. In
> particular the sequence
> (assuming src/file* is a list of files file1...file10 and read only to
> caller)
>
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the
> delete, dest/file1 will exist
> # delete file dest/file1
> # rename src/file* dest/ ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a
> tombstone marker and the update at the end of the rename() didn't take place:
> the old data is still there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
