[
https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355804#comment-16355804
]
Steve Loughran commented on HADOOP-15183:
-----------------------------------------
Let me review that other patch this am.
I think we can handle rename failures by getting that list of failed-to-rename
entries & removing them from the latest batch, update s3guard with everything
that has gone over already (does this imply an ongoing list?) and then rethrow
* HADOOP-15176 has a test case which triggers the problem. I'll cut that out
the current patch so that it can go in without it, and the failing test suite
become part of this patch
* this may be time to add a bulk metastore delete(list<Path>) operation
(HADOOP-15193), as long as we are happy we have a good failure story there too
There's always the option for doing the updates incrementally in rename(),
which would potentially make more effective use of allocated DDB IOPs, as the
load would be spread across the copy duration. But it'd make the s3guard state
observably non-atomic
> S3Guard store becomes inconsistent after partial failure of rename
> ------------------------------------------------------------------
>
> Key: HADOOP-15183
> URL: https://issues.apache.org/jira/browse/HADOOP-15183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Priority: Major
> Attachments: org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
>
>
> If an S3A rename() operation fails partway through, such as when the user
> doesn't have permissions to delete the source files after copying to the
> destination, then the s3guard view of the world ends up inconsistent. In
> particular the sequence
> (assuming src/file* is a list of files file1...file10 and read only to
> caller)
>
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the
> delete, dest/file1 will exist
> # delete file dest/file1
> # rename src/file* dest/ ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a
> tombstone marker and the update at the end of the rename() didn't take place:
> the old data is still there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
