[
https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373049#comment-16373049
]
Steve Loughran commented on HADOOP-15183:
-----------------------------------------
Patch 003.
Trying to make the s3guard inconsistency visible in a unit test. Failing. Will
need to either go for 1K+ test files, or allow the page size for delete
requests to be configured (best).
For setting the page size, two options
# Add a secret fs.s3a.x-test.delete.page.size. Simple, may get used, may need
to be maintained.
# make it a config option of the inconsistent FS client, so iff you switch to
the inconsistent client do you get to turn it on.
#2 is a bit more convoluted, but I like it. Initially though I'll do the secret
config option
> S3Guard store becomes inconsistent after partial failure of rename
> ------------------------------------------------------------------
>
> Key: HADOOP-15183
> URL: https://issues.apache.org/jira/browse/HADOOP-15183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HADOOP-15183-001.patch, HADOOP-15183-002.patch,
> org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
>
>
> If an S3A rename() operation fails partway through, such as when the user
> doesn't have permissions to delete the source files after copying to the
> destination, then the s3guard view of the world ends up inconsistent. In
> particular the sequence
> (assuming src/file* is a list of files file1...file10 and read only to
> caller)
>
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the
> delete, dest/file1 will exist
> # delete file dest/file1
> # rename src/file* dest/ ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a
> tombstone marker and the update at the end of the rename() didn't take place:
> the old data is still there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
