Steve Loughran commented on HADOOP-15183:

Patch 003.

Trying to make the s3guard inconsistency visible in a unit test. Failing. Will 
need to either go for 1K+ test files, or allow the page size for delete 
requests to be configured (best).

For setting the page size, two options
# Add a secret fs.s3a.x-test.delete.page.size. Simple, may get used, may need 
to be maintained.
# make it a config option of the inconsistent FS client, so iff you switch to 
the inconsistent client do you get to turn it on.

#2 is a bit more convoluted, but I like it. Initially though I'll do the secret 
config option

> S3Guard store becomes inconsistent after partial failure of rename
> ------------------------------------------------------------------
>                 Key: HADOOP-15183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15183
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Blocker
>         Attachments: HADOOP-15183-001.patch, HADOOP-15183-002.patch, 
> org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
> If an S3A rename() operation fails partway through, such as when the user 
> doesn't have permissions to delete the source files after copying to the 
> destination, then the s3guard view of the world ends up inconsistent. In 
> particular the sequence
>  (assuming src/file* is a list of files file1...file10 and read only to 
> caller)
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the 
> delete, dest/file1 will exist
> # delete file dest/file1
> # rename src/file* dest/  ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a 
> tombstone marker and the update at the end of the rename() didn't take place: 
> the old data is still there.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to