[ 
https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15183:
------------------------------------
    Status: Patch Available  (was: Open)

Testing, s3A ireland.

ITestAssumeRole does fail in rename with s3guard turned on, which is what I 
expect: the tombstone marker from the first delete is causing confusion.

But the enhanced delete test isn't being picked up yet, even though the 
multidelete exception is being thrown before the delete is being picked up, and 
all the successfully deleted files are no longer there. the metadata shouldn't 
have been updated yet.

{code}
[ERROR]   
ITestAssumeRole.testRestrictedRenameReadOnlyData:499->executeRenameReadOnlyData:583->assertFileCount:864->Assert.fail:88
 files copied to the destination: expected 11 files in 
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest but 
got 10
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-1
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-10
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-2
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-3
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-4
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-5
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-6
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-7
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-8
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlyData/renameDest/file-9
[ERROR]   
ITestAssumeRole.testRestrictedRenameReadOnlySingleDelete:507->executeRenameReadOnlyData:583->assertFileCount:864->Assert.fail:88
 files copied to the destination: expected 11 files in 
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest
 but got 10
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-1
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-10
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-2
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-3
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-4
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-5
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-6
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-7
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-8
s3a://hwdev-steve-london/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-9
{code}

> S3Guard store becomes inconsistent after partial failure of rename
> ------------------------------------------------------------------
>
>                 Key: HADOOP-15183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15183
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Blocker
>         Attachments: HADOOP-15183-001.patch, HADOOP-15183-002.patch, 
> org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
>
>
> If an S3A rename() operation fails partway through, such as when the user 
> doesn't have permissions to delete the source files after copying to the 
> destination, then the s3guard view of the world ends up inconsistent. In 
> particular the sequence
>  (assuming src/file* is a list of files file1...file10 and read only to 
> caller)
>    
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the 
> delete, dest/file1 will exist
> # delete file dest/file1
> # rename src/file* dest/  ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a 
> tombstone marker and the update at the end of the rename() didn't take place: 
> the old data is still there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to