[ https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944143#comment-15944143 ]
Mingliang Liu commented on HADOOP-14236: ---------------------------------------- Tested against us-west-1. {code} $ mvn -Dit.test='ITestS3A*,ITestS3Guard*,ITestDynamo*' -Dtest=none -Dscale -Ds3guard -Ddynamo -q clean verify Results : Tests run: 357, Failures: 0, Errors: 0, Skipped: 16 {code} [~fabbri] I'll hold on commit in 3 days for your review. > S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries > in metadata store > ----------------------------------------------------------------------------------------------- > > Key: HADOOP-14236 > URL: https://issues.apache.org/jira/browse/HADOOP-14236 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Reporter: Mingliang Liu > Assignee: Mingliang Liu > Priority: Critical > Attachments: HADOOP-14236-HADOOP-13345.000.patch, > HADOOP-14236-HADOOP-13345.001.patch > > > After running integration test {{ITestS3AFileSystemContract}}, I found the > following items are not cleaned up in DynamoDB: > {code} > parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir, > child=subdir > parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir, > child=file2 > {code} > At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we > need to be careful when cleaning up test data. > Then I found it’s a bug(?) in the code of integrating S3Guard with > S3AFileSystem: for rename we miss sub-directory items to put (dest) and > delete (src). The reason is that in S3A, we delete those fake directory > objects if they are not necessary, e.g. non-empty. So when we list the > objects to rename, the object summaries will only return _file_ objects. This > has two consequences after rename: > # there will be left items for src path in metadata store - left-overs will > confuse {{get(Path)}} which should return null > # we are not persisting the whole subtree for dest path to metadata store - > this will break the DynamoDBMetadataStore invariant: _if a path exists, all > its ancestors will also exist in the table_. > Existing tests are not complaining about this though. If this is a real bug, > let’s address it here. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org