[
https://issues.apache.org/jira/browse/HADOOP-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260510#comment-17260510
]
Steve Loughran commented on HADOOP-17403:
-----------------------------------------
In the log,
{code}
// end of write operations
2021-01-07 12:08:06,350 [s3a-transfer-shared-pool90-t62] DEBUG
s3guard.Operations (DynamoDBMetadataStore.java:logPut(2233)) - #(Rename-0545)
PUT
s3a:///stevel-london/fork-0004/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-1610021278.265/writableDir/dir-002/dir-003
directory
// failure of delete
2021-01-07 12:08:06,428 [JUnit-testRenameDirFailsInDelete[bulk-delete=true]]
WARN impl.MultiObjectDeleteSupport
(MultiObjectDeleteSupport.java:translateDeleteException(92)) - Bulk delete
operation failed to delete all objects; failure count = 69
2021-01-07 12:08:06,429 [JUnit-testRenameDirFailsInDelete[bulk-delete=true]]
WARN impl.MultiObjectDeleteSupport
(MultiObjectDeleteSupport.java:translateDeleteException(107)) - AccessDenied:
fork-0004/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-1610021278.265/readonlyDir/dir-000/file-008:
Access Denied
// there's no dir-001/ in the error list, only the children
2021-01-07 12:08:06,429 [JUnit-testRenameDirFailsInDelete[bulk-delete=true]]
WARN impl.MultiObjectDeleteSupport
(MultiObjectDeleteSupport.java:translateDeleteException(107)) - AccessDenied:
fork-0004/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-1610021278.265/readonlyDir/dir-001/dir-001/:
Access Denied
// but there are some of its peers, e.g. dir-002/
2021-01-07 12:08:06,442 [JUnit-testRenameDirFailsInDelete[bulk-delete=true]]
WARN impl.MultiObjectDeleteSupport
(MultiObjectDeleteSupport.java:translateDeleteException(107)) - AccessDenied:
fork-0004/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-1610021278.265/readonlyDir/dir-002/:
Access Denied
// then the duration log shows the rename finished
2021-01-07 12:08:06,446 [JUnit-testRenameDirFailsInDelete[bulk-delete=true]]
INFO impl.ITestPartialRenamesDeletes (DurationInfo.java:close(98)) -
rename(s3a://stevel-london/fork-0004/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-1610021278.265/readonlyDir,
s3a://stevel-london/fork-0004/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-1610021278.265/writableDir):
duration 0:02.921s
... then the tombstones go in for all files in a separate thread, i
2021-01-07 12:08:10,853 [s3a-transfer-shared-pool88-t52] DEBUG
s3guard.Operations (DynamoDBMetadataStore.java:logPut(2233)) - #(Delete-0547)
TOMBSTONE
s3a:///stevel-london/fork-0004/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-1610021278.265/readonlyDir/dir-000/file-000
... including, ultimately, dir-001
2021-01-07 12:08:11,185 [s3a-transfer-shared-pool88-t31] DEBUG
s3guard.Operations (DynamoDBMetadataStore.java:logPut(2233)) - #(Delete-0547)
TOMBSTONE
s3a:///stevel-london/fork-0004/test/testRenameDirFailsInDelete[bulk-delete=true]-multi-1610021278.265/readonlyDir/dir-001
2021-01-07 12:08:11,219 [s3a-transfer-shared-pool88-t27] DEBUG
s3guard.Operations (DynamoDBMetadataStore.java:logPut(2233))
... which means its happening in the delete() operations happening afterwards
{code}
And with only 70 objects to delete, this isn't a page size issue.
h3. Analysis.
We can see from the logs that delete of dir-001/ was not denied from a delete,
therefore it was not included in the list of objects to delete. Which can
happen if
* page size limitations (no)
* object not found in a listing
I'm going with object not found: it's not there. Either it didn't get created
or it was deleted during operations.
> S3A ITestPartialRenamesDeletes.testRenameDirFailsInDelete failure: missing
> directory marker
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-17403
> URL: https://issues.apache.org/jira/browse/HADOOP-17403
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> Seemingly transient failure of the test ITestPartialRenamesDeletes with the
> latest HADOOP-17244 changes in: an expected directory marker was not found.
> Test run was (unintentionally) sequential, markers=delete, s3guard on
> {code}
> -Dmarkers=delete -Ds3guard -Ddynamo -Dscale
> {code}
> Hasn't come back since.
> The bucket's retention policy was authoritative, but no dirs were declared as
> such
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
