[
https://issues.apache.org/jira/browse/HADOOP-17244?focusedWorklogId=479754&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479754
]
ASF GitHub Bot logged work on HADOOP-17244:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 07/Sep/20 18:58
Start Date: 07/Sep/20 18:58
Worklog Time Spent: 10m
Work Description: steveloughran opened a new pull request #2280:
URL: https://github.com/apache/hadoop/pull/2280
This changes directory tree deletion so that only files are incrementally
deleted
from S3Guard after the objects are deleted; the directories are left alone
until metadataStore.deleteSubtree(path) is invoked
This avoids directory tombstones being added above files/child directories,
which stop that subtree operation from actually finishing.
Also:
* callback to delete objects splits files and dirs so that
any problems deleting the dirs doesn't trigger s3guard updates
* new statistic to measure #of objects deleted, alongside request count.
* callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers
to clarify behavior.
Test enhancements in ITestDirectoryOperationCost, but I didn't manage to
replicate it there (didn't quite understand the cause); the FileContext
test works for regression testing.
Change-Id: I8f96536bc6ffaf548b3da1068b9bfc897d34a993
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 479754)
Remaining Estimate: 0h
Time Spent: 10m
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> ----------------------------------------------------------------------
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
