[jira] [Commented] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196017#comment-17196017
]
Steve Loughran commented on HADOOP-17244:
-
new PR coming here soon. delete needs work.
Partial failures of delete turns out to be the real troublespot as its fairly
deep in the code that we split the failure into deleted/undeleted and add a
tombstone in DDB For all the deleted entries. Now we have to identify which are
directories (has a / at the end), delete those which have no children, retain
those which do -which means a metastore.get(need-empty-dirctory=true) lookup.
Which also implies we need to sort the list of deleted entries to remove
children first.
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192171#comment-17192171
]
Steve Loughran commented on HADOOP-17244:
-
This is only an issue on branches with HADOOP-16430
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192080#comment-17192080
]
Steve Loughran commented on HADOOP-17244:
-
.. this fix needs to go in to all releases of Hadoop which do incremental
s3guard updates during delete. I think that is 3.3+ only.
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
