[
https://issues.apache.org/jira/browse/HADOOP-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16380:
------------------------------------
Description:
If S3AFileSystem does an S3 LIST restricted to a single object to see if a
directory is empty, and the single entry found has a tombstone marker (either
from an inconsistent DDB Table or from an eventually consistent LIST) then it
will consider the directory empty, _even if there is 1+ entry which is not
deleted_
We need to make sure the calculation of whether a directory is empty or not is
resilient to this, efficiently.
It surfaces as an issue two places
* delete(path) (where it may make things worse)
* rename(src, dest), where a check is made for dest != an empty directory.
was:
I'm seeing reproducible failures of {{ITestS3AContractRootDir}} which look like
consistency problems *even when S3Guard is enabled*.
Suspicion: root dir listings are still inconsistent, due to the way we don't
keep root entries in the table
> ITestS3AContractRootDir failing on trunk: tombstones mislead about directory
> empty status
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-16380
> URL: https://issues.apache.org/jira/browse/HADOOP-16380
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
>
> If S3AFileSystem does an S3 LIST restricted to a single object to see if a
> directory is empty, and the single entry found has a tombstone marker (either
> from an inconsistent DDB Table or from an eventually consistent LIST) then it
> will consider the directory empty, _even if there is 1+ entry which is not
> deleted_
> We need to make sure the calculation of whether a directory is empty or not
> is resilient to this, efficiently.
> It surfaces as an issue two places
> * delete(path) (where it may make things worse)
> * rename(src, dest), where a check is made for dest != an empty directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
