[
https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610185#comment-15610185
]
Lei (Eddy) Xu commented on HADOOP-13651:
----------------------------------------
Hi, [~liuml07] and [~fabbri]
[~fabbri] and I had an offline discussion today, the followings are my notes
* We agree that the atomic of creation / deletion / rename (between metadata
store and s3) are difficult to achieve, as S3 is always visible to some of the
clients. We need more thoughts on these issues. My option is making sure
namespace consistency first, but there are other understandable concerns.
* For {{rename}}, it should not expect that {{rename(recursive=true)}} can be
atomic.
* {{LocalMetadataStore}} should not use {{LRU/MRU}}, which can not provide
consistency due to that the metadata will be evicted.
* The current test suites (unit tests and integration tests) can not
reliability test eventual consistency. New tests cases are necessary before
merging this branch to branch.
* This patch is large and many of the following works are depended to this
patch. My CLI patch (HADOOP-13650) and DynamoDB metadata store (HADOOP-13449)
for instance.
As this is a feature branch, I'd give a pending +1 once these concerns are
appropriately documented (i.e., adding {{TODO}}). Thanks.
> S3Guard: S3AFileSystem Integration with MetadataStore
> -----------------------------------------------------
>
> Key: HADOOP-13651
> URL: https://issues.apache.org/jira/browse/HADOOP-13651
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Attachments: HADOOP-13651-HADOOP-13345.001.patch,
> HADOOP-13651-HADOOP-13345.002.patch
>
>
> Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata
> consistency and caching.
> Implementation should have minimal overhead when no MetadataStore is
> configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
