[
https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378372#comment-15378372
]
Aaron Fabbri commented on HADOOP-13345:
---------------------------------------
Thanks for posting [~cnauroth]. Very cool stuff. I'm interested in helping.
[~eddyxu] and I have also been working on a similar effort called S3
Consistency (S3C) and want to share it here so we can compare designs. I will
attach a design doc and patch of what we did.
Some interesting differences, from first glance:
- You are a little further along. We only started this a month or so ago,
part time. Our DynamoDB back-end is not finished. We wrote the design doc
then started prototyping. Prototyping was valuable experience. Devil is in
the details, as they say.
- Our prototype (S3C) treats performance as a primary goal. There are some
interesting tradeoffs. We demonstrated a significant performance improvement
with our "fully cached directory" concept (see the design doc), but having a
non-S3 source of truth makes failure handling trickier. Ultimately, I'd love
the policy to be configurable.
- Really interesting to compare your {{ConsistentStore}} with our
{{MetadataStore}} interface. Look forward to discussions on this, and the
"caching policy" as I call it (i.e. source of truth policy).
- Our code separation at this point needs work. We intended to refactor.
{{S3AFileSystem}} could use that in general. We wanted to keep
{{MetadataStore}} separate from {{s3a}}, so it wouldn't be to hard to pull it
out of {{s3a}} and use it for other storage connectors. We debated doing a
separate {{FileSystem}} wrapper (discussed in design doc). Also thought about
doing subclassing of {{S3AFileSystem}} as you did (might be brittle to future
change but your separation is much better). At this point, though, the code
served to make us aware of the details we need to conquer in the
{{MetadataStore}} interface as well as feasibility of using it as a source of
truth, adaptively (Policy C in design doc).
- Having a local implementation of the {{MetadataStore}} has been fun. The
{{TestLocalMetadataStore}} unit tests would probably become a general contract
test of the interface. Intention is for local unit testing, but it happens to
speed up something like {{hadoop -fs -copyFromLocal /my/dir/tree s3a://bucket}}
over 2x in my quick tests. In addition to the warning in the class comment, it
needs a log.WARN "this is not for production use" or something.. Concerned
about people turning it on for performance without realizing that it has zero
cross-node coherency.
I'm on vacation next week but wanted to send out what we have. I'm hoping we
can help you with this and combine the best of both efforts.
> S3Guard: Improved Consistency for S3A
> -------------------------------------
>
> Key: HADOOP-13345
> URL: https://issues.apache.org/jira/browse/HADOOP-13345
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13345.prototype1.patch,
> S3GuardImprovedConsistencyforS3A.pdf
>
>
> This issue proposes S3Guard, a new feature of S3A, to provide an option for a
> stronger consistency model than what is currently offered. The solution
> coordinates with a strongly consistent external store to resolve
> inconsistencies caused by the S3 eventual consistency model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
