[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905342#comment-15905342
]
Thomas Demoor commented on HADOOP-13786:
----------------------------------------
Hi Ryan, thanks for contributing to this effort.
Both approaches have the same concept of having data persistence but postponing
object visibility through holding back on the multipart commit. +1 on sharing
common code and having 2 committers.
Thoughts after my first pass, on 009.patch:
StagingCommitter:
* I can see how this is ideal in a pre s3ablockoutputstream world. Using the
stage to disk allows code which is conceptually similar to the exisiting
filoutputcommitter
* +: well tested and lots of failure handling (crash during overwrite
protection.)
* +: flexible, we could still use s3a as the wrappedCommiter for s3guard /
consistent s3 implementations and thus not require a hdfs cluster to be around
* -: s3client in committer itself (another threadpool, will they fight? :P)
* -: staging to local disk not essentail and can cause issues (disk out of
space, disk throughput limitation = network is faster, large files ...) = one
of the reasons we made the blockoutputstreams
MagicCommiter:
* +: less moving parts (no local disk, no second committer for the metadata)
* +: s3 complexity solved by reusing s3ablockoutputstream
* -: less mileage / testing: if things break where will there be garbage?
* Saw quite some unused variables, methods, etc. but I considered this work in
progress so /ignore
* There are some extra HEAD requests to verify that things really happened
which I think are superfluous. For instance,
org/apache/hadoop/fs/s3a/commit/FileCommitActions.java:78 the HEAD after PUT.
Is this to get around eventual consistency? This is head after put, so should
be consistent even on aws. Maybe useful for PUT overwrite but then you also
need to check the ETag. I think this HEAD should be optional behind some config
flag.
Varia:
* Don't think the default (hdfs) workflow has crash-during-jobcommit protection
(fileoutputcommitter algo v2 already makes files visible in taskcommit afaik) ,
so not sure it is required for our committers.
> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
> ------------------------------------------------------------------------
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch,
> HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch,
> HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch,
> HADOOP-13786-HADOOP-13345-010.patch, s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
