[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393833#comment-15393833
]
Steve Loughran commented on HADOOP-9565:
----------------------------------------
Looking at this, very nice to see that you have done a committer.
regarding algorithm versions, maybe we should do something else: have a
specific object store committer
* strengths: isolated. Can also be run against local filesystems experimentally.
* weakness: takes a long time to get picked up downstream, unless you can
explicitly request the committer (as spark does; I don't know about other
things)
There's one more thing we can look at here for committing, which is that on an
object store write you may be able to abort the write before the close(), so
avoid any data write (and with multipart xfer, kill that specific upload). That
could help with speculative writes to the same output file: they could perhaps
both target the same destination.
> Add a Blobstore interface to add to blobstore FileSystems
> ---------------------------------------------------------
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
> Affects Versions: 2.6.0
> Reporter: Steve Loughran
> Assignee: Pieter Reuse
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
