[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15397825#comment-15397825
]
Chris Nauroth commented on HADOOP-9565:
---------------------------------------
I'm afraid I'm unclear about the goals here, specifically related to the
definition of Put and Copy operations.
{code}
public void Put(Path path, OutputStream source, long length, long options)
{code}
This method signature surprised me. Was the {{OutputStream}} argument meant to
be an {{InputStream}} for the implementation read?
Like Pieter, it's unclear to me that this is significant as part of the
{{ObjectStore}} interface. It looks similar to the {{FileUtil#copy}} utility
methods, not a primitive operation on {{FileSystem}} or {{ObjectStore}}. Early
comments indicate that this could have been a way for S3A to implement a direct
streaming write instead of spooling to local disk first. Now that S3A has
{{S3AFastOutputStream}}, does that meet the need? Or are you looking to
provide clients with a guaranteed non-spooling API regardless of configuration?
Is the {{options}} the significant part that I'm missing? Is the idea that the
caller passes along options to describe desired semantics, and this gives the
{{ObjectStore}} implementation the opportunity to throw
{{UnsatisfiedSemanticsException}}?
Maybe a usage example showing how you expect callers to use Put would clarify
it.
Regarding Copy, the description indicates that method could substitute for
rename. Looking at S3A as a concrete example, the rename already uses S3
server-side copy. How do you expect an S3A implementation of Copy would differ
from the current {{S3AFileSystem#rename}}? Do you expect it can relax the
notoriously complex semantics of rename, and therefore trigger fewer S3 calls
to check metadata?
Perhaps ViewFs could be handled incrementally later based on need. The
difficulty is that I think proper ViewFs support will impact the public API
definition, and then downstream applications need to react by migrating to
ViewFs-compatible method signatures. A prior example is
{{FileSystem#getDelegationToken}} vs. {{FileSystem#addDelegationTokens}}. Even
after a long time, there are lingering bugs in downstream applications that
have not migrated to {{addDelegationTokens}}, making them unusable with ViewFs
in secured deployments.
> Add a Blobstore interface to add to blobstore FileSystems
> ---------------------------------------------------------
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3, fs/swift
> Affects Versions: 2.6.0
> Reporter: Steve Loughran
> Assignee: Pieter Reuse
> Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch,
> HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch,
> HADOOP-9565-006.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
