[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15397825#comment-15397825 ]
Chris Nauroth commented on HADOOP-9565: --------------------------------------- I'm afraid I'm unclear about the goals here, specifically related to the definition of Put and Copy operations. {code} public void Put(Path path, OutputStream source, long length, long options) {code} This method signature surprised me. Was the {{OutputStream}} argument meant to be an {{InputStream}} for the implementation read? Like Pieter, it's unclear to me that this is significant as part of the {{ObjectStore}} interface. It looks similar to the {{FileUtil#copy}} utility methods, not a primitive operation on {{FileSystem}} or {{ObjectStore}}. Early comments indicate that this could have been a way for S3A to implement a direct streaming write instead of spooling to local disk first. Now that S3A has {{S3AFastOutputStream}}, does that meet the need? Or are you looking to provide clients with a guaranteed non-spooling API regardless of configuration? Is the {{options}} the significant part that I'm missing? Is the idea that the caller passes along options to describe desired semantics, and this gives the {{ObjectStore}} implementation the opportunity to throw {{UnsatisfiedSemanticsException}}? Maybe a usage example showing how you expect callers to use Put would clarify it. Regarding Copy, the description indicates that method could substitute for rename. Looking at S3A as a concrete example, the rename already uses S3 server-side copy. How do you expect an S3A implementation of Copy would differ from the current {{S3AFileSystem#rename}}? Do you expect it can relax the notoriously complex semantics of rename, and therefore trigger fewer S3 calls to check metadata? Perhaps ViewFs could be handled incrementally later based on need. The difficulty is that I think proper ViewFs support will impact the public API definition, and then downstream applications need to react by migrating to ViewFs-compatible method signatures. A prior example is {{FileSystem#getDelegationToken}} vs. {{FileSystem#addDelegationTokens}}. Even after a long time, there are lingering bugs in downstream applications that have not migrated to {{addDelegationTokens}}, making them unusable with ViewFs in secured deployments. > Add a Blobstore interface to add to blobstore FileSystems > --------------------------------------------------------- > > Key: HADOOP-9565 > URL: https://issues.apache.org/jira/browse/HADOOP-9565 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/s3, fs/swift > Affects Versions: 2.6.0 > Reporter: Steve Loughran > Assignee: Pieter Reuse > Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, > HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch, > HADOOP-9565-006.patch > > > We can make the fact that some {{FileSystem}} implementations are really > blobstores, with different atomicity and consistency guarantees, by adding a > {{Blobstore}} interface to add to them. > This could also be a place to add a {{Copy(Path,Path)}} method, assuming that > all blobstores implement at server-side copy operation as a substitute for > rename. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org