[
https://issues.apache.org/jira/browse/HADOOP-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970195#comment-16970195
]
Steve Loughran commented on HADOOP-16629:
-----------------------------------------
bq. S3guard is one of good reasons I think this API needs to be in Hadoop
rather than forking a process out to run "s3 sync". The encryption problems are
not specific to this API, because it is equally applicable here.
Yes, but different file systems and may have different configurations such as
S3Guard DDB tables -you need to know the specific table used by the source FS
and query that for the directory listings, rather than use the the settings of
the destination FS. Similarly, the secret key for SSE-C operations needs to be
known in the source; you actually need to add that as one of the headers in the
copy operation.
Also, if you actually want a version of this CP which was optimised for this
world, you would do multipart copies. That is your source would not be a simple
URI, it would be a URI and a range; the result an opaque byte array containing
the information needed to commit the request along with the rest of the
initiated upload. Do you just want an S3 guard enabled version of "aws s3" what
do you actually want a distcp which can do cross store copying? As I know which
one will scale better.
bq. Those particular problems aren't solved by ignoring them,
I concur.
bq. but they are also not solved by forcing a ViewFS + path mounts as a
workaround for what you propose.
I think you have misunderstood or I have explained badly. I wasn't trying to
force a ViewFS model.
I am trying to say
* it is a lot harder than you think and just implementing Filesystem.copy(URI,
URI) from a single file system isn't going to work.
* we like our file system APIs to be stable and cross store.
For now: use "aws s3" and then s3guard import to build up the table at the
destination.
> support copyFile in s3a filesystem
> ----------------------------------
>
> Key: HADOOP-16629
> URL: https://issues.apache.org/jira/browse/HADOOP-16629
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.1
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
