[
https://issues.apache.org/jira/browse/HADOOP-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15527531#comment-15527531
]
Lei (Eddy) Xu commented on HADOOP-13448:
----------------------------------------
Hi, [~fabbri] Thanks for the proposals.
Regarding option (c), what is your thoughts about that it moves the complexity
to the caller (i.e., s3a filesystem)? For instance, s3a needs to determine
where to get a full list of s3 files under "{{/src}}"? The other concern, as
you mentioned, is the interface is not very intuitive to the API consumers.
As moving a directory recursively is implemented by copying actual S3 file
(object) one by one eventually, what about we just do the {{move(srcKey,
dstKey)}} together with {{copyFile()}} in the loop in
{{S3AFileSystem#innerRename()}}, so that {{move()}} does not need to validate
{{srcPath}} and {{dstPath}}
{code}
while (true) {
for (S3ObjectSummary summary : objects.getObjectSummaries()) {
keysToDelete.add(
new DeleteObjectsRequest.KeyVersion(summary.getKey()));
String newDstKey =
dstKey + summary.getKey().substring(srcKey.length());
copyFile(summary.getKey(), newDstKey, summary.getSize());
if (keysToDelete.size() == MAX_ENTRIES_TO_DELETE) {
removeKeys(keysToDelete, true);
}
}
}
{code}
> S3Guard: Define MetadataStore interface.
> ----------------------------------------
>
> Key: HADOOP-13448
> URL: https://issues.apache.org/jira/browse/HADOOP-13448
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-13448-HADOOP-13345.001.patch,
> HADOOP-13448-HADOOP-13345.002.patch, HADOOP-13448-HADOOP-13345.003.patch,
> HADOOP-13448-HADOOP-13345.004.patch, HADOOP-13448-HADOOP-13345.005.patch
>
>
> Define the common interface for metadata store operations. This is the
> interface that any metadata back-end must implement in order to integrate
> with S3Guard.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
