[
https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989523#comment-13989523
]
Ted Malaska commented on HADOOP-10560:
--------------------------------------
Hey Steve,
I hope your weekend went well. I was thinking about you initial comments on
the point of InterruptedIOException and the root cause logic.
Let me know if you would still want the InterruptedIOException. If you do I
will make a new patch today.
Also let me know if you want me to take the extra step and find the root cause
before the runtime exception in comment 7. Also if you do want that I will
update the patch today.
Thanks again for helping me through this jira.
Ted Malaska
> Update NativeS3FileSystem to issue copy commands for files with in a
> directory with a configurable number of threads
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-10560
> URL: https://issues.apache.org/jira/browse/HADOOP-10560
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Ted Malaska
> Assignee: Ted Malaska
> Priority: Minor
> Labels: performance
> Attachments: HADOOP-10560-1.patch, HADOOP-10560.patch
>
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the
> files to the new location, but it will do this with one thread. Code is
> below. This jira will allow a configurable number of threads to be used to
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey +
> file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
