[jira] [Updated] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads

[Ted Malaska (JIRA)]

     [ 
https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Malaska updated HADOOP-10560:
---------------------------------

    Attachment: HADOOP-10560.patch

First attempt to added threading to copying files in the case a directory is 
renamed.

I just use and executorService with futures.  Then try to get all the futures 
before preforming the normal deletes that were there before.

This is ready for review.

Thanks again

> Update NativeS3FileSystem to issue copy commands for files with in a 
> directory with a configurable number of threads
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-10560
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10560
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Ted Malaska
>            Assignee: Ted Malaska
>            Priority: Minor
>              Labels: performance
>         Attachments: HADOOP-10560.patch
>
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the 
> files to the new location, but it will do this with one thread. Code is 
> below. This jira will allow a configurable number of threads to be used to 
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, 
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + 
> file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);



--
This message was sent by Atlassian JIRA
(v6.2#6252)