[jira] [Created] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads

[Ted Malaska (JIRA)]

Ted Malaska created HADOOP-10560:
------------------------------------

             Summary: Update NativeS3FileSystem to issue copy commands for 
files with in a directory with a configurable number of threads
                 Key: HADOOP-10560
                 URL: https://issues.apache.org/jira/browse/HADOOP-10560
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs/s3
            Reporter: Ted Malaska
            Priority: Minor


In NativeS3FileSystem if you do a copy of a directory it will copy all the 
files to the new location, but it will do this with one thread. Code is below. 
This jira will allow a configurable number of threads to be used to issue the 
copy commands to S3.
do {
PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, 
priorLastKey, true);
for (FileMetadata file : listing.getFiles())
{ keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + 
file.getKey().substring(srcKey.length())); }
priorLastKey = listing.getPriorLastKey();
} while (priorLastKey != null);



--
This message was sent by Atlassian JIRA
(v6.2#6252)