[ https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13991318#comment-13991318 ]
Andrei Savu commented on HADOOP-10560: -------------------------------------- ... and also when fs.s3n.copyThreads = 1 we can use MoreExecutors.sameThreadExecutor() if Guava is available as a library. > Update NativeS3FileSystem to issue copy commands for files with in a > directory with a configurable number of threads > -------------------------------------------------------------------------------------------------------------------- > > Key: HADOOP-10560 > URL: https://issues.apache.org/jira/browse/HADOOP-10560 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 > Reporter: Ted Malaska > Assignee: Ted Malaska > Priority: Minor > Labels: performance > Attachments: HADOOP-10560-1.patch, HADOOP-10560.patch > > > In NativeS3FileSystem if you do a copy of a directory it will copy all the > files to the new location, but it will do this with one thread. Code is > below. This jira will allow a configurable number of threads to be used to > issue the copy commands to S3. > do { > PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, > priorLastKey, true); > for (FileMetadata file : listing.getFiles()) > { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + > file.getKey().substring(srcKey.length())); } > priorLastKey = listing.getPriorLastKey(); > } while (priorLastKey != null); -- This message was sent by Atlassian JIRA (v6.2#6252)