[
https://issues.apache.org/jira/browse/MAPREDUCE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Malaska resolved MAPREDUCE-5872.
------------------------------------
Resolution: Invalid
Release Note: Needs to be a Hadoop Jira not a MapReduce Jira
> Update NativeS3FileSystem to issue copy commands for files with in a
> directory with a configurable number of threads
> --------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5872
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5872
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: performance
> Reporter: Ted Malaska
> Assignee: Ted Malaska
> Priority: Minor
> Labels: performance
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the
> files to the new location, but it will do this with one thread. Code is
> below. This jira will allow a configurable number of threads to be used to
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles()) {
> keysToDelete.add(file.getKey());
> store.copy(file.getKey(), dstKey +
> file.getKey().substring(srcKey.length()));
> }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
