[
https://issues.apache.org/jira/browse/HADOOP-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-17139.
-------------------------------------
Fix Version/s: 3.3.2
Resolution: Fixed
added in 3.3.2 Hadoop FS Shell uploads to S3 just got a lot faster!
> Re-enable optimized copyFromLocal implementation in S3AFileSystem
> -----------------------------------------------------------------
>
> Key: HADOOP-17139
> URL: https://issues.apache.org/jira/browse/HADOOP-17139
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0, 3.2.1
> Reporter: Sahil Takiar
> Assignee: Bogdan Stolojan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> It looks like HADOOP-15932 disabled the optimized copyFromLocal
> implementation in S3A for correctness reasons. innerCopyFromLocalFile should
> be fixed and re-enabled. The current implementation uses
> FileSystem.copyFromLocal which will open an input stream from the local fs
> and an output stream to the destination fs, and then call IOUtils.copyBytes.
> With default configs, this will cause S3A to read the file into memory, write
> it back to a file on the local fs, and then when the file is closed, upload
> it to S3.
> The optimized version of copyFromLocal in innerCopyFromLocalFile, directly
> creates a PutObjectRequest request with the local file as the input.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
