Sahil Takiar created HADOOP-17139:
-------------------------------------
Summary: Re-enable optimized copyFromLocal implementation in
S3AFileSystem
Key: HADOOP-17139
URL: https://issues.apache.org/jira/browse/HADOOP-17139
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Sahil Takiar
It looks like HADOOP-15932 disabled the optimized copyFromLocal implementation
in S3A for correctness reasons. innerCopyFromLocalFile should be fixed and
re-enabled. The current implementation uses FileSystem.copyFromLocal which will
open an input stream from the local fs and an output stream to the destination
fs, and then call IOUtils.copyBytes. With default configs, this will cause S3A
to read the file into memory, write it back to a file on the local fs, and then
when the file is closed, upload it to S3.
The optimized version of copyFromLocal in innerCopyFromLocalFile, directly
creates a PutObjectRequest request with the local file as the input.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
