Steve Loughran created HADOOP-18966:
---------------------------------------
Summary: s3a file rename does double HEAD or LIST on source
fiile/dir
Key: HADOOP-18966
URL: https://issues.apache.org/jira/browse/HADOOP-18966
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.6
Reporter: Steve Loughran
going to add this as a v2 issue, though really it's been latent.
when you do a rename() we do a getFileStatus call to get the list/file
status..if this is a dir that's a single LIST; a file then it'll be LIST + HEAD.
when the actual copy is kicked off,
* if the source is a dir, then a duplicate LIST is initiated straight
afterwards.
* FIle: there's a second HEAD to get the metadata for the copy
Proposed.
* LIST initiates a full list, and if it returns a list of objects, that list
iterator is passed in to rename.
* full result of HEAD preserved and passed down to copyFile()
will cut out one round trip regardless of source type
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
