[
https://issues.apache.org/jira/browse/HADOOP-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18278:
------------------------------------
Component/s: fs/s3
> Do not perform a LIST call when creating a file
> -----------------------------------------------
>
> Key: HADOOP-18278
> URL: https://issues.apache.org/jira/browse/HADOOP-18278
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Sam Kramer
> Priority: Major
>
> Hello,
> We've noticed that when creating a file, which does not exist in S3, we see
> an extra LIST call gets issued to see if it's a directory (i.e. if key =
> "bar", it will issue an object list request for "bar/").
> Is this really necessary, shouldn't a HEAD request be sufficient to determine
> if it actually exists or not? As we're creating 1000s of files, this is quite
> expensive, as we're effectively doubling our costs for file creation. Curious
> if others have experienced similar or identical issues, or if there are any
> workarounds.
> [https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369]
>
> Thanks,
> Sam
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
