[
https://issues.apache.org/jira/browse/HADOOP-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17551714#comment-17551714
]
Steve Loughran commented on HADOOP-18278:
-----------------------------------------
note: you can use the api today, with the .opt() value...it will be ignored on
releases without the feature. so you can code for it as soon as it is
committed, at which point you'll know the final key name
> Do not perform a LIST call when creating a file
> -----------------------------------------------
>
> Key: HADOOP-18278
> URL: https://issues.apache.org/jira/browse/HADOOP-18278
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.3.3
> Reporter: Sam Kramer
> Priority: Minor
>
> Hello,
> We've noticed that when creating a file, which does not exist in S3, we see
> an extra LIST call gets issued to see if it's a directory (i.e. if key =
> "bar", it will issue an object list request for "bar/").
> Is this really necessary, shouldn't a HEAD request be sufficient to determine
> if it actually exists or not? As we're creating 1000s of files, this is quite
> expensive, as we're effectively doubling our costs for file creation. Curious
> if others have experienced similar or identical issues, or if there are any
> workarounds.
> [https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369]
>
> Thanks,
> Sam
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
