[ https://issues.apache.org/jira/browse/HADOOP-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-13884: ------------------------------------ Description: before doing a create(), s3a does a getFileStatus() to make sure there isn't a directory there, and, if overwrite=false, that there isn't a file. Because S3 caches negative HEAD/GET requests, if there isn't a file, then even after the PUT, a later GET/HEAD may return 404; we are generating create consistency where none need exist. when overwrite=true we don't care whether the file exists or not, only that the path isn't a directory. So we can just do the HEAD path +"/' and the LIST calls, skipping the {{HEAD path}}. This will save an HTTP round trip of a few hundred millis, and ensure that there's no 404 cached in the S3 front end for later callers was: before doing a create(), s3a does a getFileStatus() to make sure there isn't a directory there, and, if overwrite=false, that there isn't a file. Because S3 caches negative HEAD/GET requests, if there isn't a file, then even after the PUT, a later GET/HEAD may return 404; we are generating create consistency where none need exist. when overwrite=true we don't care whether the file exists or not, only that the path isn't a directory. So we can just to the HEAD path +"/' and the LIST calls, skipping the {{HEAD path}}. This will save an HTTP round trip of a few hundred millis, and ensure that there's no 404 cached in the S3 front end for later callers > s3a create(overwrite=true) to only look for dir/ and list entries, not file > --------------------------------------------------------------------------- > > Key: HADOOP-13884 > URL: https://issues.apache.org/jira/browse/HADOOP-13884 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 2.9.0 > Reporter: Steve Loughran > Priority: Minor > > before doing a create(), s3a does a getFileStatus() to make sure there isn't > a directory there, and, if overwrite=false, that there isn't a file. > Because S3 caches negative HEAD/GET requests, if there isn't a file, then > even after the PUT, a later GET/HEAD may return 404; we are generating create > consistency where none need exist. > when overwrite=true we don't care whether the file exists or not, only that > the path isn't a directory. So we can just do the HEAD path +"/' and the LIST > calls, skipping the {{HEAD path}}. This will save an HTTP round trip of a few > hundred millis, and ensure that there's no 404 cached in the S3 front end for > later callers -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org