westonpace commented on issue #25019: URL: https://github.com/apache/arrow/issues/25019#issuecomment-1478038079
Mentioned in #34213 > I have no idea what the implications are. @pitrou I attempted some investigation. In the AWS CLI it appears that the delimiter is [only used when the listing is non-recursive](https://github.com/aws/aws-cli/blob/develop/awscli/customizations/s3/subcommands.py#L479-L485). In S3FS the delimiter is skipped [when looking for a file recursively](https://github.com/fsspec/s3fs/blob/0dfb5893ed9dc82a11bb5ce498c1ccc25cec8235/s3fs/core.py#L811). In the [S3 docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) it states: > If you issue a list request with a delimiter, you can browse your hierarchy at only one level, skipping over and summarizing the (possibly millions of) keys nested at deeper levels. My conclusion is that the delimiter's purpose is to reduce the number of files returned when you do not need to retrieve all the files. If we are doing a recursive listing then I think it is consistent with other projects and S3's intentions that we do not specify the delimiter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
