iodeal commented on issue #327: URL: https://github.com/apache/arrow-rs-object-store/issues/327#issuecomment-2791464165
As documented on [ObjectStore::list](https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#tymethod.list) >List all the objects with the given prefix. > >Prefixes are evaluated on a path segment basis, i.e. foo/bar is a prefix of foo/bar/x but not of foo/bar_baz/x. **List is recursive**, i.e. foo/bar/more/x will be included. > >Note: the order of returned [ObjectMeta](https://docs.rs/object_store/latest/object_store/struct.ObjectMeta.html) is not guaranteed Based on this, we have two situations to discuss. Firstly, the prefix refers to the actual directory that exists(In this case, `ObjectStore::list` is able to meet the needs)： linux native fs `ls -R foo/bar/ #same as 'ls -R foo/bar'` s3 `aws s3 ls --recursive s3://bucket-name/foo/bar/` Another scenario is that the prefix is only a fuzzy matching prefix for the directory(In this case, `ObjectStore::list` is not able to meet the needs)： linux native fs `ls -R foo/bar*` s3 `aws s3 ls --recursive s3://bucket-name/foo/bar` As documented on [ObjectStore::list_with_delimiter](https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#tymethod.list_with_delimiter) >List objects with the given prefix and an implementation specific delimiter. Returns common prefixes (directories) in addition to object metadata. > >Prefixes are evaluated on a path segment basis, i.e. foo/bar is a prefix of foo/bar/x but not of foo/bar_baz/x. **List is not recursive**, i.e. foo/bar/more/x will not be included. Based on this, the prefix refers to the actual directory that exists. linux native fs `ls foo/bar/ #same as 'ls foo/bar'` s3 `aws s3 ls s3://bucket-name/foo/bar/` If we remove the tailing `/`，LocalFilesystem will return directories under the foo/bar, but S3 will only return `foo/bar/`. To sum up, `ObjectStore::list_with_delimiter` should add tailing `/`; whether `ObjectStore::list` adds tailing `/` depends on whether the caller has passed tailing `/`. As for the processing logic of `ObjectStore::list_with_offset`, its current processing logic is consistent with `ObjectStore::list` or `ObjectStore::list_with_delimiter`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
