criccomini commented on PR #348: URL: https://github.com/apache/arrow-rs-object-store/pull/348#issuecomment-2874793780
@alamb @tustvold Ok, I think this is ready for (another) review. I've added a few implementations of `ObjectStoreRegistry` that (I think) should get most users what they want. There are now three: - `DefaultObjectStoreRegistry`: A very dumb registry that simply treats URLs in `get_store` calls as the prefix. - `PrefixObjectStoreRegistry`: A registry that wraps `DefaultObjectStoreRegistry` and calls a `prefix_fn` on all `get_store` URLs. The default implementation uses DataFusion's `get_url_key` implementation style. - `ParserObjectStoreRegistry`: A registry that wraps `PrefixObjectStoreRegistry` and calls a `parser_fn` on any `get_store` calls that aren't set in the wrapped (prefix) registry. The `parser_fn` defaults to using the `parse_url` method. If the `parser_fn` returns an object store, it's cached using the prefix that was resolved in `get_store`. If a user wants to use custom `opts` for different URL prefixes, they can supply their own `parser_fn`. See the tests and docs for examples that use `parse_url_opts`. This should address @rtyler's use case; it's also something we'll need for SlateDB. Please take a pass and let me know how things look. A few notes for reviewers: 1. I'm not 100% on my Rust foo, so pay attention to silly things I might have done with Boxes, Arcs, and so on. 2. If y'all feel like this is too much, let me know. I can remove the prefix and/or parser implementations. 3. I found that `test_url_http` wasn't working as expected when I copied its code over to test the `ParserObjectStoreRegistry` on an `http` URL. I decided to fix the test as part of this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
