criccomini commented on PR #348: URL: https://github.com/apache/arrow-rs-object-store/pull/348#issuecomment-2864073148
> In particular if a user queries two files from the same bucket, we don't want to register two separate stores for this same bucket. The thing I wrestle with, which @alamb is also pointing out, is that I might have a URL like this: `s3://my-slatedb-bucket/dbs/db_1/wal` In that URL, `s3://my-slatedb-bucket/dbs/db_1` is the bucket + root path of my SlateDB database. `/wal` is the path relative to the bucket root. In SlateDB, the current recommended way to do this would be to use an `AmazonS3Builder` and then wrap the returned `ObjectStore` in a [`PrefixStore`](https://docs.rs/object_store/0.12.0/object_store/prefix/struct.PrefixStore.html). I'm not sure how we'd do this with the current `ObjectStoreRegistry`. I suppose it will be up to individual implementations. In the case of SlateDB, we'll know if the URL is for an SST file, manifest file, WAL directory, and so on. This will allow us to strip those items from the path to discover the root. We can wrap the `ObjectStore` that the registry returns with a `PrefixStore` using our discovered root. tl;dr I think the way we have it is right, though not intuitive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
