carols10cents commented on PR #4876: URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1749015043
Ok, so here's another wrinkle. The URL signing API should return a full URL so that if the user wants to, they can pass the URL to something that isn't using the `object_store` crate and use it with `cURL`, `wget`, a web browser, what have you. However, if the user _is_ doing all upload/download through the `object_store` crate, and particularly through wrappers like `LimitStore` and such, they _will_ want to send the request using the presigned URL through their `object_store`, but currently `put_request` and friends take a `Path` that gets concatenated onto the object store implementation's full URL (which wouldn't be what we'd want in this case). So I think the `ObjectStore` trait will need methods like `fn put_presigned(&self, location: &Url, bytes: Bytes)` and `fn get_presigned(&self, location: &Url)`, and these should be on the trait so that the adapters like `LimitStore` and `ThrottledStore` can support them as well. In IOx terms, the router will have something that can produce signed urls (configured using the code in clap_blocks, [I think I've figured out a good way to do that](https://github.com/influxdata/influxdb_iox/pull/8927)) and will pass that full URL to a bulk import tool, that will also have an object store configured using the code in clap_blocks (that currently returns an `Arc<DynObjectStore>` which is a `LimitStore` wrapping an `AmazonS3` in this case) so that we don't have to reimplement all those settings and code handling the upload request. Does that sound right to you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
