alamb opened a new issue, #252: URL: https://github.com/apache/arrow-rs-object-store/issues/252
ported from https://github.com/influxdata/object_store_rs/issues/18 by @tustvold Note there are several good comments on that ticket by @tustvold and @wjones127 that I did not copy over **Problem** Currently this crate uses upstream crates to provide interaction with object storage. This comes with a few downsides: * Missing features, e.g. range support, conditionals, etc... * Inconsistent error handling, e.g. it can be next to impossible to get the HTTP status code * Extreme dependency bloat **Proposal** This crate does not intend to cover more than the basic APIs of each store, which boils down to just a couple of different request types. I would like therefore like to propose: * Move to using a reqwest client directly * Use serde to serialize payloads * Use ring to handle signatures/etc... * Use rustls to handle TLS This will allow: * Consistent error handling, retries, etc... * Smaller dependency footprint * New features without waiting on upstreams * Simpler codebase It's up to whoever picks this up where to start, but I would suggest starting with GCS as: * It will unlock range requests * The authentication logic is simpler than say AWS * The JSON API is relatively straightforward If that goes well, we can then look to move onto the others **Additional Context** https://github.com/influxdata/object_store_rs/pull/15#issuecomment-1134893330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
