matthewmturner commented on issue #2489: URL: https://github.com/apache/arrow-datafusion/issues/2489#issuecomment-1120614400
Indeed the IOx `ObjectStore` implementation looks both robust and feature rich - and has the added benefit of being actively used. While `datafusion-objectstore-s3` has been released on crates i am not aware of any production workloads leveraging it. I also prefer the more generic API interface in the IOx implementation, I had actually planned on proposing something similar on https://github.com/apache/arrow-datafusion/issues/2445. From an s3 perspective the only things that come to mind are: - Rusoto is used rather than official AWS Rust Sdk. If thats good enough for IOx then im sure its fine for now, but i think would be good to move to AWS Rust Sdk sooner than later. - We can just double check that our existing test suite in `datafusion-objectstore-s3` works without issue. In particular for connecting to services such as MinIO (which is S3 compatible storage) - I have spoken to them and know they were [watching](https://www.cloudnatively.com/query-parquet-on-minio/) the project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
