tustvold commented on issue #14854: URL: https://github.com/apache/datafusion/issues/14854#issuecomment-2695614546
> What should this object store be It depends on the [`ObjectStoreUrl`](https://docs.rs/datafusion/latest/datafusion/datasource/object_store/struct.ObjectStoreUrl.html). This is the mechanism that allows doing things like `SELECT * from s3://foo/bar.parquet`, etc... > I think attempting to unify an object store trait across the I/O needs of all possible file readers could be troublesome. I don't disagree, and was what I was recommending in the initial issue that triggered the creation of this - https://github.com/apache/arrow-rs/issues/7171 The challenge is that there are advantages to having a single unified IO interface, particularly when it comes to integrating DF into a shared codebase that may perform IO elsewhere, or that might want to access multiple different types of file. I'd personally recommend an approach that keeps ObjectStore as the default, but adds similar interfaces like AsyncFileReader to allow overriding the defaults for particular operators on a case-by-case basis if people wish to do so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
