tustvold commented on issue #14854: URL: https://github.com/apache/datafusion/issues/14854#issuecomment-2684642530
I think this proposal makes a lot of sense, one thing I would suggest is thinking about if it is makes sense for datafusion-storage to be a higher-level abstraction, than just concerned with shuffling around ranges of bytes, or lists of files. Just implementing a trait that is a superset of ObjectStore or OpenDAL feels like it would just move the challenges around, instead of introducing a more meaningful abstraction layer. For example, I wonder if a higher level abstraction concerned with say reading parquet files, or something might be more flexible. It may be this already exists in the form of the new DataSource abstraction, although I have not followed this closely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
