GitHub user chitralverma edited a comment on the discussion: (rust) How to use opendal to load data into DuckDB or Polars LazyFrame?
@Xuanwo I had one idea to make this work for all 3 - pyarrow, polars and duckdb. Instead of doing things on rust/ c side in each project, I saw that the python bindings all all 3 projects support fsspec based file systems - [pyarrow](https://arrow.apache.org/docs/python/filesystems.html#filesystem-fsspec), [duckdb](https://duckdb.org/docs/stable/guides/python/filesystems) and [polars](https://docs.pola.rs/user-guide/io/cloud-storage/#scanning-with-pyarrow) So if the opendal team sees it as a good fit, we can provide an opendal backed implementation of the [fsspec.AsyncFileSystem](https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/asyn.html#AsyncFileSystem) + maybe [pyarrow.fs.FSSpecHandler](https://arrow.apache.org/docs/python/generated/pyarrow.fs.FSSpecHandler.html#pyarrow.fs.FSSpecHandler), directly in the python bindings for end users to use (maybe behind an extra/ group??). Beyond polars and duckdb, also this will help users to use opendal backends through these interfaces. Someone is also partially maintaining this in a separate [repo](https://github.com/fsspec/opendalfs), might as well do it officially wdyt? GitHub link: https://github.com/apache/opendal/discussions/5972#discussioncomment-13062336 ---- This is an automatically sent email for dev@opendal.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@opendal.apache.org
