changhiskhan commented on issue #33986: URL: https://github.com/apache/arrow/issues/33986#issuecomment-1412841204
> I've had similar challenges with supporting datasets in delta-rs. Another aspect you'll need to think about is supporting Filesystems. In Rust, that means calling into Python functions, which I fear can be sub-optimal because of the GIL. I don't think there's a practical way to directly access the underlying C++ implemented FS unless we made the ABI stable (which I don't see us doing in the foreseeable future). > Does object store rs work for this? > > > A route I'm exploring right now is using the ADBC as a stable ABI for pushing down scan queries to storage formats and systems. It probably make more sense for table formats like Delta Lake, which have database-like semantics, than file formats like Lance (which I assume is the projection with the use case you are discussing). > > > > > if possible, make the top level abstraction pure python, so subclasses doesn't need to deal with cython etc if coming from Rust > Yeah most def pure python > > > It's harder, but part of me would prefer a stable C ABI, because it would mean the extension could be used in any language, not just Python. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
