GitHub user paleolimbot added a comment to the discussion: Proposal: ADBC file drivers with pluggable storage
Ours is DataFusion-specific, but we have something similar to implement the long tail of vector and raster spatial formats outside core SedonaDB: https://github.com/apache/sedona-db/blob/3e9ca68d97782b202bfdd12dd0d7c2b15c2bc8a1/c/sedona-extension/src/sedona_extension.h#L363-L421 . We haven't done filesystems or fileformats yet but it's next (the current functionality is basically to implement an iceberg reader; the next phase is to support pointcloud file formats). Filter and projection expressions are also a can of worms here, although I see the proposal is to use substrait. We use DataFusion protobuf because we have DataFusion on both sides and our functions are mostly not supported...I've liked having "expression" as an object you can ask questions about (e.g., serialize me in this format!) rather than a declarative serialized statement. Using ADBC for this would be great, and I'll happily replace whatever we do with it if the performance is the same. Maintaining the ability to do a parallelized scan is important for us but might not be a priority here (and could be implemented using the execute partitions feature). GitHub link: https://github.com/apache/arrow-adbc/discussions/4877#discussioncomment-18910205 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
