GitHub user CurtHagenlocher edited a comment on the discussion: Proposal: ADBC file drivers with pluggable storage
> Ok, I guess this could work as an ADBC driver. It could also work as its own > spec. The attraction of attaching it to ADBC is that there's already a driver manager and I think this kind of specification benefits dramatically from dynamic loading (as Felipe pointed out in the Office Hours). EDIT: to be clear, he actually talked about dependency isolation but that comes almost of necessity from dynamically loading a self-contained library with a well-defined C-compatible API. > I think we'd want a way to pass the entire driver manager to a driver, > perhaps? So it could use the registry and load any file drivers (without > having to embed its own driver manager)? I have a very vague notion that a host might want to supply the storage implementation through the driver manager to the file driver, but will concede that I haven't gone to the trouble of crisply enumerating possible scenarios and their implementation needs. > Note that Spark also has a proposal for native (JNI/FFM) data sources: > [apache/spark#59313](https://github.com/apache/spark/pull/59313) Interesting! And wouldn't it be nice to do something that works well with Spark but isn't tied to it? :D > The filesystem API seems reasonable at a first glance. We could probably > compare notes with/steal inspiration from OpenDAL and Arrow Filesystems > though it looks similar to the latter off the top of my head. +1 to taking prior art into account. (I had AI generate me a "dream filesystem API" based on examples it could find in OpenDAL, Arrow, Vortex, etc. but focused strictly on performance. It was ... perhaps not very usable.) GitHub link: https://github.com/apache/arrow-adbc/discussions/4877#discussioncomment-18910841 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
