GitHub user ianmcook added a comment to the discussion: Proposal: ADBC file drivers with pluggable storage
I agree that it would be valuable for ADBC to define a "data asset" abstraction and a related set of interfaces and registries. But: - I am skeptical that reusing the ADBC "driver" abstraction is the right choice. Despite the ostensible clean mapping of driver concepts to the needs you describe, I think that reusing "driver" for this would cause great confusion for everyday users who understand ADBC drivers fundamentally as libraries for interacting with systems that speak SQL. - I am doubtful that using Substrait to express filters is the right choice. I suspect that further research will reveal many ways in which Substrait is impractical without significant work to extend it, and I fear that this will introduce large unexpected complexity. I worry that the use of Substrait for this might end up being high-cost interoperability theater. - I think we need to consider how writes would work before we proceed too far with this. - I think we need to consider whether this could be rolled together with other related ADBC scope expansion proposals, such as the idea of an Iceberg driver (or some other abstraction for representing interoperable tables in object storage), into a single new abstraction, or whether those are best kept as separate abstractions. - I think we must ask questions about which organizations will fund the ADBC maintainer capacity that will be required to implement this ambitious expansion of ADBC's scope across all the libraries without creating a too-sparse support matrix across ADBC's language implementations. GitHub link: https://github.com/apache/arrow-adbc/discussions/4877#discussioncomment-18913493 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
