thinkharderdev opened a new issue, #25124: URL: https://github.com/apache/datafusion/issues/25124
### Is your feature request related to a problem or challenge? Currently everything in DataFusion is released together which makes the upgrade path painful and causes dependency conflicts if you have a project which depends on multiple 3rd party dependencies which themselves depend on DataFusion. For example, if I am building a query engine which should support multiple different table formats and want to use 3rd party dependencies which expose DataFusion `TableProvider`s (e.g https://github.com/apache/iceberg-rust and https://github.com/delta-io/delta-rs), but also implement the core execution engine on top of DataFusion then I need to ensure all dependencies are in sync in terms of their DF dependency. ### Describe the solution you'd like Separate the stable API for internal implementation as separate crates. For instance instead of having just a `datafusion-catalog` crate with both the stable API and implementations we can have two crate: 1. `datafusion-catalog-api` 2. `datafusion-catalog` (which depends on `datafusion-catalog-api` The `-api` crate would contain only trait definitions like: `TableProvider` `CatalogProvider` etc. The `-api` should (theoretically) evolve much more slowly than the implementation so 3rd party crates that want to expose a `TableProvider` can depend only on `datafusion-catalog-api` and I can consume that dependency while using any version of `datafusion-catalog` I want (provided it uses the same stable API). Obviously there are quite a few details to work out about which APIs to "stabilize" in this way, which I will hand-wave past :) ### Describe alternatives you've considered A lot of alternative suggestions to the exact same issue have been proposed in https://github.com/apache/datafusion/issues/16622 ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
