andygrove commented on issue #2709: URL: https://github.com/apache/arrow-datafusion/issues/2709#issuecomment-1158922350
> The main issue a few months ago was that the datafusion codebase was not split like it is now, so it created conflicts in the various places where it wasn't possible to confine differences between arrow and arrow2::io to extension traits. So my question is, can a contrib crate exist since arrow2 is essentially required at the core ? Regardless of where the code lives, I would view us as having two main options: 1. Continue to maintain a copy of DataFusion "core" with modifications to allow building with arrow or arrow2. This will be an ongoing maintenance burden and IMO only worth the effort if we are working towards having this merged into DataFusion master at some point and I am not sure how much appetite there is for this. 2. Fork `datafusion-core` to create a new query engine based on `arrow2` that can leverage the new DataFusion crates for logical plan building, SQL query planning, logical plan optimizations etc. Over time maybe more functionality can be moved out of core and re-used by other query engines. This would decouple the development work and allow those interested in arrow2 to move faster and does not put any burden on the core DataFusiuon development. That would be my choice if I were working on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
