jorgecarleitao commented on issue #2709: URL: https://github.com/apache/arrow-datafusion/issues/2709#issuecomment-1158030051
@andygrove, thanks for the ping! I do not have a preference - I am very happy to support anyone that wishes to use arrow2/parquet2, and I am very thankful for everyone that has been maintaining the arrow2 branch. I unfortunately do not have the time left to spearhead it. @dbr I would be very happy to support you in the migration - let me know how I can help and feel free to assign tasks directly. Arrow2's core API is essentially stable and that we we usually have small breaking changes in IO, so most of the changes should be on the datafusion's side. For full transparency: imo the primary question here is one of control - although we demonstrated that arrow2 had superior support, safety, performance, documentation, and UX, I sensed that the main requirement was that arrow2 was donated to Apache, which for me was unacceptable. This demotivated me from contributing here. I understand that companies that have a strong dependency on DataFusion need control over its core dependency. For me it demos that we are a bit past the idea that people in Apache are individual contributors. The core development of Polars and Databend have been more pragmatic and I have thus been more aware of the requests there. However, if there is anything needed, just drop an issue and I will gladly take it, irrespectively of where the development happens (here or in a separate repo / org). I continue to believe that DataFusion would benefit immensely from arrow2's current and future capabilities - arrow2 is the backend of likely the fastest in-node dataframe APIs out there and just this week we exposed a new API that enable compute to be 2x-5x faster ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
