andygrove commented on issue #872: URL: https://github.com/apache/datafusion-python/issues/872#issuecomment-2356421174
> One question I have is who would maintain this new code? Ballista I think suffers from a slow decline due to lack of active maintenance and community. We should try to avoid the same thing happening to Ray That's a great point. If the project were to become part of the Apache DataFusion project then I would certainly put time into maintaining it and helping build community around the project. I am not able to contribute in its current location. I have recently been attempting to maintain Ballista by upgrading to more recent versions of DataFusion, but the project is large and complex and the original contributors of much of this code are no longer available to help, so it is challenging. I believe that DataFusion + Ray is an opportunity to start fresh on a solution for distributed DataFusion as a much lighter weight alternative to Ballista and the project is small enough (~40 commits) that it will be easier for new contributors to follow along. These are the initial tasks that I would plan on working on (with the community, hopefully) if we were to move forward with this proposal. - Make sure that the current code still runs with recent versions of Python + Ray - Update README to reflect that distributed execution is now supported - Run fresh benchmarks and compare to Ballista - Fix the outstanding bug https://github.com/datafusion-contrib/ray-sql/issues/44 - Upgrade to more recent versions of DataFusion (one PR per version) Another possibility is that interested contributors could start maintaining the project in its current location, but I am not sure who would be able to approve the PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
