alamb commented on issue #23194: URL: https://github.com/apache/datafusion/issues/23194#issuecomment-4858824752
> We (Datadog), will gladly donate https://github.com/datafusion-contrib/datafusion-distributed to Apache if that implies hosting it as a new crate in https://github.com/apache/datafusion. We've built the crate leaving that door open from the beginning, both from a philosophy and code standpoint. > > We've discussed this in the past, but at that moment it was not the right time. Now that we've been running it in production at a huge scale for a while and we can no longer break things and move fast, the door is open. I am much more in favor of adopting datafusion-distributed compared to starting with a new crate in the datafusion repo -- `datafusion-distributed` has had a lot of thought in it and I have heard from many users they are using it / plan to use it (as are we at InfluxData). This gives me confidence that its APIs are broadly applicable and we can maintain it. To be clear, datafusion-distributed is not a complete distributed query engine, it is more like "the common pieces needed to build one" -- like shuffles and network communication traits. > I'm going to study datafusion-distributed over the next couple of days to understand the architecture and how it varies with Ballista so that I can make sure that I am not proposing something that would help with Ballista and not with datafusion-distributed. If we could somehow migrate Ballista to use datafusion-distributed I think that would also be amazing (and would remove all doubt about moving datafusion-distributed into the core datafusion crate) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
