alamb commented on PR #24353: URL: https://github.com/apache/datafusion/pull/24353#issuecomment-5400987652
> [#24044](https://github.com/apache/datafusion/issues/24044) shows the issue that arises if we don't support "streaming". > > If we agree on supporting more "streaming" execution in DataFusion, I think we should extend that support across all existing operators. +1 from me on the overall direction. How to design this while keeping maintenance complexity low could be a separate discussion — maybe in [#24265](https://github.com/apache/datafusion/issues/24265) Yeah -- in my mind DataFusion is in a half way state now -- it has some features / support for streaming (e.g. `Boundedness`), but as shown in #24044 since there is no clear design / tests / documentation implicit assumptions that streaming systems may be counting on can (and do) get broken So I think we should first agree on if we want to try and make DataFusion more useful for building streaming engines -- I think it is important and there are a bunch of people already doing so (and have done so for a while -- e.g. Arroyo and earlier versions of Synnada). Rewriting Apache Flink in rust seems to be all the rage now too (e.g. @jordepic 's StreamFusion, etc) , and many people are using DataFusion to try. > @jayzhan211 thanks for the pointer to that EPIC > > My main blocking suggestion for this PR is to completely separate the bounded and unbounded execution paths, rather than adding the unbounded behavior into the existing implementation. > > If we can do that, I think this PR should be good to go. I explained the reasoning in more detail in the comment in [#24265 (comment)](https://github.com/apache/datafusion/issues/24265#issuecomment-5383969323) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
