alamb commented on issue #24265: URL: https://github.com/apache/datafusion/issues/24265#issuecomment-5401004383
Pulling some discussion out into https://github.com/apache/datafusion/pull/24353#issuecomment-5400987652 > If we agree on supporting more "streaming" execution in DataFusion, I think we should extend that support across all existing operators. +1 from me on the overall direction. How to design this while keeping maintenance complexity low could be a separate discussion — maybe in [#24265](https://github.com/apache/datafusion/issues/24265) In my mind DataFusion is in a half way state now -- it has some features / support for streaming (e.g. `Boundedness`), but as shown in #24044 since there is no clear design / tests / documentation implicit assumptions that streaming systems may be counting on can (and do) get broken So I think we should first agree on if we want to try and make DataFusion more useful for building streaming engines -- I think it is important and there are a bunch of people already doing so (and have done so for a while -- e.g. Arroyo and earlier versions of Synnada). Rewriting Apache Flink in rust seems to be all the rage now too (e.g. @jordepic 's StreamFusion, etc) , and many people are using DataFusion to try. it seems like that is the consensus of this ticket too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
