ameyc opened a new issue, #11365: URL: https://github.com/apache/datafusion/issues/11365
### Is your feature request related to a problem or challenge? Currently DataFusion operators communicate via a narrow API i.e. forwarding `SendableRecordBatchStreams`. In some instances, in particular the ExecutionPlans operating on unbounded streams need to snapshot their state and co-ordinate with source operators. It'd be a powerful primitive to add a `StateBackend` concept to the `RuntimeEnv` where users could then write operators to store adhoc durable state into a backend such as rocksdb. Realise this may not be useful for many use cases but [RuntimeEnv](https://github.com/apache/datafusion/blob/e65c3e919855c9977cf4d80c0630ee26b7fd03ee/datafusion/execution/src/runtime_env.rs#L46-L55) does seem to have ability to plug in an object store registry as well as a catalog manager. This would be a crucial unlock to make stateful stream processing application with DataFusion. If the current, API contains such a pathway already, would love to get pointers in the right direction. ### Describe the solution you'd like _No response_ ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
