alamb commented on issue #24044: URL: https://github.com/apache/datafusion/issues/24044#issuecomment-5251986594
> A bug because rows that are already available can be withheld indefinitely: the producer-side coalescer flushes its residual only when the input ends, and an unbounded input never does. For a source emitting small batches the practical effect is no output at all rather than delayed output. In the repro above, at the default 8192 and a source producing 10 rows/second, the first batch would appear after roughly 14 minutes. The counter to this is that for batch workloads, emitting more frequently is less efficient (as there is now more overhead per batch) I recently watched a presentation from @sap1ens and he also mentioned this mismatch between streaming and batch oriented workflows for several built in operators: https://www.youtube.com/watch?v=0-BIHyzODH8&t=2s <img width="1799" height="967" alt="Image" src="https://github.com/user-attachments/assets/e0c68699-2cde-48c6-b8d9-1bce2b22ecc2"; /> Given how many systems seem to want to use DataFusion for streaming systems (rather than batch oriented ones) and the natural tension between batching (maximize throughput) vs streaming (minimize latency) I wonder if we should try and add some way to configure which behavior is desired. For example, there is a way for operators today to report their boundedness behavior: https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlanProperties.html#tymethod.boundedness Maybe it would be worth some sort of "bounded configuration mode" setting to make implementing streaming systems easier For example, 1. A global config setting like `streaming` or `optimize_for_latency` 2. A flag on operators like FilterExec and RepartitionExec that controls their emission behavior (aka should they flush at the earliest opportunity) > I think there is a more sensible default without new deadlines if the user doesn't specify an explicit intent. Namely, bias unbounded execution towards low latency. What if to drain all Poll::Ready batches, i.e. until there is an await point? @ahirner, this is an interesting idea to try and avoid the flag and figure it out from the input declarations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
