calvinchengx commented on issue #24044: URL: https://github.com/apache/datafusion/issues/24044#issuecomment-5229014416
@alamb It's a bug rather than a change in behaviour, but I don't think it should hold 55.0.0. A bug because rows that are already available can be withheld indefinitely: the producer-side coalescer flushes its residual only when the input ends, and an unbounded input never does. For a source emitting small batches the practical effect is no output at all rather than delayed output. In the repro above, at the default 8192 and a source producing 10 rows/second, the first batch would appear after roughly 14 minutes. Not a regression in this cycle, though. I checked 52.0.0: the `coalesce_batches` optimizer rule there had no boundedness handling either, so the external `CoalesceBatchesExec` it inserted after `RepartitionExec` had the same blind spot. #18782 moved coalescing inside the operator, which made a longstanding gap structural rather than introducing it. So holding the release would not be protecting anyone from something newly broken. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
