goutamadwant commented on issue #24044: URL: https://github.com/apache/datafusion/issues/24044#issuecomment-5229019257
#24193 is the short-term correctness fix. unbounded repartitions bypass the completion-driven SharedCoalescer, ensuring partial batches are emitted while bounded inputs retain the existing size-based coalescing. @jayzhan211 as per your comment too, i feel longer term, restoring unbounded coalescing with a size-or-deadline policy per output partition—flush at `batch_size`, a configurable maximum batch age, or completion. A real timer provides deterministic progress; poll-count is scheduler-dependent, while watermark/barrier flushing can remain an optional extension if DataFusion later introduces control-message propagation would help. Tx! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
