calvinchengx commented on issue #24044: URL: https://github.com/apache/datafusion/issues/24044#issuecomment-5229095294
Size-or-deadline sounds right to me, and I agree a real timer beats poll counting: poll frequency tells you about the scheduler rather than about the data. One number in support of the deadline half, since it is easy to underweight. I measured a file sink that commits once per input batch, fed by a source emitting 1 to 2 rows per batch: 7,667 files for 7,667 rows in 8 seconds, about 958 files per second. That is what no coalescing costs a couple of operators downstream. Correct, but not something you would point at object storage. The other thing worth settling early is the default maximum batch age, not just the knob. If the default is long, a slow stream still looks stalled and it is this same report with a smaller constant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
