stuhood commented on issue #27109: URL: https://github.com/apache/datafusion/issues/27109#issuecomment-6664477800
@2010YOUY01 : That's correct. The concern that I had with adjusting the batch size is that when it comes to sizing the batches, you're either: 1. sizing them "smaller than ideal" (in which case you have more interpretation overhead per batch, and still end up pulling more than you need on the final batch) 2. sizing them "larger than ideal" (in which case you're pulling more than you need to from the expensive probe scan) With disabling of coalescing, the expensive scan just drives the batch size decision directly, and can therefore execute the exact minimum number of operations it can. If we think that a `batch_size` option is more general, then I'm fine with that... but it wouldn't be optimal for our case. On the other hand, I could imagine that `PushDownLimit`/`LimitPushdown` could potentially also push a `batch_size` down a bit below a successfully pushed limit, potentially? Maybe that would be a pessimization though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
