berkaysynnada commented on issue #9792: URL: https://github.com/apache/arrow-datafusion/issues/9792#issuecomment-2037363411
> Cross posting from #9815 as I am not sure about this proposal > > It seems like adding a limit to `CoalesceBatches` seems like somewhat of a workaround for a limit in StreamingTableExec -- it seems like if we handled the limit in `StreamingTableExec` then > > 1. It could be more efficient as the `StreamingTableExec `could stop as soon as the limit was hit > 2. We would not need any changes to `CoalesceBatches` > We can of course inform the `StreamingTableExec` about how many rows it needs to read for a `Limit` query. It enhances our position but still `CoalesceBatches` may need a fetch value. `CoalesceBatches` fetch count is directly related with the `Limit` operator above it. However, `StreamingTableExec` have to read more value than that limit fetch count. Am I missing something? Limit: fetch=5 --CoalesceBatches: target_size=1000 ----Aggregate: to produce 5 rows, needs 500 rows ------StreamingTableExec Assuming the plan above, I think `CoalesceBatches` must know fetch count. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
