alamb commented on issue #9792: URL: https://github.com/apache/arrow-datafusion/issues/9792#issuecomment-2037330732
Cross posting from https://github.com/apache/arrow-datafusion/pull/9815 as I am not sure about this proposal It seems like adding a limit to `CoalesceBatches` seems like somewhat of a workaround for a limit in StreamingTableExec -- it seems like if we handled the limit in `StreamingTableExec` then 1. It could be more efficient as the `StreamingTableExec `could stop as soon as the limit was hit 2. We would not need any changes to `CoalesceBatches` ``` EXPLAIN SELECT c1, c2, c3 FROM sink_table WHERE c3 > 0 LIMIT 5; ---- logical_plan Limit: skip=0, fetch=5 --Filter: sink_table.c3 > Int16(0) ----TableScan: sink_table projection=[c1, c2, c3]. <--- Why not apply the LIMIT here? physical_plan GlobalLimitExec: skip=0, fetch=5 --CoalescePartitionsExec ----CoalesceBatchesExec: target_batch_size=8192 ------FilterExec: c3@2 > 0 --------RepartitionExec: partitioning=RoundRobinBatch(3), input_partitions=1 ----------StreamingTableExec: partition_sizes=1, <---if the LIMIT was applied here there would be no need to apply it later? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
