Lordworms commented on PR #9815: URL: https://github.com/apache/arrow-datafusion/pull/9815#issuecomment-2035901450
> Thanks @Lordworms -- I took a quick look of this PR > > I am probably missing something obvious but I don't understand the need for the pushdown pass in the physical optimizer. > > If the usecase is to get a limit closer to `StreamingTableExec` then maybe we can pushing the `fetch` to the `CoalesceBatchesExec` rather than the `StreamingTableExec` ? actually it has been pushed to CoalesceBatchesExec, and then carries down to StreamingTableExec > > It seems to me that a limit in the StreamingTable exec can likely be implemented more efficiently, _and_ would already be handled by the existing Limit pushdown in the LogicalPlan. I agree I should add this logic in LogicalPlan Phases, I will save this one to draft and work on #9873 first. > > Maybe @berkaysynnada or @mustafasrepo have some more context Thanks for your review and Sorry for my immature design, I'll complete it later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
