andygrove commented on issue #7365: URL: https://github.com/apache/datafusion-comet/issues/7365#issuecomment-6707952019
A smaller first step fixes the failure without the per-partition limit. Build a stage for `CometCollectLimitExec` the way `takeOrderedStage` builds one for top-K: the stage reads the child's whole output through a single-partition exchange, and its block is `CometExecUtils.getLimitNativePlan(child.output, limit, offset)`, Comet's final limit plan. A query that ends in `LIMIT`, with or without `OFFSET`, then runs on the cluster, though every row of the child still crosses the exchange. The per-partition limit and dropping the special cases, as described above, can follow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
