galaxytemple opened a new pull request, #27060: URL: https://github.com/apache/datafusion/pull/27060
## Which issue does this PR close? - Closes #26162. ## Rationale for this change In #26162, running `ORDER BY b, a` over a multi-partition subquery with `ORDER BY b LIMIT n` produces incorrect ordering on `a`. Pushing `[b, a]` down removed the outer `SortExec`, but SPM only merges on `[b]`. As a result, `a` was not sorted in the final output. In the discussion, @jayzhan211 suggested two possible approaches: declining pushdown for SPM with fetch, or rewriting SPM's expressions when pushing through it. I chose `Ok(None)` (declining pushdown) because: - The subquery did not explicitly request the parent's sort. When a limit is present, forcing that sort on the entire dataset across partitions before a limit is unnecessary work. - Since `pushdown_requirement_to_children` only adds requirements to children, I think a logic to rewrite SPM adds extra complexity. ## What changes are included in this PR? - Return `Ok(None)` from `pushdown_requirement_to_children` when `plan` is `SortPreservingMergeExec` and has a `fetch`. - Add an SLT regression query in `limit.slt`. ## What is the testing strategy for this PR? - query on a partitioned table in `limit.slt` reproducing #26162. ## Are there any user-facing changes? Yes, bug fix: queries like `ORDER BY b, a` over a multi-partition subquery with `ORDER BY b LIMIT n` now return correctly ordered results on `a`. No API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
