geoffreyclaude opened a new pull request, #26736:
URL: https://github.com/apache/datafusion/pull/26736
## Which issue does this PR close?
None.
## Rationale for this change
Sort pushdown can make an `ORDER BY ... LIMIT` query fail plan validation
when a source guarantees the requested ordering but cannot apply the row limit
itself.
A partition-preserving TopK keeps up to `k` rows **per partition**. Removing
its sort must retain that local limit. The standalone exact-pushdown path
always inserted `GlobalLimitExec` instead, which requires a single input
partition. With a four-partition source, the resulting plan fails with a
`SinglePartition` distribution error.
The direct merge/sort path already uses a local fallback. A projection
between the merge and local sort makes the local sort reach the standalone path
and exposes the bug.
## What changes are included in this PR?
When the source cannot absorb the fetch, choose `LocalLimitExec` if the
removed sort preserves partitions, and `GlobalLimitExec` otherwise.
For a global TopK of three rows, the corrected plan retains both levels of
selection:
```text
Merge: keep 3 overall
Projection: a, a + 1
LocalLimit: keep 3 per partition
Ordered source: 4 partitions
```
## What is the testing strategy for this PR?
An executable regression in `sort_pushdown.slt` uses a four-partition source
that accepts exact sort pushdown and declines fetch pushdown. SQL supplies the
intervening computed projection. The test checks identical returned rows with
sort pushdown disabled and enabled, verifies that all twelve locally selected
rows remain when there is no outer limit, and checks that `EXPLAIN` retains a
local limit below the projection and the global merge above it.
The enabled query fails plan validation before the fix.
Validation: the focused SQL file, existing `PushdownSort` integration tests,
formatting, and Clippy with the standard CI feature set
(`avro,integration-tests,extended_tests`).
## Are there any user-facing changes?
Affected queries execute successfully while preserving the per-partition and
overall row limits. No public API changes.
Developed with AI assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
