discord9 opened a new pull request, #23800:
URL: https://github.com/apache/datafusion/pull/23800
## Which issue does this PR close?
- Closes #23799.
## Rationale for this change
An operator-level fetch on a plan with multiple output partitions is applied
to
each partition. It can reduce work, but it does not satisfy a global limit.
The
LimitPushdown rule could conflate those scopes and remove the remaining
global
enforcement, producing too many rows or composing global and local limits
incorrectly.
## What changes are included in this PR?
- Track whether a pushed requirement still needs global enforcement.
- Preserve or tighten per-partition fetches as early-stop hints without
treating
them as global limits.
- Materialize a `CoalescePartitionsExec` or `SortPreservingMergeExec`
boundary
before a multi-partition global requirement is considered satisfied.
- Keep OFFSET and per-partition `LocalLimitExec` semantics separate from the
global fetch.
- Ignore no-op global limits without changing pending limit state.
## Are these changes tested?
Yes. Physical optimizer regression tests cover:
- fetch-capable and unfetchable multi-partition inputs;
- LIMIT, OFFSET + LIMIT, and OFFSET-only plans;
- an existing smaller per-partition fetch;
- Union input fan-out;
- global limits above multi-partition local limits;
- no-op global limits; and
- fetch absorption at HashJoin without leaking fetch to its inputs.
The complete `physical_optimizer::limit_pushdown` integration test module and
workspace clippy were run from the final PR head.
## Are there any user-facing changes?
Yes. Queries using these physical plan shapes now preserve the requested
global
LIMIT/OFFSET rather than returning too many rows or over-restricting results.
There are no public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
