xudong963 opened a new pull request, #24809:
URL: https://github.com/apache/datafusion/pull/24809
## Which issue does this PR close?
- Closes #24807.
## Rationale for this change
Reoptimizing an already optimized physical plan could silently remove a
pushed-down `LIMIT` stored on `SortPreservingMergeExec` or
`CoalescePartitionsExec`. This could make a query return more rows than
requested.
## What changes are included in this PR?
- Consume a removed `fetch` when a replacement merge operator is inserted.
- Remember the outermost removed fetch-capable distribution operator and
rebuild it around the optimized child when no replacement consumes its limit.
- Preserve the minimum effective fetch across nested distribution operators.
- Carry a fetched ordered merge's limit to a replacement sort when
order-preserving variants are removed.
## What is the testing strategy for this PR?
Added targeted physical optimizer regression tests covering:
- reoptimizing a fetched `SortPreservingMergeExec`;
- reoptimizing a fetched `CoalescePartitionsExec`;
- moving a fetched ordered merge's limit to a replacement sort;
- updating an existing fetched single-partition merge snapshot that
previously
encoded the incorrect removal of its limit.
Validated with:
```text
# Fails on e4cf35cbc (current main before this patch): both regression tests
fail
cargo test -p datafusion --test core_integration
physical_optimizer::enforce_distribution::preserve_fetch_when_reoptimizing
# Passes with this patch
cargo test -p datafusion --test core_integration
physical_optimizer::enforce_distribution::preserve_fetch_when_reoptimizing
cargo test -p datafusion --test core_integration
physical_optimizer::enforce_distribution::move_fetch_to_replacement_sort
cargo test -p datafusion --test core_integration
physical_optimizer::enforce_distribution::test_replace_order_preserving_variants_with_fetch
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples
--exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib
--tests --bins --features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
```
## Are there any user-facing changes?
Queries preserve their requested global limit when physical distribution
requirements are optimized more than once. There are no changes to SQL behavior
other than fixing the incorrect result, and no changes to documented public
APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
