discord9 opened a new pull request, #23359: URL: https://github.com/apache/datafusion/pull/23359
## Which issue does this PR close? - Closes none. ## Rationale for this change `SortPreservingMergeExec` supports `fetch`, but its statistics currently pass through child statistics unchanged. This can make cost-based optimizers overestimate top-k merge outputs. For example, a `SortPreservingMergeExec(fetch=1)` side can still look as large as its input to `JoinSelection`, causing hash join build/probe choices to miss the small top-k side. Other fetch-aware operators such as `SortExec`, `CoalescePartitionsExec`, and limit execs already cap their statistics by fetch. This PR aligns `SortPreservingMergeExec` with that behavior. ## What changes are included in this PR? - Cap `SortPreservingMergeExec` statistics with `Statistics::with_fetch(self.fetch, 0, 1)`. - Preserve no-op statistics for `fetch = None` and `skip = 0`. - Avoid scaling byte-size estimates by `0.0` when input row count is unknown; use absent byte-size stats instead. - Add regression tests for SPM fetch statistics and for `JoinSelection` swapping an SPM(fetch=1) side to the hash join build side. ## Are these changes tested? Yes. - `cargo test -p datafusion-common test_with_fetch` - `cargo test -p datafusion-physical-plan sort_preserving_merge` - `cargo test -p datafusion join_selection --test core_integration` - `cargo check -p datafusion --test core_integration` - `cargo fmt --all -- --check` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No user-facing API changes. This improves optimizer statistics for fetch-limited sort-preserving merge plans. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
