kumarUjjawal opened a new pull request, #26763: URL: https://github.com/apache/datafusion/pull/26763
## Which issue does this PR close? Closes #23534. This follows the closed, unmerged #23744. ## Rationale for this change A grouped query over `ORDER BY sort_key OFFSET 1` can return `sorted_first` instead of `physical_second`. A projection hides the sort key, so the physical optimizer drops the sort even though it determines which rows the offset retains. ## What changes are included in this PR? Preserve linked sorts through operators that maintain input order, including when a projection removes their ordering metadata. Remove superseded sorts when enforcing a new ordering, and stop sort removal at explicit limits and operators with embedded fetch limits. Add SQL and physical optimizer regressions for projections, intervening filters, local and global limits, replacement sorts, and optimizer idempotence. An execution test verifies that an embedded filter limit still selects the correct rows after one or two optimizer passes. ## What is the testing strategy for this PR? - The ordered-offset SQL regression and embedded-fetch execution regression fail without their respective fixes. - The full extended suite passes: 12,987 Rust tests and 542 SQL files. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and the complete `./dev/rust_lint.sh` suite pass. - Compared `physical_plan_struct_join_agg_sort` and `physical_sorted_union_order_by_10_int64` against main at `350ca76505c9513d16e7dd9910bcd9bde2c911ed`, using 30 samples each. Neither benchmark showed a statistically significant performance change. ## Are there any user-facing changes? Queries retain the rows selected by ordered LIMIT/OFFSET when a projection hides the sort key. No public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
