Theodus opened a new pull request, #24273: URL: https://github.com/apache/datafusion/pull/24273
## Which issue does this PR close? - Closes #24272. ## Rationale for this change Queries that filter ordered data and apply `ORDER BY ... LIMIT` can return incorrect rows when the physical plan contains a fetched `SortPreservingMergeExec`. When the fetch is pushed into an ordered Parquet scan, `LimitPushdown` currently treats it as order-insensitive. This allows Parquet's limit-based row-group pruning to discard an earlier partially matched row group in favor of a later fully matched row group. The scan can therefore return later rows instead of the first rows in the requested ordering. For example, a query shaped like: ```sql SELECT key FROM table WHERE key >= 1 ORDER BY key ASC LIMIT 5; ``` may skip the row group containing the lowest matching values and return values from a later row group. ## What changes are included in this PR? - Treat a fetched `SortPreservingMergeExec` as order-sensitive during limit pushdown. - Propagate `preserve_order = true` when pushing its fetch into the underlying scan. - Add a regression test verifying that an ordered Parquet scan receives both `limit = Some(5)` and `preserve_order = true`. This prevents order-insensitive Parquet limit pruning from changing which rows are eligible for an ordered limit. ## Are these changes tested? Yes. The new `preserves_order_when_pushing_fetch_from_sort_preserving_merge` regression test fails without this change because limit pushdown resets the scan's `preserve_order` flag to `false`. ## Are there any user-facing changes? Yes. Ordered queries with a pushed-down limit now return the earliest matching rows instead of rows from a later Parquet row group. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
