cool9850311 commented on issue #27457: URL: https://github.com/apache/datafusion/issues/27457#issuecomment-6952365740
@jayzhan211 I have a fix for this, and one question: which of two guards would you want? **(A)** Set the fetch only on a `SortExec` with one output partition — what the issue suggests. **(B)** Set it on a multi-partition `SortExec` too, but only when the limit came from a `SortPreservingMergeExec` whose ordering covers the window's `PARTITION BY`. Both fix every wrong-result shape I can find; (B) just changes fewer plans. The difference shows up on plans that are already sound: when the merge above the window orders the window's partitions apart, a row the window computed a wrong value for lost to `fetch` rows that all reach that merge before it, so it is past the merge's own fetch and never emitted. `datafusion/core/benches/topk_repartition.rs` is one of those plans. It runs with `target_partitions = 4`, so the `SortExec` below the window has four output partitions: (A) takes its fetch away and it becomes a full sort, (B) leaves the fetch in place. | `LIMIT` | `main` | with (A) | with (B) | |---|---|---|---| | 10 | 907 µs | **15.3 ms** | 836 µs | | 1,000 | 3.66 ms | **15.5 ms** | 3.56 ms | | 10,000 | 28.3 ms | 14.7 ms | 28.7 ms | | 100,000 | 128 ms | 32.3 ms | 123 ms | Medians, `release-nonlto`, 10 samples, 3 s measurement, default config. Two runs of one build differ by up to 13% at the two largest limits, so (B) is unchanged from `main`. Above roughly `LIMIT 4,000` (A) is *faster*, because a TopK that keeps a large fraction of its input stops paying off at all. The wrong results, which (A) and (B) fix alike: | shape | on `main` | with either guard | |---|---|---| | bare `LIMIT`, `TopKRepartition` on (this issue) | 20/20 wrong | 0/20 | | bare `LIMIT`, `TopKRepartition` off (this issue) | 20/20 wrong | 0/20 | | `PARTITION BY a, b` under `ORDER BY a` | 20/20 wrong | 0/20 | | the same, with a constant leading the `ORDER BY` | 19/20 wrong | 0/20 | The bottom two are shapes I hit while testing. Each returns a row whose `row_number()` is 1 where the full result says 2. Reproducers will be in the PR either way. So: (A) now with that regression and (B) as a follow-up, or (B) straight away? Either is fine by me — I just did not want to pick the larger change for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
