cool9850311 commented on issue #27457:
URL: https://github.com/apache/datafusion/issues/27457#issuecomment-6953280916

   I have a fix for this, in #27532 as a draft, and one question: which of two 
guards would
   you want? They are the two commits there.
   
   **(A)** `apply_limit` sets the fetch only on a `SortExec` with one output 
partition — what
   this issue suggests.
   
   **(B)** It sets the fetch on a multi-partition `SortExec` as well, but only 
when the limit
   came from a `SortPreservingMergeExec` whose ordering covers the window's 
`PARTITION BY`.
   
   Both fix every wrong-result shape I can find; (B) just changes fewer plans. 
The difference
   shows up on plans that are already sound: when the merge above the window 
orders the
   window's partitions apart, a row the window computed a wrong value for lost 
to `fetch`
   rows that all reach that merge before it, so it is past the merge's own 
fetch and never
   emitted.
   
   `datafusion/core/benches/topk_repartition.rs` is one of those plans. It runs 
with
   `target_partitions = 4`, so the `SortExec` below the window has four output 
partitions:
   (A) takes its fetch away and it becomes a full sort, (B) leaves the fetch in 
place.
   
   | `LIMIT` | `main` | with (A) | with (B) |
   |---|---|---|---|
   | 10 | 844 µs | **13.2 ms** | 858 µs |
   | 1,000 | 3.48 ms | **13.0 ms** | 3.49 ms |
   | 10,000 | 28.3 ms | 13.6 ms | 28.1 ms |
   | 100,000 | 123 ms | 30.6 ms | 127 ms |
   
   Medians of the `enabled` arm (the default config), `release-nonlto`, 10 
samples, 3 s
   measurement. Each version measured in its own run with the machine otherwise 
idle; runs
   taken back to back under sustained load came out 10-30% high, so I discarded 
those. (A)
   is 16x slower at `LIMIT 10` and 3.7x at `LIMIT 1,000`, and 2.1x and 4.0x 
*faster* at the
   two larger limits — a TopK that keeps a large fraction of its input stops 
paying off at
   all, with the crossover near `LIMIT 4,000` on this data.
   
   The wrong results, measured on `120049489` and on each commit, 20 runs each:
   
   | shape | `main` | with (A) | with (B) |
   |---|---|---|---|
   | bare `LIMIT`, `TopKRepartition` on (this issue) | 20/20 wrong | 0/20 | 
0/20 |
   | bare `LIMIT`, `TopKRepartition` off (this issue) | 20/20 wrong | 0/20 | 
0/20 |
   | `PARTITION BY a, b` under `ORDER BY a` | 20/20 wrong | 0/20 | 0/20 |
   | the same, with a constant leading the `ORDER BY` | 20/20 wrong | 0/20 | 
0/20 |
   
   The bottom two are shapes I hit while testing; each returns a row whose 
`row_number()` is
   1 where the full result says 2. Reproducers for all four are in 
`window_limits.slt`.
   
   So: (A) now with that regression and (B) as a follow-up, or (B) straight 
away? Dropping
   the second commit gives (A). Either is fine by me — I just did not want to 
pick the larger
   change for you.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to