Sunninsky commented on issue #27463:
URL: https://github.com/apache/datafusion/issues/27463#issuecomment-6903845086

   Thanks for the reproducer. Yes, #27410 covers this shape: the frames of all 
rows of the complete partitions of an input batch are computed in one pass and 
`NthValueEvaluator` evaluates them with a single `evaluate_ranges` call, so 
there is one `take` instead of one per partition.
   
   I ran your four queries unchanged with `datafusion-cli` built with 
`release-nonlto` from `main` at adffd19e2 (the merge base of both PRs), #27410 
at e82bd186e and #27466 at ed7a1f74f. `target_partitions = 1`, on a dedicated 
r7i.8xlarge node (Xeon Platinum 8488C), the process pinned to two cores. 
`elapsed_compute` of `BoundedWindowAggExec`, median of 6 runs (min and max are 
within 4% of the median):
   
   | Query | `main` | #27410 | #27466 |
   |---|---:|---:|---:|
   | 1. `last_value(s) IGNORE NULLS`, 10 carried columns | 14.7 s | 1.18 s | 
1.87 s |
   | 2. `last_value(s) IGNORE NULLS`, no carried columns | 12.1 s | 1.10 s | 
1.77 s |
   | 3. `last_value(i) IGNORE NULLS` on an int, 10 carried columns | 8.07 s | 
0.915 s | 1.58 s |
   | 4. `max(i)`, 10 carried columns | 9.37 s | 0.913 s | 5.86 s |
   
   The results of queries 1, 3 and 4 are identical in all three builds, 
compared on all 5,000,000 rows. For #27466 on query 1 I get 7.9x, in line with 
your 5.3 s -> 0.65 s.
   
   The two PRs overlap: both evaluate the complete partitions of a batch 
directly in `SortedSearch`, each through its own new provided methods 
(`evaluate_partitions` and `evaluates_frames_independently` in #27466, 
`evaluate_complete_partitions` and `evaluate_segments` in #27410), and a trial 
merge of #27466 into #27410 conflicts in five files. #27466 is the smaller 
change, so I am happy for it to go first, and I can rebase #27410 on it 
afterwards. #27410 would then shrink to what #27466 does not cover, mainly 
aggregates over all complete partitions in one pass (query 4), bulk frame 
bounds for moving and `RANGE` offset frames, and `row_number` / `rank` / `lag`. 
Does that order work for you, or would you split it differently?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to