avantgardnerio commented on PR #24035: URL: https://github.com/apache/datafusion/pull/24035#issuecomment-5197054712
> access to the window state for the single last row Yes, this is exactly what is required. > need access to the window state for the last N rows No, not for prefix scanning. (answer below) > e.g the HALO rows The HPC "halo" concept is a good fit for bounded preceding/following (surrounding cells, in 1D) but doesn't extend cleanly to "last row of every other partition." Regardless of the name, this PR doesn't take that approach - because although it works for SUM, and decomposes for AVG (sum+count), it fails by the time you get to arbitrary accumulators like approx_distinct. > add some sort of example / test Which is exactly where (the newly added) `test_prefix_scan_across_tasks_matches_single_bwag()` comes in. It shows that with only the accumulator state of the very last row of the entire partition, parallel partitioned prefix-scanned sums produce exactly the same results as a single partition BWAG. Given the observer exposes Accumulator::state() directly, any function that supports Accumulator::merge_batch - including approx_distinct - can be prefix-scanned the same way, with the downstream consumer merging sketches instead of adding scalars. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
