HyukjinKwon commented on PR #58142: URL: https://github.com/apache/spark/pull/58142#issuecomment-5349621499
Follow-up on review point 3 (the O(n^2) bounded-frame refold): rather than leave it as a TODO, I implemented the running-buffer reuse in `fd77dd5`. The bounded-frame handler now keeps a running buffer: when consecutive per-row frames share the same lower bound and only grow on the right (e.g. `rowsBetween(unboundedPreceding, currentRow)`), it extends the buffer with one `reduce` per newly-included row -- O(n) overall instead of O(n^2). Frames whose lower bound advances (a row leaves the window, which `reduce` can't subtract) or that shrink still refold from `zero`, which is unavoidable with a reduce-only aggregator. The aggregator's input columns are also materialized once per partition now instead of re-sliced per row. Added `test_window_bounded_preceding_frame` (`rowsBetween(-3, currentRow)`) to exercise both branches in one window -- the clamped/growing prefix (extend) and the advancing lower bound (refold). Local: classic 29/29 and Connect parity 27/27 pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
