HyukjinKwon commented on PR #58142: URL: https://github.com/apache/spark/pull/58142#issuecomment-5349780201
Thanks @dongjoon-hyun! **1. Bounded-frame quadratic Arrow->Python conversion** — already resolved in `fd77dd5` (pushed before this review): the column materialization is hoisted out of the per-row loop (`value_cols = [concatenated.column(o).to_pylist() for o in data_offsets]` once per partition, indexed by row). That commit also reuses a running buffer so the `reduce` calls are O(n) for fixed-lower-bound growing frames (e.g. the default RANGE unbounded-preceding..current-row), leaving only genuinely-sliding frames to refold. **2. `_LEGACY_ERROR_TEMP_1214` for the mixed Python window case** — fixed in `a46daa1`. Since `df.select(udaf(A())(v).over(w), pandas_agg(v).over(w))` is user-reachable, I replaced it with a named `UNSUPPORTED_FEATURE.MULTIPLE_PYTHON_UDF_TYPES_IN_WINDOW`: > Cannot use Python user-defined functions of different types together over a single window: `Mean`, `pandas_mean`. Use a separate window specification for each. It names the offending functions and suggests separate windows, avoiding the self-contradictory "different window function type" message (both are `WindowFunctionType.Python`). `test_window_mixed_python_udf_rejected` now asserts the condition. Local: classic 29/29 and Connect parity 27/27 pass. On the **nit** (splitting the `udaf` relocation into its own PR) — good point; I'll follow up on that separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
