HyukjinKwon commented on PR #57804: URL: https://github.com/apache/spark/pull/57804#issuecomment-5234776854
Thanks @dongjoon-hyun for the thorough review. Addressed all points in 95ec08d: - **`f(g(x) + 1)` / `f(-g(x))`** (correctness): nested lifted UDFs inside a composite argument are now substituted before lifting, so no raw UDF is left inside a lambda (avoiding the SPARK-48706 mode). - **Pairwise `array_sort` comparator** (perf): `flatCells` moved into the sort argument (carried via `array_repeat`), so each comparison is O(1) instead of rebuilding the cells. - **Predicate alignment**: `isRewritableShape` now requires a result-type marker trait; `isElementwiseRewritableUDF` rejects named-argument and UDT UDFs. - **Key-form comparator dedup**: Python now runs n times, not 2n. - Removed the unused `lambdaExprIds` param, stray blank lines, and stale test comment. - **Connect parity test**: added `pyspark.sql.tests.connect.test_parity_udf_in_higher_order_function` and registered it in `modules.py`, per your suggestion. Verified: `ExtractPythonUDFFromLambdaSuite` (21 tests) and the end-to-end `pyspark.sql.tests.test_udf_in_higher_order_function` suite both pass locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
