charliec05 opened a new pull request, #58400: URL: https://github.com/apache/spark/pull/58400
### What changes were proposed in this pull request? Cache Catalyst UDF category inference by AST node while lowering each transpilation variant. The cache is reset for every variant because parameter-category assumptions can differ between variants. The pull request also adds a regression test with a long addition expression and verifies that uncached category computations are bounded by the AST size. ### Why are the changes needed? Category inference recursively revisited the same subtrees while converting nested expressions. For a left-deep expression, this made transpilation quadratic in the expression size. Memoizing each node keeps inference linear without changing the generated Catalyst expression. ### Does this PR introduce _any_ user-facing change? No. It improves the performance of Python UDF transpilation only. ### How was this patch tested? - `build/sbt -Phive package` - `python/run-tests --testnames pyspark.sql.tests.test_udf_transpile_unit` - `dev/lint-python --ruff` - `dev/lint-python --compile --custom-pyspark-error` Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
