mounikavas opened a new pull request, #59081: URL: https://github.com/apache/spark/pull/59081
### What changes were proposed in this pull request? This is a scoped slice of [SPARK-55214](https://issues.apache.org/jira/browse/SPARK-55214) (parent [SPARK-54783](https://issues.apache.org/jira/browse/SPARK-54783) / [SPIP](https://docs.google.com/document/d/1cHc6tiR4yO3hppTzrK1F1w9RwyEPMvaeEuL2ub2LURg/edit?usp=sharing)): lower Python `len(s)` on a string UDF argument to Catalyst `length`. String `+` / `*` already lower to `concat` / `repeat`. This PR only adds `len`. Other `len` arguments (numbers, lists, ...) still fall back to interpreted Python. NULL stays NULL, matching the existing unguarded-NULL caveat for `x + 1`. Remaining string ops (`upper` / `lower` / `strip` / slicing / ...) are out of scope here. ### Why are the changes needed? Python UDF `len(s)` is a common, cheap string operation that still paid the Python worker tax. Mapping it to native `length` is the smallest missing string op that can land before 4.3 RC. ### Does this PR introduce _any_ user-facing change? Only with the experimental flags already required for transpilation (`spark.sql.experimental.optimizer.transpilePyUDFs=true` and `spark.sql.ansi.enabled=true`): ```python strlen = udf(lambda x: len(x), "long") df.select(strlen(col("s"))) ``` Previously this stayed an interpreted Python UDF. Now it can run as Catalyst `length`. Default (flags off) is unchanged. ### How was this patch tested? Added `test_udf_transpile_string_len` in `python/pyspark/sql/tests/test_udf_transpile_unit.py` (empty / ASCII / concat / NULL, plus `len` on a long column falling back). Ran: ``` python/run-tests --testnames pyspark.sql.tests.test_udf_transpile_unit ``` ### Was this patch authored or co-authored using generative AI tooling? Yes, co-authored using Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
