guan404ming opened a new pull request, #59580: URL: https://github.com/apache/spark/pull/59580
### What changes were proposed in this pull request? This PR addresses [SPARK-60188](https://issues.apache.org/jira/browse/SPARK-60188), a sub-task of the transpilation SPIP [SPARK-54783](https://issues.apache.org/jira/browse/SPARK-54783). `_get_transpilers` in `python/pyspark/sql/transpile.py` now emits a `UserWarning` when `spark.sql.experimental.optimizer.pyTranspilers` names a transpiler that is not in `AbstractTranspiler.varieties`. The warning lists the unknown names and the registered ones. The unknown names are still skipped, as before. The PR also adds a unit test in `test_udf_transpile_unit.py` that covers a known name, a mix of known and unknown names, and only unknown names. ### Why are the changes needed? `_get_transpilers` silently drops names that are not registered, so a typo such as `catalist` turns transpilation off with no signal. `_transpile_func` calls `_get_transpilers` when the UDF is constructed, so the warning shows at UDF construction, next to the existing transpilation warnings. ### Does this PR introduce _any_ user-facing change? Yes. When `spark.sql.experimental.optimizer.pyTranspilers` contains an unregistered name, Python UDF construction with transpilation on now emits a warning such as: ``` UserWarning: Ignoring unknown transpilers ['catalist'] in spark.sql.experimental.optimizer.pyTranspilers; registered transpilers are ['catalyst'] ``` Before, no warning was emitted. The set of transpilers that run does not change. The conf is experimental and unreleased. ### How was this patch tested? - Added `UDFTranspileUnitTests.test_get_transpilers_warns_on_unknown_names`. It has not been run locally because no JVM-backed Spark build is available here. It needs to pass in CI. - `ruff check` and `ruff format --check` with ruff 0.14.8 (pinned in `dev/requirements.txt`) on the changed files: passed. - Called `_get_transpilers` with a stub session and no JVM for `catalyst`, `catalyst,catalist`, `catalist`, and an empty value: it warned only for `catalist` and returned the expected transpilers. - mypy did not run. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
