srielau commented on code in PR #58549:
URL: https://github.com/apache/spark/pull/58549#discussion_r4213736294
##########
python/pyspark/sql/udf.py:
##########
@@ -321,6 +328,7 @@ def _conf_is_true(key: str, default: Optional[str] = None)
-> bool:
@staticmethod
def _check_return_type(returnType: DataType, evalType: int) -> None:
+ _check_no_char_varchar(returnType, "Python UDF return types")
Review Comment:
Good catch. ``self.returnType`` for a DDL string like ``"char(3)"`` parses
then calls ``_check_return_type``, which raises ``PySparkNotImplementedError``
(``CHAR_VARCHAR_NOT_SUPPORTED_IN_PYTHON``). That was being swallowed by the
transpile ``except Exception`` and turned into a UserWarning.
Fixed by re-raising ``PySparkNotImplementedError`` before the fallback
warning. Classic-only coverage is in
``UDFTranspileUnitTests.test_udf_transpile_does_not_mask_char_varchar_ddl``
(transpile + ANSI, ordinary warnings, and ``warnings.simplefilter("error")``).
The existing ``test_udf_transpile_chained_comparison_falls_back`` still covers
a real transpile failure remaining a warning.
Connect is unchanged: its constructor has no transpile try around
``returnType``.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
