Hyukjin Kwon created SPARK-61005:
------------------------------------
Summary: Create the UDFs of udf of the Python client as PySpark
does
Key: SPARK-61005
URL: https://issues.apache.org/jira/browse/SPARK-61005
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{udf(f, returnType)}} without {{useArrow}}
({{_create_py_udf}}) makes an Arrow-optimized UDF ({{SQL_ARROW_BATCHED_UDF}})
when the active session enables
{{spark.sql.execution.pythonUDF.arrow.enabled}}, which the 4.3.0 server does by
default, and infers the eval type of a function with type hints
({{infer_eval_type_for_udf}}): a function of {{pd.Series}} makes a pandas UDF
({{SQL_SCALAR_PANDAS_UDF}}), and one of {{pa.Array}} an Arrow UDF
({{SQL_SCALAR_ARROW_UDF}}). It returns the function wrapper of the UDF
({{UserDefinedFunction._wrapped}}), with the {{__name__}} and {{__doc__}} of
{{f}} and the {{func}}, {{returnType}}, {{evalType}}, {{deterministic}},
{{asNondeterministic}} and {{_unwrapped}} of the UDF.
The Rust-backed Python client always makes a {{SQL_BATCHED_UDF}}, so the
results differ from those of PySpark: for {{udf(lambda x: str(x), "int")}},
PySpark 4.3.0 returns {{0}} for {{0}}, as Arrow converts the string, and the
client {{None}}. It also returns the {{UserDefinedFunction}} rather than its
wrapper, which has no {{__name__}} or {{__doc__}}. Create UDFs as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
