[
https://issues.apache.org/jira/browse/SPARK-60999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60999.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 204
[https://github.com/apache/spark-connect-rust/pull/204]
> Create the UDTFs of udtf and arrow_udtf of the Python client as PySpark does
> ----------------------------------------------------------------------------
>
> Key: SPARK-60999
> URL: https://issues.apache.org/jira/browse/SPARK-60999
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{arrow_udtf}} creates a PyArrow-native UDTF, of eval
> type {{SQL_ARROW_UDTF}} (302), whose {{eval}} takes {{pyarrow}} arrays and
> returns {{pyarrow}} tables. {{udtf}} creates one of {{SQL_TABLE_UDF}} (300),
> or of {{SQL_ARROW_TABLE_UDF}} (301) when {{useArrow}} is True, or is not
> given and {{spark.sql.execution.pythonUDTF.arrow.enabled}} of the session is
> true. Both are not deterministic, unless {{asDeterministic()}} makes them so.
> The Rust-backed Python client creates an {{arrow_udtf}} of
> {{SQL_ARROW_TABLE_UDF}}, so the worker runs it as a Python UDTF whose
> {{eval}} takes Python values, and every {{arrow_udtf}} fails. For example,
> PySpark returns {{Row(x=2)}} for {{PlusOne(lit(1))}} of
> {code:python}
> @arrow_udtf(returnType="x long")
> class PlusOne:
> def eval(self, x):
> yield pa.table({"x": pc.add(x, 1)})
> {code}
> where the client raises {{PYTHON_EXCEPTION}}. The client also ignores
> {{spark.sql.execution.pythonUDTF.arrow.enabled}}, and creates deterministic
> UDTFs. Create the UDTFs as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
