Hyukjin Kwon created SPARK-60957:
------------------------------------
Summary: Import pyspark.sql.conversion, and PandasUDFType from
pyspark.sql.functions, in Spark Connect Rust as in PySpark
Key: SPARK-60957
URL: https://issues.apache.org/jira/browse/SPARK-60957
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
These imports work in PySpark 4.3.0 and fail in the Rust-backed Python client:
* {{import pyspark.sql.conversion}} raises {{ImportError: cannot import name
'_has_type' from 'pyspark.sql.types'}}: the vendored module needs
{{_has_type}}, which the client's {{pyspark.sql.types}} lacks.
* {{from pyspark.sql.functions import PandasUDFType}} raises {{ImportError}}
when {{pyspark.sql.pandas.functions}} was imported first: the re-export of
{{PandasUDFType}} and {{ArrowUDFType}} at the end of {{pyspark.sql.functions}}
meets the half-imported {{pyspark.sql.pandas.functions}}, and its error is
swallowed.
Make both imports work as in PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
