[
https://issues.apache.org/jira/browse/SPARK-60957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60957.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 179
[https://github.com/apache/spark-connect-rust/pull/179]
> Import pyspark.sql.conversion, and PandasUDFType from pyspark.sql.functions,
> in Spark Connect Rust as in PySpark
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60957
> URL: https://issues.apache.org/jira/browse/SPARK-60957
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> These imports work in PySpark 4.3.0 and fail in the Rust-backed Python client:
> * {{import pyspark.sql.conversion}} raises {{ImportError: cannot import name
> '_has_type' from 'pyspark.sql.types'}}: the vendored module needs
> {{_has_type}}, which the client's {{pyspark.sql.types}} lacks.
> * {{from pyspark.sql.functions import PandasUDFType}} raises {{ImportError}}
> when {{pyspark.sql.pandas.functions}} was imported first: the re-export of
> {{PandasUDFType}} and {{ArrowUDFType}} at the end of
> {{pyspark.sql.functions}} meets the half-imported
> {{pyspark.sql.pandas.functions}}, and its error is swallowed.
> Make both imports work as in PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
