[
https://issues.apache.org/jira/browse/SPARK-60967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60967:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Deserialize collected values of user-defined types, and give their dtypes, in
> Spark Connect Rust as PySpark does
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60967
> URL: https://issues.apache.org/jira/browse/SPARK-60967
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, a value of a user-defined type that a DataFrame collects
> is deserialized by its UDT, and {{dtypes}} gives the UDT's
> {{simpleString()}}. With {{df = spark.sql("SELECT array(1.5D, 2.0D) AS
> a").select(array_to_vector("a").alias("vec"))}} of {{pyspark.ml.functions}},
> {{df.first().vec}} is {{DenseVector([1.5, 2.0])}} and {{df.dtypes}} is
> {{[('vec', 'vector')]}} in PySpark 4.3.0.
> The Rust-backed Python client gives the value of the UDT's SQL type, a
> {{Row}}, and {{[('vec', 'udt')]}}, though the field type of {{df.schema}} is
> the {{VectorUDT}}. Deserialize the collected values of a user-defined type,
> and give the dtypes of their UDTs, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
