[
https://issues.apache.org/jira/browse/SPARK-60959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60959:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Keep the values of a collected Row, and compare and pickle VariantVal, in
> Spark Connect Rust as PySpark does
> ------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60959
> URL: https://issues.apache.org/jira/browse/SPARK-60959
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, a Row is a tuple that holds the same Python objects for as long
> as it lives. The Rust-backed Python client makes new Python objects from a
> collected row's values each time they are read, so these PySpark 4.3.0
> results differ:
> * For {{r = spark.sql("SELECT double('NaN') x").first()}}, {{r == r}} is
> {{True}} in PySpark, as a tuple compares its own elements by identity first;
> the client gives {{False}}, and {{hash(r)}} can change between calls. The
> same holds for a row of a VARIANT value, as the client's {{VariantVal}} has
> no {{__eq__}}.
> * {{pickle.dumps(spark.sql("SELECT parse_json('1') v").collect())}} works in
> PySpark; the client raises {{TypeError: cannot pickle
> 'pyspark.sql.types.VariantVal' object}}.
> Keep the values of a collected row once read, and make {{VariantVal}} compare
> and pickle as in PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
