Hyukjin Kwon created SPARK-60959:
------------------------------------
Summary: Keep the values of a collected Row, and compare and
pickle VariantVal, in Spark Connect Rust as PySpark does
Key: SPARK-60959
URL: https://issues.apache.org/jira/browse/SPARK-60959
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark, a Row is a tuple that holds the same Python objects for as long as
it lives. The Rust-backed Python client makes new Python objects from a
collected row's values each time they are read, so these PySpark 4.3.0 results
differ:
* For {{r = spark.sql("SELECT double('NaN') x").first()}}, {{r == r}} is
{{True}} in PySpark, as a tuple compares its own elements by identity first;
the client gives {{False}}, and {{hash(r)}} can change between calls. The same
holds for a row of a VARIANT value, as the client's {{VariantVal}} has no
{{__eq__}}.
* {{pickle.dumps(spark.sql("SELECT parse_json('1') v").collect())}} works in
PySpark; the client raises {{TypeError: cannot pickle
'pyspark.sql.types.VariantVal' object}}.
Keep the values of a collected row once read, and make {{VariantVal}} compare
and pickle as in PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
