HyukjinKwon opened a new pull request, #190: URL: https://github.com/apache/spark-connect-rust/pull/190
### What changes were proposed in this pull request? [SPARK-60959](https://issues.apache.org/jira/browse/SPARK-60959): rows and `VariantVal` behave as in PySpark 4.3.0. - **Collected rows keep their values.** A row that the server returned now makes the Python objects of its values when they are first read and keeps them, as a PySpark row is a tuple of them. So: - `row[0] is row[0]`, and `row.x is row[0]`; - a row equals itself, as a tuple compares its own elements by identity first, even with a NaN or a VARIANT value; - its hash stays the same. The values are made all at once on first read, as PySpark's row holds them all. - **`VariantVal` pickles** as `VariantVal(value, metadata)`, which PySpark's own class also takes. So a collected row of VARIANT values pickles, and unpickles on an official worker too. `VariantVal` has no `__eq__`, as in PySpark, whose class compares by identity. ### Why are the changes needed? A row made new Python objects of its values each time they were read, so: - `r == r` was `False` for `r = spark.sql("SELECT double('NaN') x").first()`, and for a row of a VARIANT value; - `hash(r)` could change between calls. `pickle.dumps(spark.sql("SELECT parse_json('1') v").collect())` also raised `TypeError: cannot pickle 'pyspark.sql.types.VariantVal' object`. ### Does this PR introduce _any_ user-facing change? Yes, these behave as in PySpark 4.3.0. ### How was this patch tested? - Against a local Spark 4.3.0 server, a probe of these rows gives the same results as official PySpark 4.3.0: - equality and hashes with a NaN and a VARIANT value; - value identity; - nested rows; - pickling of rows and of `VariantVal`. - A new offline test pickles a `VariantVal`, and a new live test checks the collected rows. Both also pass with official PySpark 4.3.0, and fail without this change. - The lib tests and every integration test, e2e included, against the local server (645 passed), the build of all tests with `wasm-udf`, the offline Python suites (648 passed, 8 live cases skipped), and the live drop-in and Arrow suites (64 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
