HyukjinKwon opened a new pull request, #327: URL: https://github.com/apache/spark-connect-rust/pull/327
### What changes were proposed in this pull request? [SPARK-61138](https://issues.apache.org/jira/browse/SPARK-61138): Two behaviors of `VariantVal` now match PySpark 4.3.0: - **`str` of a `VariantVal` is its JSON**, as PySpark's `VariantVal.__str__` (`VariantUtils.to_json`, in UTC, as `toJson()`). Its `repr` is still that of its bytes. - **`createDataFrame` rejects rows that are `VariantVal`s** with `PySparkValueError` "Rows cannot be of type VariantVal". This holds with a struct schema, without a schema, and with a schema that is not a struct, such as `"variant"`. PySpark's `VariantVal` has a `__dict__`, so a first item that is one is taken as a row, and `to_row` of `LocalDataToArrowConversion` rejects it. The client's `VariantVal`, a class of the Rust extension, has no `__dict__`, so the client took it as a single value. It is now taken as a row, as PySpark takes it. As with PySpark, rows whose first item is `None` are still rows of single values, so a `VariantVal` after that is a value. ### Why are the changes needed? - `str(v)` gave `VariantVal(b'...', b'...')`, not the JSON. - `createDataFrame([VariantVal.parseJson("2")], "v variant")` gave a DataFrame of one row, where PySpark raises. Apache Spark's `test_variant_type` failed on this client for both. ### Does this PR introduce _any_ user-facing change? Yes: - `str` of a `VariantVal` is its JSON. - `createDataFrame` of rows that are `VariantVal`s raises `PySparkValueError`, as with PySpark 4.3.0. ### How was this patch tested? - A new offline test checks: - `str` and `repr` of `VariantVal`s - the `PySparkValueError` of rows that are `VariantVal`s, of each kind of schema - the rows of a first `None` It passes with official PySpark 4.3.0, and fails without this change. - Apache Spark's variant tests of `test_parity_types.py`, with #326 (SPARK-61133), which lets the file be collected, run on this client against a local server, and all 3 pass, including `test_variant_type`. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2197 passed, 8 skipped), and the live drop-in and Arrow suites (182 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
