HyukjinKwon opened a new pull request, #154: URL: https://github.com/apache/spark-connect-rust/pull/154
### What changes were proposed in this pull request? [SPARK-60912](https://issues.apache.org/jira/browse/SPARK-60912): the Python client's `Row` now behaves as PySpark's `Row`, a `tuple` subclass, does. - `repr()` and `str()` give `Row(x=1, y='a')`, with the `repr()` of each value, and `<Row(1, 2)>` for a row without field names, so `print(df.collect())` prints `[Row(id=0)]` rather than `[[0]]`. - A row made in Python holds the values that it was made with, as they are, as PySpark's does. A row from the server converts its values on access, as before. The rest of the client converts the values of a row made in Python when it needs them, as it converts those of a tuple. - As the tuple of its values does, a row hashes, compares with rows and tuples regardless of field names, orders, slices, concatenates and repeats into tuples, reverses, and pickles. It pickles through `_create_row`, now PySpark's, so that field names survive, duplicate ones too, and the server's PySpark can unpickle it. - `Row("name", "age")` is a row class: calling it makes a row named by its values, and too many values raise `TOO_MANY_VALUES`. - A row without field names, such as `Row(1, 2)`, has no `__fields__`, its `asDict()` raises `CANNOT_CONVERT_TYPE`, and `in` tests its values. For a row with names, `in` tests the names. `asDict(recursive=True)` turns nested rows into dicts, also in lists and dicts. - The errors are PySpark's: a missing field raises `PySparkAttributeError` (`ATTRIBUTE_NOT_SUPPORTED`) as an attribute and `PySparkValueError` as an item, setting an attribute other than `__fields__`, which renames the fields, raises `READ_ONLY`, and mixing positional and keyword arguments raises `CANNOT_SET_TOGETHER`. `isinstance(row, tuple)` stays false, as a PyO3 class cannot subclass `tuple`. ### Why are the changes needed? A row printed as a list of its values in a Rust format: `print(spark.range(1).collect())` printed `[[0]]`, and a date printed as its day number and `None` as `NULL`. `Row("name", "age")("Alice", 11)` raised `TypeError`, as did `hash()`, `sorted()`, `pickle.dumps()`, slicing and concatenation, `Row(x=1) == (1,)` was false, `"x" in row` tested the values, `Row(1, 2).asDict()` returned a dict keyed by `_1` and `_2`, and missing fields raised a plain `AttributeError` and a `KeyError`. A row also turned its values into Spark values when made, so that a tuple became a list and other objects, such as a set, raised `TypeError`. ### Does this PR introduce _any_ user-facing change? Yes, `Row` now behaves as in PySpark Connect 4.3.0, except that it is not an instance of `tuple`. Two existing tests that expected the previous behavior, `in` over the values of a row with names, and `_create_row` reading the names of a row with names, now expect PySpark's. ### How was this patch tested? - Probes of 59 cases, covering making rows, the repr of made and collected rows, field access and its errors, the tuple operations, pickling, `asDict`, row classes, and the row of a join with duplicate names, give output identical to official PySpark 4.3.0, except for `isinstance(row, tuple)`. - New offline tests check the repr of ten kinds of rows, the tuple behaviors, the field names and their errors, and `asDict`. They also pass with official PySpark 4.3.0. - A new live test checks collected rows: their repr, equality and hash with a made row, pickling, nested values, and duplicate names. It also passes with official PySpark 4.3.0. - The offline Python suites (433 passed, 8 live cases skipped), and the live drop-in and Arrow suites (22 passed) and live cases (9 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
