HyukjinKwon opened a new pull request, #307: URL: https://github.com/apache/spark-connect-rust/pull/307
### What changes were proposed in this pull request? [SPARK-61107](https://issues.apache.org/jira/browse/SPARK-61107): `DataFrame.toArrow` now casts the table to the Arrow schema of the DataFrame, as PySpark Connect does. As PySpark Connect's `toArrow`, the table of the result is cast to the Arrow schema of the DataFrame's schema, `to_arrow_schema(schema, error_on_duplicated_field_names_in_struct=True, timezone="UTC")` (vendored). So: - The table has the Arrow types of the Spark types. Strings and binaries are `string` and `binary`, even with `spark.sql.execution.arrow.useLargeVarTypes`. Timestamps are of UTC, whatever the session time zone. - The fields of nested structs have the names of the schema. - The field metadata of spatial types is PySpark's. - A type that PySpark does not convert to Arrow raises `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`. These are year-month intervals, calendar intervals, char and varchar. - Duplicate names of nested fields raise `DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT`. - An array or map of `NullType` that is not nullable raises PyArrow's "A null type field may not be non-nullable". - A result without batches is the empty table of the schema. The schema is the one that the server sends with the result (`ExecutePlanResponse.schema`), which is that of the analysis of the query. PySpark analyzes the query before it runs it, so for an unsupported type it raises without running the query; the client raises after. The data from a server that sends no schema is returned as it is. The Arrow handoff tests' mock service is one such server: it implements no analysis. ### Why are the changes needed? The client returned the Arrow data of the server as it was. That data differs from PySpark's table: - `large_string` and `large_binary` columns with `useLargeVarTypes` - timestamps of the session time zone - `a_0`, `a_1` for duplicate nested names - `month_interval` columns that PyArrow cannot read (`to_pylist()` raises `KeyError: 21`) ### Does this PR introduce _any_ user-facing change? Yes. `toArrow` returns the table, or raises the errors, of PySpark 4.3.0. ### How was this patch tested? - A new live test checks: - the types of a string and a timestamp, with large var types and a session time zone of America/Los_Angeles - the errors of year-month and calendar intervals - the error of duplicate nested names It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks, against a mock Connect server: - the cast of `large_string` and `large_binary` data to the schema sent with the result - that a year-month interval raises - that data without a schema is as it is - On a local server, 252 cases of `toArrow` and `toPandas` of a value of each type, with and without `useLargeVarTypes` and `structHandlingMode`, give the same results as official PySpark 4.3.0. - The lib tests of both crates (589 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2056 passed, 12 skipped), and the live drop-in and Arrow suites (160 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
