HyukjinKwon opened a new pull request, #277: URL: https://github.com/apache/spark-connect-rust/pull/277
### What changes were proposed in this pull request? [SPARK-61075](https://issues.apache.org/jira/browse/SPARK-61075): the `toInternal` and `fromInternal` of the data types now convert values to the internal values of Spark and back, as PySpark's do. As PySpark's: - a date is the days since the epoch, `DateType().toInternal(date(2020, 1, 1)) == 18262` - a timestamp is the microseconds since the epoch, of the local time zone or of its own - a timestamp without a time zone is the microseconds of its wall clock in UTC - a time is the nanoseconds since midnight, and a day-time interval its microseconds - a variant, a geometry and a geography are the dicts of their values - the elements, keys, values and fields of arrays, maps and structs are converted - a struct takes a tuple, a list, a `Row`, a dict or an object, and its `fromInternal` gives a `Row` - year-month and calendar intervals raise `NOT_IMPLEMENTED` `needConversion` is true for variants, spatial types and user-defined types, also in arrays and maps. It is false for a year-month interval with `PYSPARK_YM_INTERVAL_LEGACY=1`. `StructType` has PySpark's `_needSerializeAnyField` and `_needConversion`. The conversions are those of the shim `pyspark.sql.types`, which follow PySpark's code. The core's `need_conversion` is true for variants, spatial types and UDTs. ### Why are the changes needed? The client returned the value itself for every data type: - `TimestampType().toInternal(datetime(2020, 1, 1))` returned the datetime rather than `1577836800000000` (in UTC) - `DateType().fromInternal(18262)` returned `18262` rather than the date 2020-01-01 - `StructType.fromInternal` returned the tuple rather than a `Row` ### Does this PR introduce _any_ user-facing change? Yes. These methods convert values as in PySpark 4.3.0. ### How was this patch tested? - A new live test converts a collected row of dates, timestamps, an interval, an array and a map with `toInternal`. It checks the result against the internal values that the server computes with `unix_date` and `unix_micros`, and checks that `fromInternal` gives the row back. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the conversions of each type, the structs of tuples, dicts and rows, the errors, and `needConversion`, including `PYSPARK_YM_INTERVAL_LEGACY`. - The core lib test of `need_conversion` checks variants, spatial types and UDTs. - On 75 cases of the conversions, and 23 more of user-defined types, nested types and the legacy flag, the client gives the same results as official PySpark 4.3.0 in the UTC, America/Los_Angeles and Asia/Seoul time zones. - The lib tests of both crates (589 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1978 passed, 8 skipped), and the live drop-in and Arrow suites (134 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
