HyukjinKwon opened a new pull request, #214: URL: https://github.com/apache/spark-connect-rust/pull/214
### What changes were proposed in this pull request? [SPARK-60947](https://issues.apache.org/jira/browse/SPARK-60947): the Python client now sends and collects TIME values, and parses the TIME type of DDL strings, as PySpark Connect does. - `createDataFrame` infers `TimeType()` of a `datetime.time`, and sends a TIME value as PyArrow's `time64[ns]` of it, as `to_arrow_type` declares. That type also takes the nanoseconds of an int. - `lit()` of a `datetime.time` is a time literal of its nanoseconds since midnight. - Collected TIME values are `datetime.time`, as `TimeType.fromInternal` reads them. - A bare `time`, in a DDL string or in the JSON of a type, is `TimeType(6)`. So, as for PySpark, the server raises `UNSUPPORTED_TIME_TYPE` unless the session enables `spark.sql.timeType.enabled`, and with it TIME values round-trip. The core values have `Time`, in nanoseconds since midnight, for these. ### Why are the changes needed? TIME values were unsupported, and differed from PySpark 4.3.0: - `createDataFrame([(datetime.time(1, 2, 3),)])` raised `TypeError: unsupported value type for createDataFrame: time`, rather than the server's `UNSUPPORTED_TIME_TYPE`. - `createDataFrame(..., "t time")` raised `ValueError: invalid type string 't time'`. - `lit(datetime.time(...))` raised `UNSUPPORTED_DATA_TYPE`. - Collected TIME values, such as of `SELECT TIME'12:34:56.789'`, were strings. ### Does this PR introduce _any_ user-facing change? Yes. TIME values work as in PySpark 4.3.0. ### How was this patch tested? - A new live test round-trips TIME values of `createDataFrame`, inferred and of a DDL schema, of `lit()` and of SQL, with the TIME type enabled. It also checks `UNSUPPORTED_TIME_TYPE` without the TIME type. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks, against a mock Connect server, the `time64[ns]` values and the `time(6)` schema that `createDataFrame` sends, and the time literal of `lit()`. Another checks the parsing of `time` and `time(p)`. They fail without this change. - New lib tests check the decoding of Arrow times, the encoding of TIME values, and the parsing of `time`. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1330 passed, 8 skipped), and the live drop-in and Arrow suites (88 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
