dbtsai opened a new pull request, #116: URL: https://github.com/apache/spark-connect-rust/pull/116
### What changes were proposed in this pull request? Support day-time interval inputs in the Rust and Python clients ([SPARK-60659](https://issues.apache.org/jira/browse/SPARK-60659)). Encode `Value::DayTimeInterval` as Arrow `Duration(Microsecond)`; convert Python `timedelta` using integer arithmetic with checked conversion to signed 64-bit microseconds. Infer nested interval types across null-first rows and empty collections, and support explicit schemas and collected-row round trips. **Dependency:** this branch includes the decoder prerequisite from #110, which is still open. The input-specific implementation and tests are in commit c5473ba. It builds on master's column-wise Arrow encoder from #109 and leaves #113's shared-name work separate. ### Why are the changes needed? Decoding interval results alone does not allow users to send them back. For example, `spark.createDataFrame([(None,), (timedelta(microseconds=-1),)], ["interval"])` and `spark.createDataFrame(df.collect(), df.schema)` now support intervals without floating-point precision loss. ### Does this PR introduce _any_ user-facing change? Yes. Rust and Python interval inputs work in scalar columns, structs, lists, and maps, including nulls. Out-of-range Python timedeltas raise `OverflowError` before RPC. Inference examines non-null values across rows and nested containers and rejects incompatible value kinds; the existing all-null/empty inference fallback remains unchanged. ### How was this patch tested? - 396 Rust library tests passed; all integration targets compiled in default and `wasm-udf` configurations. - 284 Python checks passed, plus one existing skip, using pandas 2.3.3 and the rebuilt extension. - Real Spark 4.2 server: Rust interval integration test and Python interval tests passed for negative values, nulls, nested values, exact `i64::MIN/MAX`, empty and null-only explicit inputs, and collected-row round trips. - Local gRPC fixtures inspect outgoing Arrow IPC for exact Python input and collected-row values. Overflow tests cover both signed boundaries, explicit schemas, nested inputs, and `timedelta.min/max`. - Spark 4.2's official interval creation regression passed through the Rust transport. - Formatting and core-library Clippy passed. The Python binding's 140 Clippy errors reproduce identically on unmodified upstream; workspace Clippy also encounters existing approximate-constant test errors. - Public class/member snapshots match the baseline. The pre-existing `SparkSession.Hook` gap remains. - Full official Connect gate is not green locally: unrelated TIME tests fail because the local server does not enable `spark.sql.timeType.enabled`. The broader run is still being evaluated; no full-suite parity claim is made. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
