dongjoon-hyun opened a new pull request, #510: URL: https://github.com/apache/spark-connect-swift/pull/510
### What changes were proposed in this pull request? This PR aims to support the nanosecond timestamp types `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`, Apache Spark 4.3) in `DataType`. - Add `DataType.timestampNtzNanos(precision:)` and `DataType.timestampLtzNanos(precision:)`. - `simpleString` follows Apache Spark's `typeName`: `timestamp_ntz(p)` / `timestamp_ltz(p)`. - Convert both ways with the `timestamp_ntz_nanos` / `timestamp_ltz_nanos` proto kinds. A missing `precision` defaults to `9`. ### Why are the changes needed? Since Apache Spark 4.3.0, the server sends these proto types in `AnalyzePlan` and `ExecutePlan` responses. - https://github.com/apache/spark/pull/57699 ([SPARK-57161] Convert nanosecond-capable timestamp types and literals between proto and Catalyst in Spark Connect) `DataType.init(_ proto:)` threw `SparkConnectError.InvalidType` for them, so `DataFrame.schema`/`dtypes`/`printSchema`, `parseDDL`, and `collect()` failed on any DataFrame with such a column. ```swift let df = try await spark.sql("SELECT CAST(TIMESTAMP_NTZ'2026-01-01 00:00:00.123456789' AS TIMESTAMP_NTZ(9)) ts") try await df.schema // Before: SparkConnectError.invalidType // After : struct<ts:timestamp_ntz(9)> ``` ### Does this PR introduce _any_ user-facing change? Yes, this adds two new `DataType` cases. ### How was this patch tested? Pass the CIs with newly added test cases. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
