dongjoon-hyun opened a new pull request, #513:
URL: https://github.com/apache/spark-connect-swift/pull/513
### What changes were proposed in this pull request?
This PR aims to support `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (`p` in
`7...9`) columns in `SparkSession.createDataFrame` with `TimestampNanos` values.
```swift
let ts = TimestampNanos(epochMicros: 1_706_000_000_123_456,
nanosWithinMicro: 789)!
let df = try await spark.createDataFrame([[ts], [nil]], "t TIMESTAMP_NTZ(9)")
try await df.dtypes // [("t", "timestamp_ntz(9)")]
try await df.collect() // [Row(2024-01-23 08:53:20.123456789), Row(nil)]
```
- `ConvertToArrow` maps both types to an `Apache Arrow`
`Timestamp(NANOSECOND)` column (no timezone for `TIMESTAMP_NTZ(p)`, `UTC` for
`TIMESTAMP_LTZ(p)`), writing `TimestampNanos` as `Int64` epoch nanoseconds,
symmetrically with the `collect()` decoding path added by SPARK-59198. A value
outside the `Int64` nanosecond range throws `SparkConnectError.InvalidType`
instead of overflowing silently.
- The precision is sent as the `SPARK::timestampNanos::precision` field
metadata key, which `ArrowUtils.fromArrowField` reads on the server, like
`SPARK::time::precision` for `TIME(p)` (SPARK-59155).
### Why are the changes needed?
This is the last missing piece of nanosecond timestamp support after
`DataType` (SPARK-59197), `collect()` (SPARK-59198), and `lit`/SQL parameters
(SPARK-59203). `createDataFrame` rejected these schemas with
`SparkConnectError.InvalidType`.
### Does this PR introduce _any_ user-facing change?
No behavior change. Only `createDataFrame` now accepts `TIMESTAMP_NTZ(p)`
and `TIMESTAMP_LTZ(p)` columns with `TimestampNanos` values against Apache
Spark 4.3.0 and later.
### How was this patch tested?
Pass the CIs with a newly added test case,
`CreateDataFrameTests.timestampNanosTypes`, which verifies `dtypes` and a
`createDataFrame` → `collect` round trip for both types and precision 7 to 9,
including sub-microsecond digits, a pre-1970 value, `nil`, and the out-of-range
error.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
