dongjoon-hyun opened a new pull request, #511:
URL: https://github.com/apache/spark-connect-swift/pull/511
### What changes were proposed in this pull request?
This PR aims to support nanosecond timestamp values in `DataFrame.collect`
by introducing a new `TimestampNanos` struct, following up SPARK-59197 which
covered the `DataType` path.
- A new public struct `TimestampNanos` represents a timestamp with
nanosecond precision. Like `TimestampNanosVal` of Apache Spark, it stores
`epochMicros: Int64` and `nanosWithinMicro: Int16`, so it is not limited to the
`1677 ~ 2262` range of Arrow's 64-bit nanosecond timestamps. It provides
`init(epochNanos:)`, a lossy `date` conversion, and a UTC `description`.
- `DataFrame.collect` now decodes Arrow `timestamp(NANOSECOND)` columns into
`TimestampNanos` values. Apache Spark serializes both `TIMESTAMP_NTZ(p)` and
`TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`) columns this way. Microsecond timestamp
columns still return `Date`.
- `Row.==` supports `TimestampNanos` value comparison.
`lit`, SQL parameter binding, and `createDataFrame` for this type will be
handled separately.
### Why are the changes needed?
`Date` stores `Double` seconds, so `DataFrame.collect` lost sub-microsecond
digits of the nanosecond timestamp types added in Apache Spark 4.3.0.
```swift
try await spark.conf.set("spark.sql.timestampNanosTypes.enabled", "true")
let df = try await spark.sql("SELECT CAST(TIMESTAMP_NTZ'2026-01-01
00:00:00.123456789' AS TIMESTAMP_NTZ(9))")
try await df.collect() // Before: [Row(2026-01-01 00:00:00.123456717)] as
`Date`
// After: [Row(2026-01-01 00:00:00.123456789)] as
`TimestampNanos`
```
### Does this PR introduce _any_ user-facing change?
Yes, `DataFrame.collect` returns `TimestampNanos` values instead of lossy
`Date` values for `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` columns, and a new
public type `TimestampNanos` is added. Existing `TIMESTAMP` and `TIMESTAMP_NTZ`
columns still return `Date`.
### How was this patch tested?
Pass the CIs with the newly added test cases. Manually verified the full
test suite against Apache Spark 4.3.0 RC1 and 4.2.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
