This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new b46468d [SPARK-60061] Fix Row.getAsTimestampNanos and Row.== to not
crash on out-of-range values
b46468d is described below
commit b46468db4e7a81aad60e4af5c265a558d240fa45
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 7 16:14:22 2026 -0700
[SPARK-60061] Fix Row.getAsTimestampNanos and Row.== to not crash on
out-of-range values
### What changes were proposed in this pull request?
This PR fixes two crashes in `Row`.
1. `Row.getAsTimestampNanos` converts a `Date` value by rounding to the
nearest microsecond and building `TimestampNanos(epochMicros:)`, instead of
computing `Int64` epoch nanoseconds. A `Date` that cannot be represented, such
as `infinity`, now throws `SparkConnectError.InvalidType`, like the other
type-safe getters.
2. `Row.==` compares integer values, and arrays of integer values, through
`Int128` instead of `Int64`.
### Why are the changes needed?
1. `DataFrame.collect()` returns `TIMESTAMP` and `TIMESTAMP_NTZ` columns as
`Date`. `Int64` epoch nanoseconds only cover the years 1677-2262, so
`getAsTimestampNanos` crashed on common sentinel values such as
`TIMESTAMP'0001-01-01'` or `TIMESTAMP'9999-12-31'`:
```
Fatal error: Double value cannot be converted to Int64 because the
result would be less than Int64.min
```
The old code also multiplied by `1_000_000_000` in `Double`, which added
rounding error and produced wrong sub-microsecond and even microsecond digits.
For example, `1_700_000_000.999999` became `999998` microseconds and `976`
nanoseconds. `Date` cannot hold nanosecond precision, and Spark `TIMESTAMP` has
microsecond precision, so rounding to the nearest microsecond gives the correct
value.
2. `Row(UInt64.max) == Row(UInt64.max)` crashed because `Int64(UInt64.max)`
traps:
```
Fatal error: Not enough bits to represent the passed value
```
### Does this PR introduce _any_ user-facing change?
Yes. These cases no longer crash. `getAsTimestampNanos` on a `Date` value
now returns a microsecond-precision `TimestampNanos` (`nanosWithinMicro == 0`).
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `RowTests`: `getAsTimestampNanosOutOfInt64NanosRange`,
`getAsTimestampNanosRoundsDateToMicroseconds`, `compareUInt64`, `compareData`
- `DataFrameTests`: `collectTimestampAsTimestampNanos`, an end-to-end test
that collects `TIMESTAMP_NTZ'0001-01-01 00:00:00'`, `TIMESTAMP_NTZ'2026-01-01
00:00:00.999999'`, and `TIMESTAMP_NTZ'9999-12-31 00:00:00'`
I also ran the tests manually against an Apache Spark 4.2.0 Spark Connect
server.
```
$ swift test --no-parallel --filter 'RowTests|DataFrameTests'
...
Test run with 182 tests in 3 suites passed after 35.407 seconds.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.
Closes #588 from dongjoon-hyun/SPARK-60061.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/Row.swift | 11 ++++++--
Tests/SparkConnectTests/DataFrameTests.swift | 15 ++++++++++
Tests/SparkConnectTests/RowTests.swift | 41 ++++++++++++++++++++++++++++
3 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/Sources/SparkConnect/Row.swift b/Sources/SparkConnect/Row.swift
index 61db2bf..668fece 100644
--- a/Sources/SparkConnect/Row.swift
+++ b/Sources/SparkConnect/Row.swift
@@ -353,6 +353,7 @@ public struct Row: Sendable, Equatable {
/// - Returns: A `TimestampNanos` value of the field.
/// - Throws: `SparkConnectError.InvalidArgument` if the index is out of
range, or
/// `SparkConnectError.InvalidType` if the value is `nil` or not convertible
to `TimestampNanos`.
+ /// A `Date` value is rounded to the nearest microsecond.
public func getAsTimestampNanos(_ i: Int) throws -> TimestampNanos {
if i < 0 || i >= self.length {
throw SparkConnectError.InvalidArgument
@@ -364,7 +365,11 @@ public struct Row: Sendable, Equatable {
return ts
}
if let date = raw as? Date {
- return TimestampNanos(epochNanos: Int64((date.timeIntervalSince1970 *
1_000_000_000).rounded()))
+ // `Date` is not precise enough for nanoseconds, so round to the nearest
microsecond.
+ guard let epochMicros = Int64(exactly: (date.timeIntervalSince1970 *
1_000_000).rounded()) else {
+ throw SparkConnectError.InvalidType
+ }
+ return TimestampNanos(epochMicros: epochMicros)!
}
throw SparkConnectError.InvalidType
}
@@ -420,7 +425,7 @@ public struct Row: Sendable, Equatable {
} else if let a = x as? Bool, let b = y as? Bool {
return a == b
} else if let a = x as? any FixedWidthInteger, let b = y as? any
FixedWidthInteger {
- return Int64(a) == Int64(b)
+ return Int128(a) == Int128(b)
} else if let a = x as? Float, let b = y as? Float {
return a == b
} else if let a = x as? Double, let b = y as? Double {
@@ -440,7 +445,7 @@ public struct Row: Sendable, Equatable {
} else if let a = x as? [Bool], let b = y as? [Bool] {
return a == b
} else if let a = x as? [any FixedWidthInteger], let b = y as? [any
FixedWidthInteger] {
- return a.map { Int64($0) } == b.map { Int64($0) }
+ return a.map { Int128($0) } == b.map { Int128($0) }
} else if let a = x as? [Float], let b = y as? [Float] {
return a == b
} else if let a = x as? [Double], let b = y as? [Double] {
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift
b/Tests/SparkConnectTests/DataFrameTests.swift
index f30fc8b..290817c 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -216,6 +216,21 @@ struct DataFrameTests {
await spark.stop()
}
+ @Test
+ func collectTimestampAsTimestampNanos() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let expected = [
+ ("TIMESTAMP_NTZ'0001-01-01 00:00:00'", Int64(-62_135_596_800_000_000)),
+ ("TIMESTAMP_NTZ'2026-01-01 00:00:00.999999'",
Int64(1_767_225_600_999_999)),
+ ("TIMESTAMP_NTZ'9999-12-31 00:00:00'", Int64(253_402_214_400_000_000)),
+ ]
+ for pair in expected {
+ let row = try await spark.sql("SELECT \(pair.0)").collect()[0]
+ #expect(try row.getAsTimestampNanos(0) == TimestampNanos(epochMicros:
pair.1))
+ }
+ await spark.stop()
+ }
+
@Test
func collectTimestampNanos() async throws {
let spark = try await SparkSession.builder.getOrCreate()
diff --git a/Tests/SparkConnectTests/RowTests.swift
b/Tests/SparkConnectTests/RowTests.swift
index 2144a7c..4bb9a94 100644
--- a/Tests/SparkConnectTests/RowTests.swift
+++ b/Tests/SparkConnectTests/RowTests.swift
@@ -445,6 +445,30 @@ struct RowTests {
}
}
+ @Test
+ func getAsTimestampNanosOutOfInt64NanosRange() throws {
+ // 9999-12-31 00:00:00 UTC and 0001-01-01 00:00:00 UTC
+ let maxDate = Date(timeIntervalSince1970: 253_402_214_400)
+ let minDate = Date(timeIntervalSince1970: -62_135_596_800)
+ let row = Row(maxDate, minDate, Date(timeIntervalSince1970: .infinity))
+ #expect(try row.getAsTimestampNanos(0) == TimestampNanos(epochMicros:
253_402_214_400_000_000))
+ #expect(try row.getAsTimestampNanos(1) == TimestampNanos(epochMicros:
-62_135_596_800_000_000))
+ #expect(throws: SparkConnectError.InvalidType) {
+ try row.getAsTimestampNanos(2)
+ }
+ }
+
+ @Test
+ func getAsTimestampNanosRoundsDateToMicroseconds() throws {
+ let row = Row(
+ Date(timeIntervalSince1970: 1_700_000_000.000_001),
+ Date(timeIntervalSince1970: 1_700_000_000.999_999),
+ Date(timeIntervalSince1970: -1.000_001))
+ #expect(try row.getAsTimestampNanos(0) == TimestampNanos(epochMicros:
1_700_000_000_000_001))
+ #expect(try row.getAsTimestampNanos(1) == TimestampNanos(epochMicros:
1_700_000_000_999_999))
+ #expect(try row.getAsTimestampNanos(2) == TimestampNanos(epochMicros:
-1_000_001))
+ }
+
@Test
func getAsDecimal() throws {
let dec = Decimal(12.34)
@@ -575,6 +599,23 @@ struct RowTests {
#expect([Row(1), Row(2)] != [Row(1), Row(3)])
}
+ @Test
+ func compareUInt64() {
+ #expect(Row(UInt64.max) == Row(UInt64.max))
+ #expect(Row(UInt64.max) != Row(UInt64.max - 1))
+ #expect(Row(UInt64.max) != Row(-1))
+ #expect(Row(UInt64(1)) == Row(1))
+ #expect(Row([UInt64.max]) == Row([UInt64.max]))
+ #expect(Row([UInt64.max]) != Row([-1]))
+ }
+
+ @Test
+ func compareData() {
+ #expect(Row(Data([1, 2])) == Row(Data([1, 2])))
+ #expect(Row(Data([1, 2])) != Row(Data([1, 3])))
+ #expect(Row(Data()) != Row(nil))
+ }
+
@Test
func compareArray() {
#expect([Row([1, 2])] == [Row([1, 2])])
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
