This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new ac834d8 [SPARK-58571] Support `TIME` type values in
`DataFrame.collect`
ac834d8 is described below
commit ac834d8636eb727d8ca8af4207a67c74586986d8
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Aug 4 11:29:17 2026 -0700
[SPARK-58571] Support `TIME` type values in `DataFrame.collect`
### What changes were proposed in this pull request?
This PR aims to support `TIME` type values in `DataFrame.collect` by
introducing a new
`LocalTime` struct, following up SPARK-58568 which covered the schema path.
- A new public struct `LocalTime` represents a time of day without a date
or a time zone,
like `java.time.LocalTime` of the Scala client and `datetime.time` of
PySpark. It stores
`nanoOfDay` and exposes `hour`/`minute`/`second`/`nanosecond` components.
- `DataFrame.collect` now decodes Arrow `time64` columns into `LocalTime`
values. Apache
Spark serializes `TIME(n)` columns as Arrow `time64(NANOSECOND)`
regardless of precision.
- `Row.==` supports `LocalTime` value comparison.
No vendored `Arrow*.swift` file is touched; the Arrow layer already decodes
`time64`.
### Why are the changes needed?
`DataFrame.collect` silently returned `nil` for every `TIME` column value
because the
Arrow-to-Row conversion had no `time64` case and fell through to the
`String` cast.
```swift
let df = try await spark.sql("SELECT TIME'12:34:56'")
try await df.collect() // Before: [Row(nil)]
// After: [Row(LocalTime(hour: 12, minute: 34,
second: 56))]
```
### Does this PR introduce _any_ user-facing change?
Yes, `DataFrame.collect` returns `LocalTime` values instead of `nil` for
`TIME` columns,
and a new public type `LocalTime` is added. This is a bug fix from the
unreleased perspective.
### How was this patch tested?
Pass the CIs with the newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #481 from dongjoon-hyun/SPARK-58571.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/DataFrame+Actions.swift | 12 +++
.../Documentation.docc/SparkConnect.md | 1 +
Sources/SparkConnect/LocalTime.swift | 92 ++++++++++++++++++++++
Sources/SparkConnect/Row.swift | 2 +
Tests/SparkConnectTests/DataFrameTests.swift | 26 ++++++
Tests/SparkConnectTests/LocalTimeTests.swift | 69 ++++++++++++++++
6 files changed, 202 insertions(+)
diff --git a/Sources/SparkConnect/DataFrame+Actions.swift
b/Sources/SparkConnect/DataFrame+Actions.swift
index 89a004e..3b9c20f 100644
--- a/Sources/SparkConnect/DataFrame+Actions.swift
+++ b/Sources/SparkConnect/DataFrame+Actions.swift
@@ -133,6 +133,18 @@ extension DataFrame {
values.append(array.asAny(i) as? Decimal)
case .primitiveInfo(.date32):
values.append(array.asAny(i) as! Date)
+ case .timeInfo(.time64):
+ // Spark serializes `TIME` columns as Arrow `time64` values
since midnight.
+ let time64Type = column.data.type as! ArrowTypeTime64
+ let time = array.asAny(i) as! Int64
+ let nanoOfDay =
+ switch time64Type.unit {
+ case .microseconds:
+ time * 1_000
+ case .nanoseconds:
+ time
+ }
+ values.append(LocalTime(nanoOfDay: nanoOfDay))
case .timeInfo(.timestamp):
let timestampType = column.data.type as! ArrowTypeTimestamp
let timestamp = array.asAny(i) as! Int64
diff --git a/Sources/SparkConnect/Documentation.docc/SparkConnect.md
b/Sources/SparkConnect/Documentation.docc/SparkConnect.md
index 092a7b0..3d6bc5a 100644
--- a/Sources/SparkConnect/Documentation.docc/SparkConnect.md
+++ b/Sources/SparkConnect/Documentation.docc/SparkConnect.md
@@ -30,6 +30,7 @@ SparkConnect is a modern Swift library that provides a native
interface to Apach
- ``DataFrame``
- ``GroupedData``
- ``Row``
+- ``LocalTime``
- ``StorageLevel``
### Data I/O
diff --git a/Sources/SparkConnect/LocalTime.swift
b/Sources/SparkConnect/LocalTime.swift
new file mode 100644
index 0000000..28d92f7
--- /dev/null
+++ b/Sources/SparkConnect/LocalTime.swift
@@ -0,0 +1,92 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+#if canImport(FoundationEssentials)
+import FoundationEssentials
+#else
+import Foundation
+#endif
+
+/// A time of day without a date or a time zone, like `java.time.LocalTime`.
+/// Values of Spark's `TIME` type are represented as `LocalTime` in ``Row``s.
+public struct LocalTime: Sendable, Equatable, Hashable,
CustomStringConvertible {
+ static let nanosPerSecond: Int64 = 1_000_000_000
+ static let nanosPerMinute: Int64 = 60 * nanosPerSecond
+ static let nanosPerHour: Int64 = 60 * nanosPerMinute
+ static let nanosPerDay: Int64 = 24 * nanosPerHour
+
+ /// The number of nanoseconds since midnight, in the range
`0..<86_400_000_000_000`.
+ public let nanoOfDay: Int64
+
+ /// Creates a `LocalTime` from a time of day.
+ /// - Parameters:
+ /// - hour: An hour in the range `0...23`.
+ /// - minute: A minute in the range `0...59`.
+ /// - second: A second in the range `0...59`.
+ /// - nanosecond: A nanosecond in the range `0...999_999_999`.
+ /// - Returns: `nil` if any component is out of range.
+ public init?(hour: Int, minute: Int, second: Int = 0, nanosecond: Int = 0) {
+ guard (0..<24).contains(hour), (0..<60).contains(minute),
(0..<60).contains(second),
+ (0..<1_000_000_000).contains(nanosecond)
+ else {
+ return nil
+ }
+ self.nanoOfDay =
+ Int64(hour) * Self.nanosPerHour + Int64(minute) * Self.nanosPerMinute
+ + Int64(second) * Self.nanosPerSecond + Int64(nanosecond)
+ }
+
+ /// Creates a `LocalTime` from the number of nanoseconds since midnight.
+ /// - Parameter nanoOfDay: A nanosecond of the day in the range
`0..<86_400_000_000_000`.
+ /// - Returns: `nil` if `nanoOfDay` is out of range.
+ public init?(nanoOfDay: Int64) {
+ guard (0..<Self.nanosPerDay).contains(nanoOfDay) else {
+ return nil
+ }
+ self.nanoOfDay = nanoOfDay
+ }
+
+ /// The hour of the day, in the range `0...23`.
+ public var hour: Int { Int(nanoOfDay / Self.nanosPerHour) }
+
+ /// The minute of the hour, in the range `0...59`.
+ public var minute: Int { Int((nanoOfDay / Self.nanosPerMinute) % 60) }
+
+ /// The second of the minute, in the range `0...59`.
+ public var second: Int { Int((nanoOfDay / Self.nanosPerSecond) % 60) }
+
+ /// The nanosecond of the second, in the range `0...999_999_999`.
+ public var nanosecond: Int { Int(nanoOfDay % Self.nanosPerSecond) }
+
+ /// A string like `12:34:56`, `12:34:56.123`, `12:34:56.123456`, or
+ /// `12:34:56.123456789` depending on the smallest non-zero fractional unit.
+ public var description: String {
+ var result = String(format: "%02d:%02d:%02d", hour, minute, second)
+ let nanosecond = self.nanosecond
+ if nanosecond != 0 {
+ if nanosecond % 1_000_000 == 0 {
+ result += String(format: ".%03d", nanosecond / 1_000_000)
+ } else if nanosecond % 1_000 == 0 {
+ result += String(format: ".%06d", nanosecond / 1_000)
+ } else {
+ result += String(format: ".%09d", nanosecond)
+ }
+ }
+ return result
+ }
+}
diff --git a/Sources/SparkConnect/Row.swift b/Sources/SparkConnect/Row.swift
index caeda7c..ad3a983 100644
--- a/Sources/SparkConnect/Row.swift
+++ b/Sources/SparkConnect/Row.swift
@@ -137,6 +137,8 @@ public struct Row: Sendable, Equatable {
return a == b
} else if let a = x as? Date, let b = y as? Date {
return a == b
+ } else if let a = x as? LocalTime, let b = y as? LocalTime {
+ return a == b
} else if let a = x as? String, let b = y as? String {
return a == b
} else if let a = x as? [Bool], let b = y as? [Bool] {
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift
b/Tests/SparkConnectTests/DataFrameTests.swift
index d812745..e4d0858 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -171,6 +171,32 @@ struct DataFrameTests {
await spark.stop()
}
+ @Test
+ func collectTime() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ if await spark.version >= "4.2" {
+ try await spark.conf.set("spark.sql.timeType.enabled", "true")
+ let expected = [
+ ("TIME'00:00:00'", LocalTime(hour: 0, minute: 0)),
+ ("TIME'12:34:56'", LocalTime(hour: 12, minute: 34, second: 56)),
+ ("CAST(TIME'12:34:56' AS TIME(0))", LocalTime(hour: 12, minute: 34,
second: 56)),
+ (
+ "CAST(TIME'12:34:56.123' AS TIME(3))",
+ LocalTime(hour: 12, minute: 34, second: 56, nanosecond: 123_000_000)
+ ),
+ (
+ "TIME'23:59:59.999999'",
+ LocalTime(hour: 23, minute: 59, second: 59, nanosecond: 999_999_000)
+ ),
+ ]
+ for pair in expected {
+ #expect(try await spark.sql("SELECT \(pair.0)").collect() ==
[Row(pair.1)])
+ }
+ #expect(try await spark.sql("SELECT CAST(NULL AS TIME)").collect() ==
[Row(nil)])
+ }
+ await spark.stop()
+ }
+
@Test
func array() async throws {
let spark = try await SparkSession.builder.getOrCreate()
diff --git a/Tests/SparkConnectTests/LocalTimeTests.swift
b/Tests/SparkConnectTests/LocalTimeTests.swift
new file mode 100644
index 0000000..d3cfac4
--- /dev/null
+++ b/Tests/SparkConnectTests/LocalTimeTests.swift
@@ -0,0 +1,69 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+import SparkConnect
+import Testing
+
+/// A test suite for ``LocalTime``.
+struct LocalTimeTests {
+ @Test
+ func components() async throws {
+ let time = try #require(LocalTime(hour: 12, minute: 34, second: 56,
nanosecond: 123_456_789))
+ #expect(time.hour == 12)
+ #expect(time.minute == 34)
+ #expect(time.second == 56)
+ #expect(time.nanosecond == 123_456_789)
+ #expect(time.nanoOfDay == 45_296_123_456_789)
+ }
+
+ @Test
+ func componentBounds() async throws {
+ #expect(LocalTime(hour: 0, minute: 0) != nil)
+ #expect(LocalTime(hour: 23, minute: 59, second: 59, nanosecond:
999_999_999) != nil)
+ #expect(LocalTime(hour: -1, minute: 0) == nil)
+ #expect(LocalTime(hour: 24, minute: 0) == nil)
+ #expect(LocalTime(hour: 0, minute: 60) == nil)
+ #expect(LocalTime(hour: 0, minute: 0, second: 60) == nil)
+ #expect(LocalTime(hour: 0, minute: 0, second: 0, nanosecond:
1_000_000_000) == nil)
+ }
+
+ @Test
+ func nanoOfDay() async throws {
+ #expect(LocalTime(nanoOfDay: 0) == LocalTime(hour: 0, minute: 0))
+ #expect(LocalTime(nanoOfDay: 45_296_000_000_000) == LocalTime(hour: 12,
minute: 34, second: 56))
+ #expect(LocalTime(nanoOfDay: 86_399_999_999_999) != nil)
+ #expect(LocalTime(nanoOfDay: -1) == nil)
+ #expect(LocalTime(nanoOfDay: 86_400_000_000_000) == nil)
+ }
+
+ @Test
+ func description() async throws {
+ #expect(LocalTime(hour: 1, minute: 2, second: 3)?.description ==
"01:02:03")
+ #expect(LocalTime(hour: 12, minute: 34, second: 56)?.description ==
"12:34:56")
+ #expect(
+ LocalTime(hour: 12, minute: 34, second: 56, nanosecond:
123_000_000)?.description
+ == "12:34:56.123")
+ #expect(
+ LocalTime(hour: 12, minute: 34, second: 56, nanosecond:
123_456_000)?.description
+ == "12:34:56.123456")
+ #expect(
+ LocalTime(hour: 12, minute: 34, second: 56, nanosecond:
123_456_789)?.description
+ == "12:34:56.123456789")
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
