This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 2ecd79c [SPARK-58568] Support `TIME` type in `DataType.simpleString`
2ecd79c is described below
commit 2ecd79c24600bb5f999c89c397987d130446ed38
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Aug 4 10:40:37 2026 -0700
[SPARK-58568] Support `TIME` type in `DataType.simpleString`
### What changes were proposed in this pull request?
This PR aims to support `TIME` type (added in Apache Spark 4.1) in
`DataType.simpleString`
so that `DataFrame.schema`/`dtypes`/`printSchema` work on `TIME` columns.
Following Apache Spark's `TimeType.typeName` convention, the mapping uses
`time(<precision>)`.
When the proto `precision` field is absent, it falls back to `time(6)` like
Apache Spark's
`TimeType.DEFAULT_PRECISION` (`MICROS_PRECISION`).
### Why are the changes needed?
`Spark_Connect_DataType`'s `time` kind fell through to the `default:`
branch of
`DataType.simpleString` and threw `SparkConnectError.InvalidType`. As a
result,
`DataFrame.dtypes` failed entirely on any DataFrame containing a `TIME`
column.
```swift
let df = try await spark.sql("SELECT TIME'12:34:56'")
try await df.dtypes // Before: throws SparkConnectError.InvalidType
// After: [("TIME '12:34:56'", "time(6)")]
```
### Does this PR introduce _any_ user-facing change?
Yes, `DataFrame.dtypes` no longer throws on `TIME` columns and returns
Spark-style type
strings instead. This is a bug fix from the unreleased perspective.
### How was this patch tested?
Pass the CIs with a newly added test case. The new test enables
`spark.sql.timeType.enabled`
at the session level and is guarded to run on Apache Spark 4.2+ only.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #480 from dongjoon-hyun/SPARK-58568.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/Extension.swift | 6 ++++++
Tests/SparkConnectTests/DataFrameTests.swift | 17 +++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/Sources/SparkConnect/Extension.swift
b/Sources/SparkConnect/Extension.swift
index ab88213..b56ecf4 100644
--- a/Sources/SparkConnect/Extension.swift
+++ b/Sources/SparkConnect/Extension.swift
@@ -329,6 +329,12 @@ extension DataType {
"timestamp"
case .timestampNtz:
"timestamp_ntz"
+ case .time:
+ if self.time.hasPrecision {
+ "time(\(self.time.precision))"
+ } else {
+ "time(6)"
+ }
case .calendarInterval:
"interval"
case .yearMonthInterval:
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift
b/Tests/SparkConnectTests/DataFrameTests.swift
index 34a5108..d812745 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -154,6 +154,23 @@ struct DataFrameTests {
await spark.stop()
}
+ @Test
+ func dtypesTime() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ if await spark.version >= "4.2" {
+ try await spark.conf.set("spark.sql.timeType.enabled", "true")
+ let expected = [
+ ("TIME'12:34:56'", "time(6)"),
+ ("CAST(TIME'12:34:56' AS TIME(0))", "time(0)"),
+ ("CAST(TIME'12:34:56.123' AS TIME(3))", "time(3)"),
+ ]
+ for pair in expected {
+ #expect(try await spark.sql("SELECT \(pair.0)").dtypes[0].1 == pair.1)
+ }
+ }
+ await spark.stop()
+ }
+
@Test
func array() async throws {
let spark = try await SparkSession.builder.getOrCreate()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
