This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 858da1729be4 [SPARK-54203][SQL] Support TimeType in
RowToColumnConverter
858da1729be4 is described below
commit 858da1729be4c468395dcba8da55e1a3d7eb35f4
Author: Maxim Gekk <[email protected]>
AuthorDate: Sat Jun 20 09:16:51 2026 +0200
[SPARK-54203][SQL] Support TimeType in RowToColumnConverter
### What changes were proposed in this pull request?
Add a `TimeType` branch (via the abstract `AnyTimeType`) to
`RowToColumnConverter.getConverterForType` in
`sql/core/.../execution/Columnar.scala`, routing TIME columns through the
long-backed `LongConverter` path. TIME is stored as nanos-of-day `Long`, and
the on/off-heap column vectors already reserve long storage for it
(`OnHeapColumnVector`, `OffHeapColumnVector`), so no column-vector changes are
required.
### Why are the changes needed?
Previously `getConverterForType` handled `LongType | TimestampType |
TimestampNTZType | DayTimeIntervalType` (all long-backed) but had no `TimeType`
case, so a TIME column fell through to `unsupportedDataTypeError`. This blocked
row-to-column conversion paths (e.g. `RowToColumnar` transitions, vectorized
execution) for the TIME data type.
This is a sub-task of
[SPARK-57550](https://issues.apache.org/jira/browse/SPARK-57550) (Extend
support for the TIME data type).
### Does this PR introduce _any_ user-facing change?
No. It enables an existing code path for the TIME data type that previously
threw an unsupported-type error.
### How was this patch tested?
Added round-trip and null-handling tests for `TimeType` to
`RowToColumnConverterSuite`:
- `TimeType column roundtrip` (precisions 0, 3, 6; values including `0L`
and end-of-day `86399999999999L`)
- `TimeType column with nulls`
Ran `build/sbt 'sql/testOnly *RowToColumnConverterSuite'` (all 13 tests
pass) and scalastyle on the sql module (0 errors).
### Was this patch authored or co-authored using generative AI tooling?
Yes, Generated-by: Cursor.
Closes #56613 from MaxGekk/time-RowToColumnConverter.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit a7ad18b858da75f4ae8e5c4691d77d7944e25685)
Signed-off-by: Max Gekk <[email protected]>
---
.../org/apache/spark/sql/execution/Columnar.scala | 3 ++-
.../sql/execution/RowToColumnConverterSuite.scala | 24 ++++++++++++++++++++++
2 files changed, 26 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
index 81fc5d2c1f73..5c98e3685327 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
@@ -265,7 +265,8 @@ private object RowToColumnConverter {
case ShortType => ShortConverter
case IntegerType | DateType | _: YearMonthIntervalType => IntConverter
case FloatType => FloatConverter
- case LongType | TimestampType | TimestampNTZType | _:
DayTimeIntervalType => LongConverter
+ case LongType | TimestampType | TimestampNTZType | _:
DayTimeIntervalType | _: TimeType =>
+ LongConverter
case DoubleType => DoubleConverter
case StringType => StringConverter
case _: GeographyType | _: GeometryType => BinaryViewConverter
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/RowToColumnConverterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/RowToColumnConverterSuite.scala
index 96d7fb6fbd09..d1d8e89de391 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/RowToColumnConverterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/RowToColumnConverterSuite.scala
@@ -184,6 +184,30 @@ class RowToColumnConverterSuite extends SparkFunSuite {
assert(vectors.head.getTimestampLTZNanos(2) ===
TimestampNanosVal.fromParts(200L, 1.toShort))
}
+ test("TimeType column roundtrip") {
+ Seq(0, 3, 6).foreach { precision =>
+ val schema = StructType(Seq(StructField("t", TimeType(precision))))
+ val values = Seq(0L, 12L * 60 * 60 * 1000 * 1000 * 1000, 86399999999999L)
+ val rows = values.map(v => InternalRow(v))
+ val vectors = convertRows(rows, schema)
+ values.zipWithIndex.foreach { case (v, i) =>
+ assert(vectors.head.getLong(i) === v)
+ }
+ }
+ }
+
+ test("TimeType column with nulls") {
+ val schema = StructType(Seq(StructField("t", TimeType(6), nullable =
true)))
+ val rows = Seq(
+ InternalRow(0L),
+ InternalRow(null),
+ InternalRow(86399999999999L))
+ val vectors = convertRows(rows, schema)
+ assert(vectors.head.getLong(0) === 0L)
+ assert(vectors.head.isNullAt(1))
+ assert(vectors.head.getLong(2) === 86399999999999L)
+ }
+
test("multiple columns") {
val schema = StructType(
Seq(StructField("s", ShortType), StructField("i", IntegerType),
StructField("l", LongType)))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
