MaxGekk opened a new pull request, #56858:
URL: https://github.com/apache/spark/pull/56858
### What changes were proposed in this pull request?
This PR adds `TimeType` support to the vectorized-reader column population
path.
`ColumnVectorUtils.toBatch` (via `appendValue`) used an explicit type
dispatch with no `TimeType` case and threw `_LEGACY_ERROR_TEMP_3192` ("Datatype
not supported") for TIME values. This PR adds a `TimeType` branch that appends
the internal nanos-of-day `long` using `DateTimeUtils.localTimeToNanos`.
The constant / partition / missing column path
(`ColumnVectorUtils.populate`) already worked, because `TimeType` maps to
`PhysicalLongType` (handled by the existing `PhysicalLongType` branch, and for
nested array/map values by `RowToColumnConverter`), so no change was needed
there.
This is a sub-task of SPARK-54203 (Support TimeType in
`RowToColumnConverter`), which landed the converter and column-vector
allocation support that gated this work.
### Why are the changes needed?
So that vectorized reads with TIME partition / missing / constant columns,
and `ColumnVectorUtils.toBatch` conversions involving TIME, populate correctly
instead of failing with an unsupported-datatype error. This brings TIME in line
with `DATE` / `TIMESTAMP` / interval types in this layer.
### Does this PR introduce _any_ user-facing change?
No. TIME is an in-progress, not-yet-released data type; this only widens
internal vectorized support so that previously-failing TIME column population
now succeeds.
### How was this patch tested?
New unit tests:
- `ColumnVectorUtilsSuite` (the `populate` / constant-column path): TIME
across precisions 0/6/7/9, boundary values (`00:00:00`, `23:59:59.999999999`),
null (missing column), and TIME nested in struct / array / map.
- `ColumnarBatchSuite` (the `toBatch` path): `TimeType` added to the
random-schema test and `compareStruct` (top-level and array element), a
per-precision `testVector` (0/6/7/9 + boundaries), a nested struct/array
`toBatch` test, and a negative unsupported-type case.
Ran `build/sbt 'sql/testOnly *ColumnVectorUtilsSuite *ColumnarBatchSuite'`
(93 tests pass). Scalastyle and Java checkstyle pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
