MaxGekk opened a new pull request, #56198: URL: https://github.com/apache/spark/pull/56198
### What changes were proposed in this pull request? Implement columnar storage support for `TimestampNTZNanosType` and `TimestampLTZNanosType` across the column-vector stack. The layout mirrors `CalendarInterval`: each column gets two child vectors — a `Long` child for `epochMicros` and a `Short` child for `nanosWithinMicro` (range [0, 999]). Concretely: - **`ColumnVector`** — `getTimestampNTZNanos` / `getTimestampLTZNanos` now read from child vectors instead of throwing `SparkUnsupportedOperationException`. - **`WritableColumnVector`** — allocates the two child columns in the constructor; adds `putTimestampNTZNanos` / `putTimestampLTZNanos` write methods. - **`ConstantColumnVector`** — same child-column allocation; adds `setTimestampNanosVal` for the constant-value (partition-column) path. - **`RowToColumnConverter` (`Columnar.scala`)** — adds `TimestampNTZNanosConverter` / `TimestampLTZNanosConverter` objects (append `epochMicros` + `nanosWithinMicro` to children via `appendStruct`); routes nullable columns through `StructNullableTypeConverter`. - **`ColumnVectorUtils`** — handles both types in `populate` (constant-column path) and in `appendValue` (null and non-null branches). ### Why are the changes needed? SPARK-56981 added row-level physical representation for nanosecond timestamps, but columnar execution could not hold or move these values — any attempt to build a `ColumnarBatch` from rows containing nanosecond timestamps threw an unsupported-operation exception. This PR closes that gap. ### Does this PR introduce _any_ user-facing change? Yes. `ColumnarBatch` can now be built from `InternalRow`s containing `TimestampNTZNanosType` / `TimestampLTZNanosType` values. Previously this threw `SparkUnsupportedOperationException`. ### How was this patch tested? Added four unit tests to `RowToColumnConverterSuite`: - `TimestampNTZNanosType column roundtrip` — non-null values survive the row→column→read cycle. - `TimestampNTZNanosType column with nulls` — null slots are preserved correctly. - `TimestampLTZNanosType column roundtrip` — same for the LTZ variant. - `TimestampLTZNanosType column with nulls` — same for the LTZ variant. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 4.6 (claude.ai/code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
