Max Gekk created SPARK-57744:
--------------------------------
Summary: Types Framework - migrate vectorized column population
(ColumnVectorUtils.appendValue and Columnar row getters)
Key: SPARK-57744
URL: https://issues.apache.org/jira/browse/SPARK-57744
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
h2. What
Migrate the vectorized-reader column-population code paths to dispatch through
the Types Framework (TypeOps / PhysicalDataType) instead of hand-rolled
per-type {{instanceof}} chains, so any framework-registered type (TimeType
today, future types tomorrow) is handled without per-type edits.
h2. Where
* {{sql/core .../execution/vectorized/ColumnVectorUtils.java}} --
{{appendValue(WritableColumnVector, DataType, Object)}} (the {{toBatch}} path):
route the external -> internal conversion through {{TypeOps.toCatalyst}} and
append according to {{getPhysicalType}}, replacing the tail of the explicit
{{instanceof}} chain.
* {{sql/catalyst .../vectorized/ColumnarRow.java}} and
{{ColumnarBatchRow.java}} -- {{get(int, DataType)}}: delegate to
{{SpecializedGettersReader.read(...)}} (which dispatches on
{{PhysicalDataType}}), mirroring {{ColumnarArray.get}}, which already does this.
h2. Why
The populate path and the {{copy()}} methods already dispatch on
{{PhysicalDataType}} and therefore support framework types for free, and
{{ColumnarArray.get}} already delegates to {{SpecializedGettersReader}}.
{{appendValue}} and the two row getters are the only remaining hand-maintained
{{instanceof}} chains in this layer. Bringing them onto the framework removes
duplicated dispatch and means new framework types need no edits here.
h2. Relationship
* Parent: SPARK-53504 (Types Framework umbrella).
* Related: SPARK-56076 (Phase 1b - Generic Row Access) overlaps with the
{{get(int, DataType)}} delegation.
* Follows: SPARK-57570, which added an explicit TimeType case to
{{appendValue}} as a focused fix; this task generalizes that case through the
framework.
h2. Acceptance criteria
* {{appendValue}} and the columnar row getters no longer special-case
individual framework types; TIME (and other framework types) flow through
automatically via {{PhysicalDataType}} / {{TypeOps}}.
* Behavior preserved for existing types. Call out in review: delegating the
getters changes the unsupported-type error code ({{_LEGACY_ERROR_TEMP_3152}} /
{{_LEGACY_ERROR_TEMP_3155}} -> {{_LEGACY_ERROR_TEMP_3131}}), and
{{ColumnarRow.get}} gains {{CalendarIntervalType}} support it currently lacks.
* {{ColumnVectorUtilsSuite}} / {{ColumnarBatchSuite}} continue to pass; the
per-precision TIME {{testVector}} in {{ColumnarBatchSuite}} can then assert via
{{batchRow.get(0, dt)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
