David Milicevic created SPARK-56076:
---------------------------------------
Summary: Types Framework - Phase 1b - Generic Row Access
Key: SPARK-56076
URL: https://issues.apache.org/jira/browse/SPARK-56076
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.2.0
Reporter: David Milicevic
*Description:*
{*}{*}Add generic row read/write support so that framework types can be stored
in and retrieved from UnsafeRow without per-type edits to row interfaces.
*What* *this* *includes:*
* {*}{*}Two new TypeOps methods: getValue(input: SpecializedGetters, ordinal:
Int): Any and unsafeWrite(writer: UnsafeWriter, ordinal: Int, value: Any): Unit
* PhysicalFrameworkDelegatedType(ops: TypeOps) — a general-purpose delegated
physical type. Dispatchers match on it and call ops.getValue() /
ops.unsafeWrite() without knowing the storage format (struct, binary, etc.)
* TypeOps.getOrNull(dt) helper for clean dispatch in Java files (ColumnarRow,
ColumnarBatchRow)
* Integration in ~10 files: TypeOps.scala, TimeTypeOps.scala,
PhysicalDataType.scala, SpecializedGettersReader.java,
InterpretedUnsafeProjection.scala, ColumnarRow.java, ColumnarBatchRow.java,
InternalRow.scala, literals.scala
* TimeType reference implementation (Long-backed, simple case)
*Note:*
{*}{*}unsafeRead was dropped — it's redundant because UnsafeRow implements
SpecializedGetters, so getValue(unsafeRow, ordinal) already covers the read
path. PhysicalFrameworkDelegatedType replaces the earlier PhysicalOpaqueType
concept from the design doc — the name conveys both framework ownership and
delegated behavior. It takes no structFields parameter; the TypeOps handles
storage format internally, making it work for struct-backed types,
binary-backed types, or anything else.
Struct-writing utility methods in InterpretedUnsafeProjection are widened to
private[catalyst] so that future TypeOps implementations can reuse the same
struct-writing machinery.
*Design doc:*
Linked in the parent work item.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
