[
https://issues.apache.org/jira/browse/SPARK-57569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094073#comment-18094073
]
Shrirang Mhalgi commented on SPARK-57569:
-----------------------------------------
Investigated this - it appears blocked on SPARK-52621 (TIME Variant encoding),
which is still unresolved and has an open design question on aligning with the
Variant spec. The shredding path needs the TIME type ID to be defined before
reconstruction can be implemented.
> Support TimeType in Parquet Variant shredding
> ---------------------------------------------
>
> Key: SPARK-57569
> URL: https://issues.apache.org/jira/browse/SPARK-57569
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. What
> Support {{TimeType}} in Parquet Variant shredding (columnar reconstruction of
> shredded
> Variant values).
> h2. Where
> {{SparkShreddingUtils}}
> (sql/core/.../execution/datasources/parquet/SparkShreddingUtils.scala, around
> line 926)
> reconstructs shredded Variant values through a columnar batch.
> h2. Relationship to SPARK-54203
> This area is gated at the row-to-columnar converter level by SPARK-54203
> ({{RowToColumnConverter.getConverterForType}} in
> sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala), which
> currently has
> no {{TimeType}} case. Once SPARK-54203 lands, the work below becomes
> possible. It may also
> require TIME support in this component's own layer (Arrow type mapping,
> Parquet/ORC logical
> types, or Variant encoding).
> h2. Related
> * TIME <-> VARIANT casting and encoding is tracked by SPARK-52621; this
> ticket covers the
> shredded columnar reconstruction path specifically.
> h2. Acceptance criteria
> * A shredded Variant containing TIME values reconstructs correctly via the
> columnar path.
> * Tests added in the Variant shredding suites.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
