peterxcli opened a new issue, #5433: URL: https://github.com/apache/datafusion-comet/issues/5433
### What is the problem the feature request solves? Comet currently falls back when an ordinary native Parquet write has a Spark `VariantType` column, even when the child already produces a whole Variant value. This prevents native scan-to-write copies such as reading `SELECT v FROM parquet_table` and writing the result back to Parquet. Spark 4.1.3 defines the ordinary, unshredded Parquet representation as an annotated group with required binary `value` and `metadata` children ([schema conversion](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L858-L866), [value writing](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala#L282-L292)). The existing arrow-rs 58.4 writer already maps an Arrow Struct Field carrying the Variant extension marker to the Parquet Variant logical annotation ([extension mapping](https://github.com/apache/arrow-rs/blob/58.4.0/parquet/src/arrow/schema/extension.rs#L110-L121)). This issue is limited to whole-value, unshredded output. #3983 continues to track shredded Variant reader/writer support. ### Describe the potential solution - Allow direct, top-level Variant fields only at the ordinary native Parquet writer boundary when the native child already provides canonical `[value, metadata]` storage. - Preserve `ARROW:extension:name=arrow.parquet.variant`, field name/nullability, Binary child types, and child order in the writer schema. - Reuse the existing protobuf-to-Arrow Field path and `parquet::arrow::ArrowWriter`; do not add another Variant dependency or encoder. - Write a Parquet `VARIANT(1)` annotated group containing exactly `value` and `metadata`. Verify Spark and Comet round trips for objects, arrays/scalars, Variant JSON null, SQL NULL, nullable parents, multiple Variant columns, and fields before/after Variant. Assert the plan contains the native writer, the Parquet footer has the Variant annotation, and Spark reads the result as `VariantType`. Preserve Spark 3.x behavior. ### Additional context Depends on the whole-value scan/Field identity work in #5407. Variant-valued native expression producers from #5425 may later feed this writer but are not required for the scan-to-write case. Shredded/`typed_value` output, nested Variant, Variant-producing expressions, Iceberg writes, casts, C2R, shuffle, and spill remain separate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
