peterxcli opened a new issue, #5438: URL: https://github.com/apache/datafusion-comet/issues/5438
### What is the problem the feature request solves? Spark 4 introduced [<code>VariantType</code>](https://github.com/apache/spark/blob/v4.1.3/sql/api/src/main/scala/org/apache/spark/sql/types/VariantType.scala#L22-L43) for semi-structured values. Comet currently has isolated Variant fallbacks and partial scan work, but no single roadmap for carrying Variant through an otherwise native query: <pre> Parquet / Iceberg -> Arrow Field + array -> native expressions and operators -> Arrow C Data Interface -> Spark ColumnVector / UnsafeRow / Python -> shuffle, spill, and writes </pre> This epic coordinates the existing Variant issues into an incremental, fail-closed implementation plan. It does not replace the focused trackers: [#4295](https://github.com/apache/datafusion-comet/issues/4295) remains the scan umbrella and [#3983](https://github.com/apache/datafusion-comet/issues/3983) remains the shredded-Parquet tracker. #### Why this matters - Keep otherwise native Spark 4 plans native when they project, inspect, transform, transport, or write semi-structured data. - Avoid unnecessary JVM/native transitions and repeated Variant decoding and encoding. - Establish one Spark-compatible representation across Parquet, Arrow, FFI, native operators, UnsafeRow, Python Arrow, shuffle, and spill. - Unlock useful JSON-like workloads incrementally: whole-value projection first, then extraction and construction, followed by wider transport and storage. - Make shredded subfield pruning and predicate pushdown an optimization built on a correct whole-value path rather than a separate representation. - Preserve explicit Spark fallback for every boundary that has not yet been audited, with Spark 3.x behavior unchanged. #### Canonical representation The contract used by every tracker in this epic should be: - Spark logical type: <code>VariantType</code>. - Arrow physical storage: <code>Struct<value: Binary, metadata: Binary></code>. - Spark output child order: exactly <code>[value, metadata]</code>. Spark's [Arrow conversion](https://github.com/apache/spark/blob/v4.1.3/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala#L183-L197) creates that order, and [<code>ColumnVector.getVariant</code>](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java#L329-L337) consumes child 0 as value and child 1 as metadata. - Logical identity: <code>ARROW:extension:name=arrow.parquet.variant</code> on the individual outer Arrow <code>Field</code> representing the Variant column. It is not a property of <code>DataType::Struct</code>, and an arbitrary struct named <code>value</code>/<code>metadata</code> must never be inferred as Variant. Arrow-rs recognizes the marker when constructing a [<code>VariantArray</code>](https://github.com/apache/arrow-rs/blob/58.4.0/parquet-variant-compute/src/variant_array.rs#L77-L109). ### Describe the potential solution #### Implementation principles 1. Reuse the canonical marked Arrow Field and whole-value normalization established by [#5407](https://github.com/apache/datafusion-comet/pull/5407). 2. Normalize Parquet output once. Use Arrow-rs [<code>unshred_variant</code>](https://github.com/apache/arrow-rs/blob/58.4.0/parquet-variant-compute/src/unshred_variant.rs#L46-L96), remove <code>typed_value</code>, and export ordinary Binary children in Spark order. 3. Preserve the full Arrow Field—not only its DataType—at every schema-producing and FFI boundary. This also matters for expressions because DataFusion's default [<code>PhysicalExpr::return_field</code>](https://github.com/apache/datafusion/blob/54.1.0/datafusion/physical-expr-common/src/physical_expr.rs#L75-L95) derives a Field from the DataType unless an expression supplies richer metadata. 4. Keep Variant admission explicit per operator. Serialization of the type is not evidence that an expression, C2R path, shuffle, writer, or Python boundary supports its semantics. 5. Use Spark-version shims so Spark 3.x neither references Variant classes nor changes behavior. 6. Deliver each tracker with value-parity, plan/admission, schema/vector-layout, and fallback tests appropriate to that boundary. #### Implementation order ##### Phase 0 — safety baseline and representation foundation - [x] [#5377](https://github.com/apache/datafusion-comet/pull/5377) — prune an unread Variant column without decoding it. - [ ] [#5407](https://github.com/apache/datafusion-comet/pull/5407) — project a top-level whole Variant value from ordinary Parquet; establish protobuf identity, canonical Arrow Field construction, one-time unshredding/normalization, Field-aware FFI export, and Spark <code>getVariant</code> bridging. All later phases should reuse #5407's representation instead of adding a second Variant encoding. ##### Phase 1 — first native read/compute vertical slice - [ ] [#5429](https://github.com/apache/datafusion-comet/issues/5429) — <code>is_variant_null</code> and Spark 4.2 <code>is_valid_variant</code>. This can proceed after #5407 and validates basic decoding, SQL NULL, and Variant JSON null behavior. - [ ] [#5424](https://github.com/apache/datafusion-comet/issues/5424) — literal-path <code>variant_get</code>/<code>try_variant_get</code> with scalar targets. This establishes the path parser, strict/try errors, and scalar conversion semantics. - [ ] [#5425](https://github.com/apache/datafusion-comet/issues/5425) — Variant-valued expression output and two-argument <code>variant_get</code>. This depends on #5407's Field contract and should reuse #5424's extraction kernel. ##### Phase 2 — parallel scan, storage-shape, and row consumers Once #5407 is merged, these can proceed largely in parallel: - [ ] [#4295](https://github.com/apache/datafusion-comet/issues/4295) — complete whole-value Variant projection for Iceberg, including tables with legal primitive equality-delete keys. - [ ] [#5435](https://github.com/apache/datafusion-comet/issues/5435) — project Variant nested in struct, array, and map columns. - [ ] [#5433](https://github.com/apache/datafusion-comet/issues/5433) — write top-level whole-value Variant to ordinary Parquet. - [ ] [#5436](https://github.com/apache/datafusion-comet/issues/5436) — encode Variant in native columnar-to-row/UnsafeRow. Spark uses a dedicated [length + value + metadata payload](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeWriter.java#L164-L182), not ordinary Struct row encoding. ##### Phase 3 — expression completion and Variant producers - [ ] [#5426](https://github.com/apache/datafusion-comet/issues/5426) — dynamic paths and nested targets for <code>variant_get</code>/<code>try_variant_get</code>, after #5424 and #5425. - [ ] [#5430](https://github.com/apache/datafusion-comet/issues/5430) — casts to and from Variant, reusing the extraction/conversion and Variant-output contracts from #5424/#5425. - [ ] [#5428](https://github.com/apache/datafusion-comet/issues/5428) — <code>parse_json</code>/<code>try_parse_json</code>, after Variant-valued output is supported. - [ ] [#5431](https://github.com/apache/datafusion-comet/issues/5431) — <code>to_variant_object</code>, preferably reusing the scalar-to-Variant encoder from #5430. - [ ] [#5427](https://github.com/apache/datafusion-comet/issues/5427) — <code>schema_of_variant</code>/<code>schema_of_variant_agg</code>, reusing the decoder established by #5424. - [ ] [#5432](https://github.com/apache/datafusion-comet/issues/5432) — <code>variant_explode</code>/<code>variant_explode_outer</code>, after Variant-valued output Fields are supported by #5425. Independent items in this phase may be implemented in parallel once their listed prerequisites land. ##### Phase 4 — wider transport - [ ] [#5434](https://github.com/apache/datafusion-comet/issues/5434) — carry Variant through Comet shuffle and spill. Arrow-native transport can build on #5407; any JVM row path should reuse the UnsafeRow codec from #5436. - [ ] [#5437](https://github.com/apache/datafusion-comet/issues/5437) — support top-level Variant in <code>MapInArrow</code>/<code>MapInPandas</code>, after #5407 and #5425 establish input and output Field identity. Spark's Python conversion explicitly preserves the [Variant extension Field](https://github.com/apache/spark/blob/v4.1.3/python/pyspark/sql/pandas/types.py#L201-L208). ##### Phase 5 — shredded storage and optimization - [ ] [#3983](https://github.com/apache/datafusion-comet/issues/3983) — implement in dependency-sized subphases: - shredded reader/unshredding on top of #5407; - shredded writer on top of #5433; - subfield pruning and predicate pushdown on top of #5424/#5426. #3983 should not be one serial blocker for whole-value scans, expressions, or transport. #### Difficult points 1. **Logical identity lives on an Arrow Field.** The extension marker can be lost whenever code reconstructs a schema from only an array DataType. Every projection, expression result, batch rewrite, FFI export, shuffle schema, Python schema, and writer boundary must preserve it. 2. **Physical Parquet input is not one fixed Struct shape.** Children can arrive in another order, may include <code>typed_value</code>, and metadata may be dictionary encoded. Binary children may be Binary, LargeBinary, or BinaryView. Parent nulls, field name, nullability, and metadata must survive normalization. 3. **Spark and Arrow use different object-key ordering.** Spark lookup follows Java UTF-16 ordering, while Arrow validation/encoding uses UTF-8 ordering. Values may need normalization without changing metadata field IDs or SQL-visible content. 4. **There are two kinds of null.** SQL NULL is the parent Struct null; Variant JSON null is a valid non-null Variant payload. Operators, generators, casts, writers, and transports must preserve that distinction. 5. **Expression semantics are broader than decoding bytes.** <code>variant_get</code>, casts, and parsers need Spark-compatible path syntax, strict versus try behavior, target-type conversion, decimal overflow, timestamp/time-zone behavior, malformed-input errors, and size limits. 6. **Each boundary has a different physical contract.** Arrow needs the extension Field, Spark vectors need <code>[value, metadata]</code>, UnsafeRow uses a dedicated packed payload, Python Arrow needs child metadata, and Parquet needs the Variant logical annotation. 7. **Nested Variant is recursive schema work.** Struct, list, and map offsets, child nullability, Field metadata, pruning, and rebuilding must all remain aligned. 8. **Fallback admission is part of correctness.** Until a boundary is implemented and tested, Variant must remain on Spark rather than leak into a native operator as an ordinary Struct. 9. **Version boundaries matter.** Spark 3.x has no VariantType; Spark 4.0, 4.1, and 4.2 expose different functions. Code and tests need version-specific shims rather than reflection spread through shared paths. 10. **Iceberg adds schema IDs and spec constraints.** Whole-value projection must preserve Iceberg field identity and delete semantics in addition to the Arrow/Parquet contract. ### Additional context #### Upstream semantic boundaries This epic intentionally does not add trackers for behavior that upstream does not define: - Spark does not define general Variant ordering: [<code>PhysicalVariantType.ordering</code> throws](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/PhysicalDataType.scala#L397-L407). - Spark hash expressions [reject schemas containing Variant](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala#L294-L317). Native Variant comparisons and hashing are therefore not goals until Spark defines them. - Iceberg identifier fields must be primitive, and equality-delete columns inherit those restrictions ([identifier fields](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/format/spec.md#L368-L374), [equality deletes](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/format/spec.md#L1175-L1183)). Variant itself is not a valid equality-delete key. Reading Variant alongside legal primitive-key equality deletes remains part of #4295. - Functions introduced only after the supported Spark releases should receive their own tracker when the corresponding Spark profile exists, rather than being speculatively implemented here. #### Epic-level definition of done - Supported Spark 4 profiles preserve Variant logical identity and exact <code>[value, metadata]</code> layout through every completed boundary. - Object, array, scalar, Variant JSON null, SQL NULL, nullable parents, Unicode keys, and malformed values have Spark parity where applicable. - Whole-value, shredded, and nested inputs produce the same logical values. - Supported plans remain native; unsupported plans have focused tests proving explicit Spark fallback. - Spark 3.x compilation and behavior remain unchanged. - No code path identifies Variant from an unmarked ordinary Struct. - Each completed issue updates this checklist and documents any remaining fallback boundary. Historical fallback evidence: [#2209](https://github.com/apache/datafusion-comet/issues/2209). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
