peterxcli opened a new issue, #5425: URL: https://github.com/apache/datafusion-comet/issues/5425
### What is the problem the feature request solves? Phase A of #4295 (#5407) preserves Spark `VariantType` for a whole value coming directly from an ordinary native Parquet scan. #5424 separately tracks native `variant_get` / `try_variant_get` evaluation for literal paths and scalar, non-Variant targets. Spark's two-argument forms default the target type to [`VariantType`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L601-L615): ```sql SELECT variant_get(v, '$.customer') FROM t; SELECT try_variant_get(v, '$.items[0]') FROM t; ``` These results need more than the physical Arrow `Struct<value, metadata>` datatype. Their logical identity is the parent Field extension marker `ARROW:extension:name = arrow.parquet.variant`, which Comet's JVM bridge recognizes explicitly in [`Utils.fromArrowField`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala#L72-L96). Comet builds native projects with DataFusion's [`ProjectionExec`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/operators/projection.rs#L45-L67). DataFusion's default [`PhysicalExpr::return_field`](https://github.com/apache/datafusion/blob/54.1.0/datafusion/physical-expr-common/src/physical_expr.rs#L75-L95) constructs a Field from only the expression datatype and nullability, so a Variant-producing expression that uses the default would describe its result as an ordinary Struct. DataFusion already provides the needed propagation point: [`ProjectionExprs::project_schema`](https://github.com/apache/datafusion/blob/54.1.0/datafusion/physical-expr/src/projection.rs#L445-L481) copies metadata returned by the expression into the aliased output Field. Comet's generic [`ScalarFunctionExpr` construction](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/planner.rs#L3498-L3504) currently supplies an unannotated `Field::new(...)`, so merely adding Variant as a function return datatype would still lose the logical marker. Without an explicit Variant result Field, a computed Variant can lose its logical identity and be imported by Spark as `StructType` instead of `VariantType`. ### Describe the potential solution Define the result contract for direct, top-level native expressions that return Variant, using existing Field-aware paths rather than adding another schema layer: - make each admitted Variant-producing physical expression return the canonical marked Field from `return_field`, reusing Comet's [`to_arrow_field`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/serde.rs#L203-L219); - return physical storage with exactly two ordinary Binary children in Spark order `[value, metadata]`, preserving parent nullability; - rebuild an extracted Variant so unused metadata entries are removed, matching [Spark's Variant-target behavior](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L430-L435); - allow `VariantType` through JVM expression/operator admission only for serializers that guarantee this contract, while retaining the general Variant fallback gates; - rely on DataFusion's existing projection metadata propagation and #5407's [RecordBatch-Field FFI export](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/jni_api.rs#L689-L738); and - use literal-path, two-argument `variant_get` and `try_variant_get` as the first end-to-end consumers, sharing the extraction semantics/kernel tracked by #5424. Focused tests should verify: - the native result's parent Field contains `arrow.parquet.variant`; - Spark reports the output datatype as `VariantType`, not `StructType`; - children are exactly `[value: Binary, metadata: Binary]` through aliases and with columns before/after the expression; - object, array, scalar, Variant JSON null, SQL NULL, and nullable-parent results match Spark; and - unsupported Variant-producing expressions and downstream consumers still fall back without losing the native scan beneath them. ### Additional context Related work: - #4295 — Variant scan/Iceberg umbrella - #5407 — whole-value ordinary-Parquet scan projection and Field-preserving FFI export - #5424 — scalar-target `variant_get` / `try_variant_get` evaluation - #3983 — shredded storage, subfield pruning, and predicate pushdown Non-goals: dynamic paths, nested Variant output, `parse_json` / `to_variant` and other Variant producers, C2R, shuffle/spill, Python transport, writes, and Iceberg. Supporting one direct Variant-valued native projection must not enable Variant indiscriminately for other operators. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
