peterxcli opened a new issue, #5424: URL: https://github.com/apache/datafusion-comet/issues/5424
### What is the problem the feature request solves? Phase A of #4295 (#5407) lets Comet carry a direct, top-level Spark `VariantType` value through an ordinary native Parquet scan. It intentionally leaves Variant expression evaluation on Spark: Comet still rejects a Variant input in [`CometAttributeReference`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/comet/serde/namedExpressions.scala#L37-L48) and rejects non-scan native operators whose schemas contain Variant in [`CometExecRule`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala#L741-L748). Common queries therefore still evaluate the extraction in Spark even when `v` came from a native scan: ```sql SELECT variant_get(v, '$.customer.id', 'bigint') FROM t; SELECT try_variant_get(v, '$.amount', 'decimal(18,2)') FROM t; ``` Spark implements both functions with the same [`VariantGet` expression](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L227-L280), with different cast-failure behavior. #4295 tracks Variant scans and separately mentions predicate pushdown for `variant_get(...) = literal`; it does not track native evaluation of these expressions. There is currently no dedicated issue for that work. ### Describe the potential solution Add expression-specific admission and native implementations for `variant_get` and `try_variant_get`, initially limited to foldable/literal paths and scalar, non-Variant target types. The implementation should: - admit `VariantType` only for these expressions instead of enabling it in the general datatype or operator gates; - reuse the whole-value `[value, metadata]` representation established by #5407; - implement Spark's object-key and array-index path syntax for foldable paths; - return SQL `NULL` for a missing path, an incompatible container shape, SQL NULL, or Variant JSON null; - match Spark's strict/try distinction: [`variant_get` raises on a target-type cast failure while `try_variant_get` returns `NULL`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L373-L438); - preserve Spark-compatible decimal, datetime/time-zone, nullability, invalid-path, and error behavior for supported scalar targets; and - keep dynamic paths, omitted target types / Variant-returning results, nested target types, and every unsupported shape on explicit Spark fallback. Add focused Spark SQL parity and plan tests covering projection and filtering, object and array paths, missing fields, JSON null versus SQL NULL, invalid paths, cast failure versus `try_variant_get`, decimals, and timestamp targets. The expression should run natively above an ordinary native Parquet scan for admitted inputs. ### Additional context Related work: - #4295 — Variant scan/Iceberg umbrella and the separate predicate-pushdown follow-up - #5407 — whole-value ordinary-Parquet projection - #3983 — shredded reader/writer support, subfield pruning, and predicate pushdown - #5031 — expression coverage audit that identifies Variant functions as gaps - #4084 / #2209 — safe fallback for Spark `PushVariantIntoScan` / `VariantStruct` Non-goals: predicate pushdown or subfield pruning, `parse_json` / `to_variant`, general Variant casts, C2R, shuffle/spill, Python, writes, and Iceberg. This boundary avoids pulling Variant-valued expression output and every Spark Variant function into the first native evaluator. Those cases should remain safe fallbacks and can be tracked separately when implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
