peterxcli opened a new issue, #5432: URL: https://github.com/apache/datafusion-comet/issues/5432
### What is the problem the feature request solves? Spark 4.x provides `variant_explode` and `variant_explode_outer` to turn a Variant object or array into rows. Comet does not support these generators, so queries must leave native execution: ```sql SELECT id, e.* FROM t LATERAL VIEW variant_explode(v) e AS pos, key, value; SELECT id, e.* FROM t LATERAL VIEW variant_explode_outer(v) e AS pos, key, value; ``` Spark defines the generated columns as `pos INT NOT NULL`, `key STRING`, and `value VARIANT NOT NULL`, and implements object/array iteration in [`VariantExplode`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L664-L775). The exact SQL names are registered as generators in Spark's [4.1.3 function registry](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L1233-L1240). ### Describe the potential solution Add a Spark-4-specific generator path that: - iterates arrays by index and objects by their stored field order; - emits Spark-compatible `pos`, nullable `key`, and Variant `value` columns; - preserves the Variant result Field marker and canonical `[value, metadata]` storage for each emitted value; - matches Spark behavior for scalar values, Variant null, SQL NULL, empty containers, and the outer form; and - keeps unrelated Variant-bearing generators/operators on explicit fallback. Add focused parity and native-plan tests for objects, arrays, nested values, empty containers, scalars, Variant/SQL nulls, Unicode keys, aliases, lateral-view syntax, and outer versus non-outer cardinality. ### Additional context Related work: #4295, #5407, #5425, and the general generator execution path. Non-goals: Variant mutation, subfield pruning or pushdown, shuffle/spill, C2R, writing, Python transport, and Iceberg-specific work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
