peterxcli opened a new issue, #5431:
URL: https://github.com/apache/datafusion-comet/issues/5431
### What is the problem the feature request solves?
Spark 4.x exposes `to_variant_object` for converting a struct, array, or
string-keyed map into a Variant object/array. Comet does not serialize or
evaluate it, so construction from native columns falls back:
```sql
SELECT to_variant_object(named_struct('id', id, 'tags', tags)) FROM t;
SELECT to_variant_object(array(id, id + 1)) FROM t;
```
The SQL function is named `to_variant_object`—there is no Spark 4.1.3
function named `to_variant`. Spark registers the name in its [function
registry](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L888-L896).
Scalar-to-Variant conversion is expressed as `CAST(... AS VARIANT)`.
Spark restricts the root input to struct, array, or map and recursively
validates supported children in
[`ToVariantObject`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L121-L185).
Its encoder handles supported scalar, Variant, array, string-keyed map, and
struct values recursively in
[`VariantExpressionEvalUtils`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L73-L151).
### Describe the potential solution
Add Spark-4-specific serialization and a native recursive encoder that:
- accepts exactly Spark's root and nested datatype set;
- produces canonical Variant storage with Spark-compatible object-key
ordering and metadata IDs;
- preserves nested nulls as Variant null values while a null root remains
SQL NULL;
- preserves embedded Variant values without losing logical identity; and
- returns the marked `[value, metadata]` output Field required by #5425.
Add focused parity and native-plan tests for structs, arrays, maps,
recursively nested combinations, Variant children, SQL/nested nulls, Unicode
and empty object keys, duplicate map keys, unsupported map-key types, and field
ordering.
### Additional context
Related work: #4295, #5407, #5425, and the dedicated Variant-cast tracker.
Non-goals: parsing JSON strings, general casts, nested Variant column
transport, writing, shuffle/spill, C2R, Python transport, and Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
