peterxcli opened a new issue, #5427: URL: https://github.com/apache/datafusion-comet/issues/5427
### What is the problem the feature request solves? Spark 4.x provides `schema_of_variant` for a single value and `schema_of_variant_agg` for the merged schema of a Variant column. Comet does not implement either expression, so schema discovery over a natively scanned Variant column falls back: ```sql SELECT schema_of_variant(v) FROM t; SELECT schema_of_variant_agg(v) FROM t; ``` Spark's scalar implementation infers and prints the schema, using `OBJECT<...>` rather than `STRUCT<...>`, in [`SchemaOfVariant`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L778-L890). The aggregate shares that inference and merges schemas across rows and partial buffers in [`SchemaOfVariantAgg`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L892-L953). ### Describe the potential solution Implement one shared Spark-compatible Variant schema inference/merge helper, then expose it through the scalar expression and aggregate: - infer scalar, decimal precision/scale, date/timestamp, binary, UUID, array, object, and Variant-null (`VOID`) schemas; - keep object fields in Spark's required order and print `OBJECT<...>` names with Spark-compatible quoting; - merge heterogeneous array elements and object fields with Spark's compatible-type rules; - ignore SQL NULL rows in the aggregate, start/finish an empty buffer as `VOID`, and support partial-buffer merge/serialization; and - admit Variant only for these two expressions while preserving general fallback gates. Add focused parity and native-plan tests for every scalar kind, JSON versus SQL NULL, nested arrays/objects, heterogeneous values, decimal widening, field-name quoting and ordering, empty/all-null inputs, grouping, and multi-partition partial aggregation. ### Additional context Related work: #4295, #5407, #5424, and #5425. Non-goals: schema-driven Variant casts, subfield pruning, predicate pushdown, writing, shuffle/spill of Variant values, C2R, Python transport, and Iceberg-specific work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
