peterxcli opened a new issue, #5429: URL: https://github.com/apache/datafusion-comet/issues/5429
### What is the problem the feature request solves? Comet has no native predicate for distinguishing Variant JSON null from SQL NULL. Spark 4.0+ provides `is_variant_null`, and Spark 4.2 adds `is_valid_variant` for validating the Variant value/metadata pair. ```sql SELECT is_variant_null(v) FROM t; SELECT is_valid_variant(v) FROM t; -- Spark 4.2+ ``` Spark's [`is_variant_null`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L80-L119) returns true only for Variant null and false for SQL NULL and all other values; malformed physical data raises an error in its [evaluation helper](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L58-L70). Spark 4.2's [`is_valid_variant`](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L957-L987) returns true/false for a non-null Variant and SQL NULL for SQL NULL. ### Describe the potential solution Add expression-specific serializers and native evaluators over the canonical `[value, metadata]` representation: - implement Spark's exact JSON-null, SQL-NULL, malformed-value, and nullability behavior; - validate both Variant value and metadata bytes for `is_valid_variant`; - expose `is_valid_variant` only in the Spark 4.2 shim/profile while leaving Spark 4.0/4.1 compilation and registry behavior unchanged; and - keep Variant rejected for unrelated predicates/operators. Add focused parity and native-plan tests for every Variant scalar/container kind, Variant null, SQL NULL, malformed value bytes, malformed metadata bytes, and the Spark 4.2 version boundary. ### Additional context Spark 4.2 registers `is_valid_variant` alongside the existing Variant functions in its [tagged function registry](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L973-L983). It is not present in Spark 4.0/4.1. Related work: #4295, #5407, #5424, and #5425. Non-goals: comparisons, hash functions, parsing, casts, Variant mutation functions, C2R, shuffle/spill, writing, and Python transport. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
