andygrove opened a new issue, #277:
URL: https://github.com/apache/datafusion-spark/issues/277

   Part of #273 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `is_variant_null(v)` returns `true` only when `v` is a variant null, meaning 
a JSON `null` stored inside a VARIANT.
   In every other case it returns `false`, and that includes a SQL `NULL` 
input. The result is a non-nullable
   `BOOLEAN`. The only accepted argument type is `VARIANT`, and there are no 
implicit casts. Added in Spark 4.0.0.
   
   The check is a single byte: the first value byte must be `0x00` (basic type 
PRIMITIVE, type id NULL). This depends
   on the VARIANT Arrow representation proposed in the `parse_json` issue, 
which uses the canonical
   `arrow.parquet.variant` extension with `metadata` and `value` binary 
children.
   
   ### Examples
   
   ```sql
   SELECT is_variant_null(parse_json('null'));
   -- true
   SELECT is_variant_null(parse_json('"null"'));
   -- false
   SELECT is_variant_null(parse_json('13'));
   -- false
   SELECT is_variant_null(parse_json(null));
   -- false
   SELECT is_variant_null(variant_get(parse_json('{"a":null, "b":"spark"}'), 
'$.c'));
   -- false
   SELECT is_variant_null(variant_get(parse_json('{"a":null, "b":"spark"}'), 
'$.a'));
   -- true
   SELECT is_variant_null(variant_get(parse_json('{ "a": null, "b": {"c": null, 
"d": [13, null]} }'), '$.b.d[1]'));
   -- true
   SELECT is_variant_null(variant_get(parse_json('{ "a": null, "b": {"c": null, 
"d": [13, null]} }'), '$.b.d'));
   -- false
   ```
   
   These come from the `ExpressionDescription` and `VariantEndToEndSuite`.
   
   ### Edge cases to test
   
   - SQL `NULL` returns `false`, not NULL. `StaticInvoke` uses `propagateNull = 
false, returnNullable = false`.
     A missing path in `variant_get` gives a SQL NULL, so it also returns 
`false`.
   - Cases that return `false`:
     - scalars: the string `"null"`, `""`, `0`, `true`, `false`, large ints 
such as `2147483648`, and decimals;
     - a long string (`"spark"` repeated 100 times);
     - arrays and objects, including `[]`, `[null, null]` and `[{"a" : 13, "b" 
: null}]`.
     These all come from `VariantExpressionEvalUtilsSuite."isVariantNull"`.
   - A variant null nested in an object or array reached through `variant_get` 
without a target type is still a
     variant null, so the result is `true`. With a concrete target type 
`variant_get` returns SQL NULL instead, so
     `is_variant_null(variant_get(v, '$.a', 'int'))` is `false`.
   - A malformed value with an empty `value` binary raises `MALFORMED_VARIANT` 
("Variant binary is malformed. Please
     check the data source is valid."). Spark does not validate anything beyond 
the first byte.
   - Check that metadata is ignored: any valid header, including 
`sorted_strings` set or 2-4 byte offsets, gives the
     same answer.
   - Test both a column and a literal input. Also test a variant produced by 
`to_variant_object` or by
     `cast(x as variant)` from a NULL array element.
   - The result does not depend on ANSI mode.
   
   ### Differences across Spark versions
   
   - 3.4 and 3.5 have no VARIANT type.
   - No differences between 4.0.4, 4.1.3 and 4.2.0.
   
   ### Implementation notes
   
   - Spark: `IsVariantNull` in
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala`.
     The logic is in `VariantExpressionEvalUtils.isVariantNull`, in the same 
package.
   - DataFusion core has no VARIANT support. arrow-rs 
`parquet_variant::Variant::Null` can be used for a decoded
     check, but reading `value[0] == 0` from the `value` child of the struct 
array is enough and avoids decoding.
   - When the struct row itself is null, return `false`. A row with an empty 
`value` binary should return
     `MALFORMED_VARIANT`.
   - apache/datafusion-comet#5438 tracks native Variant support in Comet.
   - This is blocked on the VARIANT type representation from the `parse_json` 
issue. Put it in a new
     `crates/spark/src/function/variant/` module. Add 
`crates/sqllogictest/test_files/variant/is_variant_null.slt`
     and document the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to