Pastoray opened a new pull request, #394:
URL: https://github.com/apache/datafusion-spark/pull/394

   <!--
   SPDX-License-Identifier: Apache-2.0
   SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
   SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under 
one or more contributor license agreements
   -->
   
   ## Which issue does this PR close?
   
   - Closes #277.
   
   ## Rationale for this change
   
   `is_variant_null` is a Spark 4.0+ function that checks whether a VARIANT 
holds a
   JSON `null`. It is not implemented in this crate, so a query that calls it 
fails
   with an unknown function error.
   
   ## What changes are included in this PR?
   
   - Adds `is_variant_null(v)` in a new `function/variant` module, registered in
     `all_default_scalar_functions` and `expr_fn`.
   - Returns `true` only when the first byte of `value` is `0`, which is a 
variant
     null. Every other value is `false`, including a SQL NULL, so the result is 
a
     non-nullable BOOLEAN.
   - An empty `value` raises `[MALFORMED_VARIANT]` (SQLSTATE 22023), with the 
same
     message as Spark.
   - Only a VARIANT is accepted, with no implicit casts, as in Spark. A bare 
`NULL`
     literal is accepted and returns `false`, as in Spark.
   - This crate has no VARIANT type or `parse_json` yet, so a VARIANT is read 
as the
     struct of the `arrow.parquet.variant` extension type: binary `metadata` and
     `value` fields, as Binary, LargeBinary or BinaryView. Extra shredding 
fields
     such as `typed_value` are ignored. The function can adopt a real VARIANT 
type
     once Arrow and DataFusion have one.
   - Regenerates `docs/functions` with a new "Variant Functions" page.
   
   ## Are these changes tested?
   
   Yes.
   
   - Unit tests in `is_variant_null.rs` cover the header byte, SQL NULL rows, a 
NULL
     `value`, the malformed case, the three binary layouts, field order, 
shredded
     structs, scalar input, the non-nullable return field and the accepted
     argument types.
   - `variant/is_variant_null.slt` covers the same cases through SQL. It runs 
in both
     the simplify and evaluate modes and includes ANSI mode on and off.
   - I checked the expected values on Spark 4.2.0 with the PySpark runner
     (`pyspark_slt_runner.py`) and against the Spark 4.2.0 source for 
`IsVariantNull`
     and `VariantExpressionEvalUtils.isVariantNull`. Spark builds a VARIANT with
     `parse_json`, which this crate does not have, so the records that build 
one with
     `named_struct` are marked `skipif spark`. The runner passes the records it 
runs
     (10 pass, 10 skipped).
   - I also flipped the header comparison locally to confirm that both the unit
     tests and the `.slt` fail.
   
   ## Are there any user-facing changes?
   
   Yes, a new function: `is_variant_null`. There are no changes to existing
   functions and no breaking changes to the public Rust API.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to