Pastoray opened a new pull request, #394:
URL: https://github.com/apache/datafusion-spark/pull/394
<!--
SPDX-License-Identifier: Apache-2.0
SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under
one or more contributor license agreements
-->
## Which issue does this PR close?
- Closes #277.
## Rationale for this change
`is_variant_null` is a Spark 4.0+ function that checks whether a VARIANT
holds a
JSON `null`. It is not implemented in this crate, so a query that calls it
fails
with an unknown function error.
## What changes are included in this PR?
- Adds `is_variant_null(v)` in a new `function/variant` module, registered in
`all_default_scalar_functions` and `expr_fn`.
- Returns `true` only when the first byte of `value` is `0`, which is a
variant
null. Every other value is `false`, including a SQL NULL, so the result is
a
non-nullable BOOLEAN.
- An empty `value` raises `[MALFORMED_VARIANT]` (SQLSTATE 22023), with the
same
message as Spark.
- Only a VARIANT is accepted, with no implicit casts, as in Spark. A bare
`NULL`
literal is accepted and returns `false`, as in Spark.
- This crate has no VARIANT type or `parse_json` yet, so a VARIANT is read
as the
struct of the `arrow.parquet.variant` extension type: binary `metadata` and
`value` fields, as Binary, LargeBinary or BinaryView. Extra shredding
fields
such as `typed_value` are ignored. The function can adopt a real VARIANT
type
once Arrow and DataFusion have one.
- Regenerates `docs/functions` with a new "Variant Functions" page.
## Are these changes tested?
Yes.
- Unit tests in `is_variant_null.rs` cover the header byte, SQL NULL rows, a
NULL
`value`, the malformed case, the three binary layouts, field order,
shredded
structs, scalar input, the non-nullable return field and the accepted
argument types.
- `variant/is_variant_null.slt` covers the same cases through SQL. It runs
in both
the simplify and evaluate modes and includes ANSI mode on and off.
- I checked the expected values on Spark 4.2.0 with the PySpark runner
(`pyspark_slt_runner.py`) and against the Spark 4.2.0 source for
`IsVariantNull`
and `VariantExpressionEvalUtils.isVariantNull`. Spark builds a VARIANT with
`parse_json`, which this crate does not have, so the records that build
one with
`named_struct` are marked `skipif spark`. The runner passes the records it
runs
(10 pass, 10 skipped).
- I also flipped the header comparison locally to confirm that both the unit
tests and the `.slt` fail.
## Are there any user-facing changes?
Yes, a new function: `is_variant_null`. There are no changes to existing
functions and no breaking changes to the public Rust API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]