WangGuangxin opened a new pull request, #58301:
URL: https://github.com/apache/spark/pull/58301
### What changes were proposed in this pull request?
This PR adds a new `variant_array_length` expression that returns the number
of elements in a variant array.
If the input is a SQL NULL, the function returns NULL. If the input variant
is not an array, including variant null, object, or scalar values, the function
returns NULL. If the input is an empty array, the
function returns 0.
Usage examples:
```sql
> SELECT variant_array_length(parse_json('[1,2,3]'));
3
> SELECT variant_array_length(parse_json('[]'));
0
> SELECT variant_array_length(parse_json('{"a":1}'));
NULL
> SELECT variant_array_length(parse_json('null'));
NULL
> SELECT variant_array_length(NULL);
NULL
```
### Why are the changes needed?
Spark already supports expanding variant arrays through variant_explode, but
using a generator is unnecessarily cumbersome when users only need the array
length. Users currently need to explode the value and
count rows, or cast the extracted value into a typed array when the element
type is known.
variant_array_length provides a direct scalar operation for a common
semi-structured data use case: inspecting array cardinality before filtering,
validating, or transforming variant data. It is also cheaper and
easier to optimize than expanding the array when only the number of elements
is needed, because the variant binary format stores array size in the array
header.
This function also improves parity with Spark's JSON function surface, which
already provides json_array_length for JSON strings. variant_array_length
provides the corresponding operation for parsed variant
values.
### Does this PR introduce any user-facing change?
Yes. This PR adds a new SQL function, variant_array_length.
Example:
```sql
SELECT variant_array_length(parse_json('[1,2,3]'));
```
returns:
```
3
```
### How was this patch tested?
Unit tests were added for variant_array_length, including:
- variant array with multiple elements
- empty variant array
- SQL NULL input
- variant null input
- non-array variant inputs such as objects and scalars
- nested usage together with variant_get
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: TraeCode (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
