andygrove opened a new issue, #222:
URL: https://github.com/apache/datafusion-spark/issues/222
Part of #178 and apache/datafusion-comet#7300.
### What the function does
`array_union(array1, array2)` returns the distinct elements of `array1`
followed by the distinct elements of `array2` that have not been seen yet.
Order is first occurrence across `array1 ++ array2`. At most one NULL element
is kept.
Both arguments must be arrays. Through
`BinaryArrayExpressionWithImplicitCast` (`ImplicitCastInputTypes`), the
analyzer casts both to `ARRAY<T>`, where `T =
TypeCoercion.findTightestCommonType(e1, e2)`. That covers numeric widening
(`INT` + `BIGINT` -> `BIGINT`, `INT` + `DOUBLE` -> `DOUBLE`), NULL type,
datetime widening, and string collation merging. It does not include string
promotion: if there is no tightest common type, analysis fails with
`DATATYPE_MISMATCH.BINARY_ARRAY_DIFF_TYPES`. An untyped `NULL` argument (type
`VOID`) is also rejected. The element type must be orderable, so `MAP` and
`VARIANT` elements fail with `DATATYPE_MISMATCH.INVALID_ORDERING_TYPE`.
The result type is `ARRAY<T>` with `containsNull = left.containsNull ||
right.containsNull` (`ComplexTypeMergingExpression`). The function is
null-intolerant: if either array is NULL, the result is NULL. Available since
2.4.0, no aliases.
### Examples
```sql
SELECT array_union(array(1, 2, 3), array(1, 3, 5));
-- [1,2,3,5]
SELECT array_union(array(1, 2, null, 4, 5), array(-5, 4, -3, 2, -1));
-- [1,2,null,4,5,-5,-3,-1]
SELECT array_union(array(-5, 4, -3, 2, 4), array(1, 2, null, 4, 5));
-- [-5,4,-3,2,1,null,5]
SELECT array_union(array('b', 'a', 'c'), array('b', null, 'a', 'g'));
-- ["b","a","c",null,"g"]
SELECT array_union(array(null, null), array(null, null));
-- [null]
SELECT array_union(array('a'), CAST(NULL AS ARRAY<STRING>));
-- NULL
SELECT array_union(array(array(1, 2), array(3, 4)), array(array(5, 6),
array(2, 1)));
-- [[1,2],[3,4],[5,6],[2,1]]
-- Note: 0.0 and -0.0 here are DECIMAL literals (both equal 0).
SELECT array_union(array(0.0, -0.0, DOUBLE("NaN")), array(0.0, -0.0,
DOUBLE("NaN")));
-- [0.0,NaN]
-- Spark 4.2.0 (SPARK-54918):
SELECT array_union(array(-0.0D), array(0.0D));
-- [0.0]
-- Spark 3.5.x to 4.1.3: [-0.0,0.0]
SELECT array_union(NULL, array('a'));
-- error: DATATYPE_MISMATCH.BINARY_ARRAY_DIFF_TYPES (leftType "VOID")
```
### Edge cases to test
- NULL in each argument (typed NULL arrays) gives NULL. Untyped `NULL` is an
analysis error.
- Empty array on either side. `array_union(array(1, 2, null, 4, 5),
CAST(array() AS ARRAY<INT>))` returns `[1,2,null,4,5]`.
- NULL elements on both sides: only one NULL, at its first position.
- Duplicates within `array1` are removed too (`array(1, 2, 3, 2)` with
`array(4, 2, 4)` gives `[1,2,3,4]`).
- Element type coercion: `ARRAY<INT>` with `ARRAY<BIGINT>`, `ARRAY<INT>`
with `ARRAY<DOUBLE>`. `ARRAY<INT>` with `ARRAY<STRING>` and `ARRAY<ARRAY<INT>>`
with `ARRAY<STRING>` both raise `BINARY_ARRAY_DIFF_TYPES`.
- Result `containsNull`: false only when both inputs have `containsNull =
false`.
- NaN: all NaN bit patterns are one value on every version.
- Signed zero in flat float arrays: one value with the result `+0.0` in
4.2.0. Both kept, with their signs, before 4.2 (see below).
- Nested floats (inside `STRUCT`/`ARRAY` elements) compare with
`SQLOrderingUtil`: `-0.0 = 0.0` and NaN = NaN on every version. Before 4.2 the
first one seen is kept as is.
- `BINARY`, `ARRAY` and `STRUCT` elements compare by value, not by reference.
- Collated strings (4.0+): `array_union(array('aaa' collate utf8_lcase),
array('AAA' collate utf8_lcase))` must dedupe under the collation.
- Literal and column for each argument. Sliced list arrays.
- A result longer than `MAX_ROUNDED_ARRAY_LENGTH` raises
`COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` (4.0+). In 3.x the error is
`_LEGACY_ERROR_TEMP_2162`.
### Differences across Spark versions
- Signed zero in flat float arrays: in 3.4, merging depends on hash
collisions. From 3.5.x to 4.1.3 the zeros stay distinct (`OpenHashSet` uses
`equals`, SPARK-45599). In 4.2.0 `NormalizeFloatingNumbers` normalizes both
arguments (SPARK-54918), so `-0.0` becomes `0.0` and NaN becomes the canonical
NaN at any depth. Comet's docs report that SPARK-59602 brings evaluation-time
normalization to 4.0.5, 4.1.4 and 4.2.1 (not in the source trees checked).
- 4.0: the size-limit error was renamed (`_LEGACY_ERROR_TEMP_2162` became
`COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`), and collation-aware comparison was
added. Otherwise none found.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`,
`ArrayUnion` (with `trait ArrayBinaryLike`, `trait ArraySetLike` and `trait
BinaryArrayExpressionWithImplicitCast`). The hash path uses `SQLOpenHashSet`
for atomic types with proper `equals`. `BINARY`, `STRUCT`, `ARRAY` and collated
strings use a nested loop with `ordering.equiv`.
- DataFusion core has `array_union`
(`datafusion/functions-nested/src/set_ops.rs`,
`generic_set_lists`/`generic_set_loop`). It already matches Spark on result
order (left then right, first occurrence), a single NULL element, and NULL list
input giving NULL. It turns `-0.0` into `+0.0` at any depth, which is 4.2.0
behavior. Differences:
- NaN payloads are kept and rows compare by bytes, so `NaN` and `-NaN`
stay distinct.
- `return_type` copies the first argument's type, so `containsNull` comes
from `array1` only. Spark uses `left || right`. Check that the output field's
nullability is correct when only `array2` has NULLs.
- Its signature (`Signature::arrays` with `FixedSizedListToList`) accepts
untyped `NULL` and may coerce element types that Spark rejects. The Spark
version needs its own `coerce_types` based on Spark's tightest-common-type rule.
- It cannot reproduce pre-4.2 signed-zero behavior.
- Reuse the float normalization in
`crates/spark/src/function/array/array_contains.rs` (`normalize_floats`,
related to apache/datafusion-spark#34) as a shared helper for all the array set
functions.
- Put the function in `crates/spark/src/function/array/`. Add
`crates/sqllogictest/test_files/array/array_union.slt` and document it with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]