andygrove opened a new pull request, #330:
URL: https://github.com/apache/datafusion-spark/pull/330
Closes #142.
## Summary
Adds a Spark-compatible `array_remove(array, element)` (`SparkArrayRemove`),
ported from DataFusion Comet's `spark_array_remove`, and registers it as
`array_remove`. It replaces the DataFusion built-in of the same name when the
Spark functions are registered.
- It removes every element equal to `element`. NULL elements in the array
are always kept.
- It returns NULL when the array or the element is NULL. The DataFusion
built-in instead removes the NULL elements when the element is NULL.
- Equality follows Spark's ordering-based `genEqual`/`ordering.equiv` at any
depth of arrays and structs: `NaN == NaN`, `-0.0 == 0.0`, and NULLs nested
inside the compared values are equal. The comparison runs on float-normalized
copies (`float_normalize::normalize_array`, as in `array_position`). The kept
elements come from the original values, so `array_remove(array(-0.0, 1.0),
1.0)` returns `[-0.0]`, as in Spark.
- It supports `List` and `LargeList`, sliced inputs, and both scalar and
column arguments. A constant non-nested element is compared with arrow's
vectorized `eq` kernel. Nested elements and column elements use
`make_comparator`. Elements of NULL rows are not carried into the result's
values.
- `invoke_with_args` does the work itself and does not rely on `simplify`.
## Differences from Comet's version
- Comet's `spark_array_remove` is only routed for float-containing element
types: other types use DataFusion's `array_remove_all`. It is registered under
a different name and uses `Signature::any(2)` with no coercion, relying on
Spark to have cast the arguments already. This version handles every element
type and does Spark's type coercion itself (see below).
- Comet supports only 32-bit `List`. This version also supports `LargeList`,
and accepts `FixedSizeList` by coercing it to `List`.
- Comet has hand-written float kernels (a one-pass constant path that reads
the mask a word at a time, plus a per-row float path) and depends on its
`float_semantics`/`nested_comparison` helpers. This version reuses this crate's
existing `normalize_array` plus arrow's `eq`/`make_comparator`, which gives the
same results with less code. The float kernels are not ported.
- Comet keeps the elements of NULL rows in the child values (hidden by the
list's null buffer). This version drops them.
- Comet's unit tests are adapted rather than copied verbatim, because they
depend on Comet-only test helpers: float edge values as a column and as a
constant, nulls, structs with float fields, and scalar float32. Tests for
sliced lists, `LargeList`, and the coercion rules are added.
## Type rules
These are Spark's `ArrayRemove.inputTypes` / `checkInputDataTypes`. The
tightest-common-type logic is shared with `array_position`:
`tightest_common_type` and `contains_map` are now `pub(super)`.
- The array element and the element are cast to `findTightestCommonType`,
and the result is an array of that type. For example `array_remove(array(1, 2),
1.5D)` is `array<double>` and `array_remove(array(1, 2), CAST(1 AS
DECIMAL(12,2)))` is `array<decimal(12,2)>`.
- Numbers widen.
- An integral type widens to a decimal only if the decimal can hold all
its values.
- DATE widens to TIMESTAMP.
- Pairs with no common type are an error: string and number, two different
decimals, decimal and float, and structs whose field names or types differ.
- All of these raise `[DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES]`.
- A first argument that is not an array raises `ARRAY_FUNCTION_DIFF_TYPES`,
as in Spark. This includes an untyped `NULL`.
- A NULL-typed common element type is accepted, unlike `array_position`:
`array_remove(array(), NULL)` returns NULL with type `array<void>`.
- Map elements, at any depth, raise
`[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE]`.
- Struct field types are not widened, which matches Spark with ANSI mode
off. With ANSI on (Spark 4's default), Spark widens them:
`array_remove(array(named_struct('a', 1)), named_struct('a', 1L))` returns `[]`
in Spark with ANSI on, and is an error here. This is the same choice
`array_position` makes.
## Testing
- Unit tests in `array_remove.rs` cover:
- float edge values (both zeros, two NaN bit patterns, ±Inf, NULL) as a
column and as a constant, checking that the kept elements keep their bits
- NULL rows, NULL elements and a NULL element
- sliced lists
- `LargeList` scalars
- struct elements with float fields
- the coercion rules
- New `crates/sqllogictest/test_files/array/array_remove.slt` covers:
- basics, NULL semantics, and error cases
- float equality, including nested and in structs
- nested NULL equality
- type coercion
- column inputs, including an `OFFSET`-sliced batch
- `LargeList`
- Verified with pyspark 4.2.0:
- `pyspark_slt_runner.py` on `array_remove.slt`: 71 records pass with no
differences. The 6 records that use `arrow_typeof` are skipped, and their
result types (bigint, bigint, int, double, decimal(12,2)) were checked by hand
with `typeof`/schema in a local SparkSession.
- About 50 ad-hoc queries using Spark's `array()` were also checked by
hand, including the error classes for an untyped NULL array, decimal and string
mismatches, and maps.
- Commands, all passing:
- `cargo fmt --all`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-spark` (341 unit tests + doc tests)
- `cargo test -p datafusion-spark-sqllogictest` (250 of 250 files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]