andygrove opened a new pull request, #331:
URL: https://github.com/apache/datafusion-spark/pull/331
Closes #140.
## Summary
Adds Spark-compatible `array_max` and `array_min`, ported from Comet's
`native/spark-expr/src/array_funcs/array_extrema.rs`.
- NULL elements are skipped. A NULL array, an empty array, or an array of
only NULL
elements returns NULL.
- Elements are compared with Spark's ordering:
- NaN is greater than every other value, including infinity, and all NaNs
are equal.
- `-0.0` equals `0.0`. On a tie the first element wins and is returned
unchanged, so
`array_max(array(-0.0, 0.0))` is `-0.0` and `array_max(array(0.0,
-0.0))` is `0.0`.
The NaN payload bits are kept as well.
- These float rules also apply inside nested arrays and structs.
- Strings and binary compare bytewise.
- Arrays compare element by element and then by length. Structs compare
field by field.
A NULL nested inside an array or struct sorts first.
- Supported element types: integers, floats, decimals, booleans, strings,
binary, dates,
timestamps, and nested arrays and structs of these. Any other type that
arrow's
`make_comparator` can order is accepted too; interval elements are
accepted but not tested
here. `List` and `LargeList` are
supported, and `FixedSizeList` is coerced to `List`.
- An untyped `NULL` argument and `array()` are accepted and return a NULL of
the NULL type,
as in Spark (`void`).
- Planning errors use Spark's error classes:
- `INVALID_ORDERING_TYPE` for map elements at any depth.
- `UNEXPECTED_INPUT_TYPE` for an argument that is not an array.
- How it works: the element values are canonicalized with the existing
`normalize_array`
helper (NaN to the canonical NaN, `-0.0` to `0.0`, at any depth) and
compared with arrow's
`make_comparator`, with nulls first. The winner is then taken from the
original,
un-normalized values. `invoke_with_args` does the work itself and does not
depend on
`simplify`.
- `contains_map` in `array_position.rs` is now `pub(super)` so that both
functions can use it.
- Both functions are documented with `#[user_doc]`.
## Differences from Comet's version
- Comet wraps DataFusion's built-in `array_max`/`array_min`: it reuses their
signature and
return type, and hands non-float, non-nested element types to the
built-in. This version
has its own signature and `coerce_types`, and one code path for all
element types.
- Comet has a dedicated float scan and a hand-written `take` for nested
values, which keeps
child buffer capacity bounded when a large list loses. This version uses
`normalize_array`
plus `make_comparator` and arrow's `take`. It is simpler, but it may be
slower on long float
lists and can over-reserve child capacity for nested results. Both produce
the same values.
- Comet supports only 32-bit `List`. This version also supports `LargeList`,
and coerces
`FixedSizeList` to `List`.
- Comet does not check types. This version rejects map elements and
non-array arguments with
Spark's error classes.
## Testing
- New unit tests in `array_extrema.rs`, adapted from Comet's tests:
- first-winner float semantics for f32 and f64: NaN payloads, signed
zeros, infinities,
long lists
- scalar and sliced input
- strings and binary
- `LargeList`
- nested lists and structs
- NULL-typed input
- type rejection
- New `array/array_extrema.slt` (34 records). All 34 match Spark when run
with the PySpark
SLT runner on `pyspark==4.2.0`. I also checked the following directly in
PySpark 4.2.0:
- error messages for maps, non-arrays and the wrong number of arguments
- the `void` result type for `NULL` and `array()`
- that the zero sign is kept
- The sign of `-0.0` is checked with `CAST(... AS STRING)`, because the test
runner prints both
zeros as `0`. `atan2` cannot be used for this: Spark adds `0.0` to its
arguments, which turns
`-0.0` into `0.0`.
- Not covered in the `.slt` file, because of literal typing (#97):
- `array_max(array(1.0, 2))`: Spark types the array as `DECIMAL(11,1)` and
returns `2.0`,
while DataFusion returns a `DOUBLE`.
- Not tested:
- Interval elements. Spark orders `DAY TO SECOND` intervals, and PySpark
cannot collect
`YEAR TO MONTH` intervals.
- The `ArrayMax`/`ArrayMin` definitions in the Spark 3.5.8 and 4.1.1 sources
have the same
logic. I did not do a full multi-version audit.
- Commands, all passing:
- `cargo fmt --all`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-spark`: 345 passed
- `cargo test -p datafusion-spark-sqllogictest`: 250 of 250 files passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]