andygrove opened a new pull request, #63: URL: https://github.com/apache/datafusion-spark/pull/63
## Summary Ports `array_position` from Comet (`native/spark-expr/src/array_funcs/array_position.rs`). - Returns the 1-based position of the first match, `0` when not found, and `NULL` when the array or the element is `NULL`. DataFusion's built-in returns `NULL` when the element is not found. - `NULL` elements inside the array never match but still count toward the position. - Floats compare as Spark does at any depth (inside arrays and structs): `NaN = NaN` and `-0.0 = 0.0`. `NULL` nested inside compared values equals `NULL`. - Supports `List` and `LargeList`. Floats inside maps or fixed-size lists return a "not supported" error (Spark rejects maps in `array_position`). Differences from the Comet version: Comet's `float_semantics` module is not available here, so the comparison is a small recursive equality; the per-type fast paths are replaced by one generic loop. Adds `datafusion-macros` and `datafusion-doc` dependencies so the function can use `#[user_doc]`. ## Testing `array/array_position.slt` (36 queries). Expected values were generated with `pyspark==4.2.0`, except the column-input table test, which is my reading of Spark's semantics. - `cargo clippy --all-targets --all-features -- -D warnings`: clean - `cargo test -p datafusion-spark`: pass - `cargo test -p datafusion-spark-sqllogictest`: 246/246 files pass Not done: the full Spark source audit across 3.5/4.0/4.1. Note: `cargo fmt --all -- --check` currently fails on `main` for ~98 existing files because `rustfmt.toml` (`max_width = 90`) is not on `main`. The new file is formatted with `max_width = 90` to match the existing code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
