sunchao opened a new pull request, #7717: URL: https://github.com/apache/datafusion-comet/pull/7717
## Which issue does this PR close? Closes #6614. Part of #6385. ## Rationale for this change Spark uses the same equality rule for all three functions. Signed zeros compare equal, and any NaN matches any other NaN, including inside arrays and structs. Comet's native `array_contains` and `array_remove` each implemented scalar float comparison and per-row search. `array_position` expanded a scalar search value to a column and normalized both nested operands before comparing them. ## What changes are included in this PR? - Share row traversal and element comparison across the three Comet kernels. A scalar search value occupies one slot. For a column, each list uses the value at its row index. Nested elements use the existing recursive Spark equality predicate directly on the input buffers. - `array_contains` and `array_position` share a search that stops at the first match. On a miss, a null element makes `array_contains` return null, while `array_position` returns zero. `array_remove` visits all elements and copies the survivors with their original bits and nulls. - For a scalar float, `array_contains` and `array_remove` use the existing vectorized Spark comparison kernel to build their match and keep masks over the visible list elements. `array_position` uses the early-exit search for scalar inputs too, so a match at the start of a long list ends the search there. This applies to the current native paths. In particular, nested floats in `array_contains` go through Spark's codegen dispatcher by default, following the [benchmark in #6599](https://github.com/apache/datafusion-comet/pull/6599#issuecomment-5998084734). ## How are these changes tested? The SQL fixture compares native `array_position` with Spark for scalar and per-row values, including signed zeros, duplicates, nulls, empty lists and nested arrays. Negating a stored NaN after the Parquet read creates an opposite-sign NaN. Searching `[NULL, -NaN, NaN]` for NaN must return 2, and the nested case exercises the same rule with a null inside each child array. A Rust test checks the first match for a scalar NaN in a sliced `LargeList` with nonzero offsets. Existing contains and remove tests cover nulls, NaN payloads, bitmap word boundaries and the bits of surviving elements. Checked `rustfmt --check` and `git diff --check` locally. The Rust and Spark SQL suites are delegated to hosted CI. Runtime performance is unmeasured. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
