sunchao opened a new pull request, #7717:
URL: https://github.com/apache/datafusion-comet/pull/7717

   ## Which issue does this PR close?
   
   Closes #6614. Part of #6385.
   
   ## Rationale for this change
   
   Spark uses the same equality rule for all three functions. Signed zeros 
compare equal, and any NaN matches any other NaN, including inside arrays and 
structs. Comet's native `array_contains` and `array_remove` each implemented 
scalar float comparison and per-row search. `array_position` expanded a scalar 
search value to a column and normalized both nested operands before comparing 
them.
   
   ## What changes are included in this PR?
   
   - Share row traversal and element comparison across the three Comet kernels. 
A scalar search value occupies one slot. For a column, each list uses the value 
at its row index. Nested elements use the existing recursive Spark equality 
predicate directly on the input buffers.
   - `array_contains` and `array_position` share a search that stops at the 
first match. On a miss, a null element makes `array_contains` return null, 
while `array_position` returns zero. `array_remove` visits all elements and 
copies the survivors with their original bits and nulls.
   - For a scalar float, `array_contains` and `array_remove` use the existing 
vectorized Spark comparison kernel to build their match and keep masks over the 
visible list elements. `array_position` uses the early-exit search for scalar 
inputs too, so a match at the start of a long list ends the search there.
   
   This applies to the current native paths. In particular, nested floats in 
`array_contains` go through Spark's codegen dispatcher by default, following 
the [benchmark in 
#6599](https://github.com/apache/datafusion-comet/pull/6599#issuecomment-5998084734).
   
   ## How are these changes tested?
   
   The SQL fixture compares native `array_position` with Spark for scalar and 
per-row values, including signed zeros, duplicates, nulls, empty lists and 
nested arrays. Negating a stored NaN after the Parquet read creates an 
opposite-sign NaN. Searching `[NULL, -NaN, NaN]` for NaN must return 2, and the 
nested case exercises the same rule with a null inside each child array.
   
   A Rust test checks the first match for a scalar NaN in a sliced `LargeList` 
with nonzero offsets. Existing contains and remove tests cover nulls, NaN 
payloads, bitmap word boundaries and the bits of surviving elements.
   
   Checked `rustfmt --check` and `git diff --check` locally. The Rust and Spark 
SQL suites are delegated to hosted CI. Runtime performance is unmeasured.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to