dwsmith1983 opened a new pull request, #7670:
URL: https://github.com/apache/datafusion-comet/pull/7670

   ## Which issue does this PR close?
   
   Part of #5781 on `branch-1.2`. #5867 fixes it on `main` and closes the 
issue; it is approved but not merged yet, so this should merge after it.
   
   ## Rationale for this change
   
   `size`, `arrays_zip` and `map_from_arrays` evaluate a nondeterministic child 
twice in 1.2.0: once as the value and again inside the native `NULL` guard. A 
stateful child such as `IF(monotonically_increasing_id() % 2 = 0, array(1), 
NULL)` then returns wrong results.
   
   #5867 routes these cases through the JVM codegen dispatcher, but it also 
moves map lookups with float, collated and complex keys to the dispatcher 
(#5580), an enhancement that #5875 measured at 9 to 11 percent slower for 
lookup-only DOUBLE-key projections. That does not belong in a patch release, so 
this change falls back to Spark instead, as `branch-1.2` already does for 
`array_append` and ANSI `element_at`.
   
   ## What changes are included in this PR?
   
   - `size` and `arrays_zip` report a nondeterministic child as unsupported, 
which falls back to Spark.
   - `map_from_arrays` declines a nondeterministic child when it is converted, 
since reporting it unsupported would send it to the codegen dispatcher. This 
also covers the `allowIncompatible` route for float keys.
   - The fallback reason uses the same text as #5867.
   - The compatibility pages do not list the `map_from_arrays` fallback, 
because the generator would describe that function's unsupported reasons as 
running through the codegen dispatcher.
   - The compatibility pages list the new reason under `ArraysZip` and a new 
`Size` section. The pages for Spark 3.5, 4.0, 4.1 and 4.2 were regenerated; the 
Spark 3.4 page got the same lines by hand.
   
   A non-nullable nondeterministic child, such as 
`size(array(monotonically_increasing_id()))`, also falls back now. 1.2.0 ran it 
natively and correctly, since the guard passes every row.
   
   ## How are these changes tested?
   
   Three SQL file tests, adapted from #5867, check each function against Spark: 
a nondeterministic nullable child falls back, a non-nullable one falls back, 
and a deterministic or NULL input still runs natively. For `map_from_arrays`, 
DOUBLE keys go to the dispatcher by default and fall back with 
`allowIncompatible=true`. On `branch-1.2` without the fix, all four 
nondeterministic cases and the `allowIncompatible` case return wrong results. 
With it, the tests pass on Spark 3.5, 4.0, 4.1 and 4.2, and the existing 
`size`, `arrays_zip` and `map_from_arrays` tests and the array and map 
expression suites still pass on 4.1.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to