dwsmith1983 opened a new pull request, #7670: URL: https://github.com/apache/datafusion-comet/pull/7670
## Which issue does this PR close? Part of #5781 on `branch-1.2`. #5867 fixes it on `main` and closes the issue; it is approved but not merged yet, so this should merge after it. ## Rationale for this change `size`, `arrays_zip` and `map_from_arrays` evaluate a nondeterministic child twice in 1.2.0: once as the value and again inside the native `NULL` guard. A stateful child such as `IF(monotonically_increasing_id() % 2 = 0, array(1), NULL)` then returns wrong results. #5867 routes these cases through the JVM codegen dispatcher, but it also moves map lookups with float, collated and complex keys to the dispatcher (#5580), an enhancement that #5875 measured at 9 to 11 percent slower for lookup-only DOUBLE-key projections. That does not belong in a patch release, so this change falls back to Spark instead, as `branch-1.2` already does for `array_append` and ANSI `element_at`. ## What changes are included in this PR? - `size` and `arrays_zip` report a nondeterministic child as unsupported, which falls back to Spark. - `map_from_arrays` declines a nondeterministic child when it is converted, since reporting it unsupported would send it to the codegen dispatcher. This also covers the `allowIncompatible` route for float keys. - The fallback reason uses the same text as #5867. - The compatibility pages do not list the `map_from_arrays` fallback, because the generator would describe that function's unsupported reasons as running through the codegen dispatcher. - The compatibility pages list the new reason under `ArraysZip` and a new `Size` section. The pages for Spark 3.5, 4.0, 4.1 and 4.2 were regenerated; the Spark 3.4 page got the same lines by hand. A non-nullable nondeterministic child, such as `size(array(monotonically_increasing_id()))`, also falls back now. 1.2.0 ran it natively and correctly, since the guard passes every row. ## How are these changes tested? Three SQL file tests, adapted from #5867, check each function against Spark: a nondeterministic nullable child falls back, a non-nullable one falls back, and a deterministic or NULL input still runs natively. For `map_from_arrays`, DOUBLE keys go to the dispatcher by default and fall back with `allowIncompatible=true`. On `branch-1.2` without the fix, all four nondeterministic cases and the `allowIncompatible` case return wrong results. With it, the tests pass on Spark 3.5, 4.0, 4.1 and 4.2, and the existing `size`, `arrays_zip` and `map_from_arrays` tests and the array and map expression suites still pass on 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
