grorge123 opened a new pull request, #6743: URL: https://github.com/apache/datafusion-comet/pull/6743
## Which issue does this PR close? Closes #6742. ## Rationale for this change When one argument of `arrays_overlap` is a scalar and the other is an array column, the native kernel turns the scalar into a one-row list while the column keeps its full length, and returns row 0's result as a scalar. The first row's answer is then used for the whole batch, or the kernel panics when the constant is the second argument. Split out of #5526. ## What changes are included in this PR? `SparkArraysOverlap` keeps a scalar argument as a one-row list that every output row reads, and returns an array result when the other argument is an array. It does not repeat the scalar to the batch length, because repeating a long list could overflow the list's 32-bit offsets. The existing NULL-scalar short-circuit is unchanged, and two scalar arguments still produce a scalar result. ## How are these changes tested? - A query at the end of `array/arrays_overlap.sql` compares a column-built array against a constant array in both argument orders, over four rows in one batch. On `main` the query panics, because of the constant-second order; the constant-first order on its own returns a wrong answer. It passes on the Spark 4.1, 3.5 and 3.4 profiles. - Rust unit tests call `invoke_with_args` with a scalar on either side, for flat and struct elements. They also cover two scalars, and a 262,144-element constant beside 8,192 rows, which panics with an offset overflow if the scalar is repeated instead. Assisted-by: Claude Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
