andygrove opened a new issue, #7535: URL: https://github.com/apache/datafusion-comet/issues/7535
## What is the problem the feature request solves? #7322 runs `filter` and `exists` lambdas natively when the body is made of natively supported expressions. Two of its conditions depend on whether an expression in the body can raise, which `NativeLambdaSerde.mayRaise` in `serde/arrays.scala` decides. An `exists` body runs natively only if no expression in it can raise, because Spark stops at the first element that matches while `array_any_match` evaluates every element. A `filter` body that uses `AND`, `OR` or `IN` runs natively only under the same condition, because the native operators evaluate every operand (the per-row case of #6006). `mayRaise` counts every `Cast` that carries the ANSI eval mode as able to raise, including a widening one that cannot fail. Spark's analyzer adds such a cast when the lambda compares the element with a column of another numeric type, and with ANSI mode on, the default since Spark 4.0, the cast carries the ANSI mode. So these lambdas stay on the JVM codegen dispatcher: | Lambda, ANSI mode on | Spark plans | Comet | | --- | --- | --- | | `exists(ints, x -> x > big)`, `big` is a BIGINT column | `CAST(x AS BIGINT) > big` | dispatcher | | `exists(dbls, x -> x > threshold)`, `threshold` is an INT column | `x > CAST(threshold AS DOUBLE)` | dispatcher | | `filter(ints, x -> x > 0 AND x < big)` | `x > 0 AND CAST(x AS BIGINT) < big` | dispatcher | With ANSI mode off all three run natively. A comparison with a literal is not affected: Spark's `UnwrapCastInBinaryComparison` removes the cast from `x > 90L`, `x > 1.5` and `x IN (1L, 95L)`, so those run natively in both modes. I checked each case on Spark 3.5 with `expect_native` and `expect_dispatch` queries at #7322's head. ## Describe the potential solution Count a cast as unable to raise, whatever its eval mode, when it cannot fail for any value. The native side already has this list: `Cast::is_infallible` in `conversion_funcs/cast.rs` accepts a cast to the child's own type, a cast from NULL, and the numeric widenings (an integer to a wider integer, a float or a double, and a float to a double), and `case_when.rs` uses it to answer the same question. Matching that list in `mayRaise` keeps the JVM and native checks in agreement. `CometCast.keyCastCannotFail`, behind `canProduceNullMapKey`, accepts more, such as an integer to a decimal with enough integral digits and a cast to a string. Those could follow once the native kernels for them are confirmed not to raise either. The three queries above would move to `expect_native(...)` in an ANSI fixture, next to a narrowing cast such as `exists(longs, x -> CAST(x AS INT) > 1)`, which must stay on the dispatcher. Part of the effort tracked in #7300. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
