andygrove opened a new pull request, #7648: URL: https://github.com/apache/datafusion-comet/pull/7648
## Which issue does this PR close? Backport of #7326 to `branch-1.2`. Closes #6823 on this branch. ## Rationale for this change Spark evaluates `AND`, `OR` and the candidates of `IN` lazily: a later operand runs only for the rows that the earlier ones have not decided. Comet evaluates every operand over the whole batch, so under ANSI mode an operand that can raise throws on a row Spark never evaluates, for example `v != 0 AND 100 / v > 1`, `v = 0 OR 100 / v > 1` and `v IN (0, 100 / v)` raise `DIVIDE_BY_ZERO` in Comet and return rows in Spark. `branch-1.2` has the same eager native evaluation. See #7326 and #6823 for the full analysis. ## What changes are included in this PR? Cherry-pick of 7deeba21a4 (`git cherry-pick -x`) with no conflicts and no adaptations. It adds `LazyLogical` and `lazy_in_list` in `native/spark-expr`, the small changes to `case_when.rs`, `null_short_circuit.rs` and `nested_comparison.rs` that they share, the `lazy_logical` benchmark, and the `expressions/conditional/logical_lazy_ansi.sql` file test. Other release branches: `branch-1.2` is the newest release branch, so there is no newer one to port to. Whether `branch-1.1` needs a backport is not part of this PR. ## How are these changes tested? On this branch, with JDK 17 and the default Spark profile (4.1): `cargo check --all-targets` and `cargo clippy --all-targets -D warnings` pass for `datafusion-comet-spark-expr`, `cargo fmt --check` is clean, and the `datafusion-comet-spark-expr` lib tests pass (1310 tests, 0 failures, including the 11 new `lazy_logical` tests). `CometSqlFileTestSuite` passes for `expressions/conditional/logical_lazy_ansi.sql` against a native library built from this branch. Against a native library built from `branch-1.2` without the fix, the same file test fails with `DIVIDE_BY_ZERO`, so the branch has the bug and the test catches it. I ran the default Spark profile only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
