andygrove opened a new pull request, #7648:
URL: https://github.com/apache/datafusion-comet/pull/7648

   ## Which issue does this PR close?
   
   Backport of #7326 to `branch-1.2`. Closes #6823 on this branch.
   
   ## Rationale for this change
   
   Spark evaluates `AND`, `OR` and the candidates of `IN` lazily: a later 
operand runs only for the rows that the earlier ones have not decided. Comet 
evaluates every operand over the whole batch, so under ANSI mode an operand 
that can raise throws on a row Spark never evaluates, for example `v != 0 AND 
100 / v > 1`, `v = 0 OR 100 / v > 1` and `v IN (0, 100 / v)` raise 
`DIVIDE_BY_ZERO` in Comet and return rows in Spark. `branch-1.2` has the same 
eager native evaluation. See #7326 and #6823 for the full analysis.
   
   ## What changes are included in this PR?
   
   Cherry-pick of 7deeba21a4 (`git cherry-pick -x`) with no conflicts and no 
adaptations. It adds `LazyLogical` and `lazy_in_list` in `native/spark-expr`, 
the small changes to `case_when.rs`, `null_short_circuit.rs` and 
`nested_comparison.rs` that they share, the `lazy_logical` benchmark, and the 
`expressions/conditional/logical_lazy_ansi.sql` file test.
   
   Other release branches: `branch-1.2` is the newest release branch, so there 
is no newer one to port to. Whether `branch-1.1` needs a backport is not part 
of this PR.
   
   ## How are these changes tested?
   
   On this branch, with JDK 17 and the default Spark profile (4.1): `cargo 
check --all-targets` and `cargo clippy --all-targets -D warnings` pass for 
`datafusion-comet-spark-expr`, `cargo fmt --check` is clean, and the 
`datafusion-comet-spark-expr` lib tests pass (1310 tests, 0 failures, including 
the 11 new `lazy_logical` tests). `CometSqlFileTestSuite` passes for 
`expressions/conditional/logical_lazy_ansi.sql` against a native library built 
from this branch. Against a native library built from `branch-1.2` without the 
fix, the same file test fails with `DIVIDE_BY_ZERO`, so the branch has the bug 
and the test catches it. I ran the default Spark profile only.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to