sunchao opened a new pull request, #7723:
URL: https://github.com/apache/datafusion-comet/pull/7723

   ## Which issue does this PR close?
   
   Closes #7480.
   
   ## Rationale for this change
   
   Both queries in #7480 encounter a Parquet row with `x = 0` and `y = NULL`. 
With ANSI enabled, Spark raises `DIVIDE_BY_ZERO` on that row for two different 
reasons.
   
   - In `WHERE y IS NOT NULL AND abs(10 DIV x) > 0`, Spark's whole-stage filter 
checks each column for NULL when a predicate first needs it. The comparison 
needs only `x`, so Spark divides before checking `y`. Comet checks `y` first 
and discards the row. [Spark filter code 
generation](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala#L140-L254)
   - In `SELECT x <> 0 AND abs(10 DIV x) + abs(10 DIV x) > 1`, Spark's 
generated projection computes the repeated `abs(10 DIV x)` before testing `x <> 
0`. Comet skips the whole right operand and returns `false`. [Whole-stage 
projection](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala#L68-L89),
 [standalone generated 
projection](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala#L286-L315)
   
   ## What changes are included in this PR?
   
   - When serializing an eligible whole-stage filter, place each required 
column null check at its first use and append the remaining checks. Treat a 
NULL conjunct as a rejection before proceeding to the next conjunct.
   - When serializing a generated projection, use Spark's 
`EquivalentExpressions` analysis to find shared terms containing ANSI division 
under the right operand of `AND` or `OR`. Compute them in an intermediate 
DataFusion projection and bind their uses to its output columns. This lets the 
rest of the operand short-circuit, including a separate division that occurs 
only once. The two projections share one Spark operator ID for metrics.
   - Restrict these rewrites to ANSI `/`, `DIV` and `%` in expression trees 
with compatible operand evaluation. For example, moving `y + 10 DIV x` earlier 
could expose a division that Spark skips when `y` is NULL. Nested operand 
evaluation is covered by #7456 and #7468. The compatibility guide gives 
examples of remaining differences for computed dividends and the per-operator 
workarounds.
   
   Both reported queries stay in the Comet pipeline. The filter change applies 
to rows delivered by the scan. Both changes use the expression's own ANSI mode 
and respect Spark's code generation settings. The projection also respects the 
subexpression elimination setting.
   
   ## How are these changes tested?
   
   `logical_ansi_division.sql` adds 12 queries over a single Parquet file. The 
two issue reproductions require a Comet plan and compare the exception type, 
error class and SQLSTATE with Spark. Controls cover a comparison that needs 
both columns, a NULL conjunct, nullable arithmetic, sharing across projection 
outputs, an unshared failing sibling, a conditional division, disabled 
subexpression elimination, standalone code generation and interpreted execution.
   
   Ran Spotless on the changed Scala files and `git diff --check`. Runtime 
validation is delegated to hosted CI.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to