sunchao opened a new pull request, #7723: URL: https://github.com/apache/datafusion-comet/pull/7723
## Which issue does this PR close? Closes #7480. ## Rationale for this change Both queries in #7480 encounter a Parquet row with `x = 0` and `y = NULL`. With ANSI enabled, Spark raises `DIVIDE_BY_ZERO` on that row for two different reasons. - In `WHERE y IS NOT NULL AND abs(10 DIV x) > 0`, Spark's whole-stage filter checks each column for NULL when a predicate first needs it. The comparison needs only `x`, so Spark divides before checking `y`. Comet checks `y` first and discards the row. [Spark filter code generation](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala#L140-L254) - In `SELECT x <> 0 AND abs(10 DIV x) + abs(10 DIV x) > 1`, Spark's generated projection computes the repeated `abs(10 DIV x)` before testing `x <> 0`. Comet skips the whole right operand and returns `false`. [Whole-stage projection](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala#L68-L89), [standalone generated projection](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala#L286-L315) ## What changes are included in this PR? - When serializing an eligible whole-stage filter, place each required column null check at its first use and append the remaining checks. Treat a NULL conjunct as a rejection before proceeding to the next conjunct. - When serializing a generated projection, use Spark's `EquivalentExpressions` analysis to find shared terms containing ANSI division under the right operand of `AND` or `OR`. Compute them in an intermediate DataFusion projection and bind their uses to its output columns. This lets the rest of the operand short-circuit, including a separate division that occurs only once. The two projections share one Spark operator ID for metrics. - Restrict these rewrites to ANSI `/`, `DIV` and `%` in expression trees with compatible operand evaluation. For example, moving `y + 10 DIV x` earlier could expose a division that Spark skips when `y` is NULL. Nested operand evaluation is covered by #7456 and #7468. The compatibility guide gives examples of remaining differences for computed dividends and the per-operator workarounds. Both reported queries stay in the Comet pipeline. The filter change applies to rows delivered by the scan. Both changes use the expression's own ANSI mode and respect Spark's code generation settings. The projection also respects the subexpression elimination setting. ## How are these changes tested? `logical_ansi_division.sql` adds 12 queries over a single Parquet file. The two issue reproductions require a Comet plan and compare the exception type, error class and SQLSTATE with Spark. Controls cover a comparison that needs both columns, a NULL conjunct, nullable arithmetic, sharing across projection outputs, an unshared failing sibling, a conditional division, disabled subexpression elimination, standalone code generation and interpreted execution. Ran Spotless on the changed Scala files and `git diff --check`. Runtime validation is delegated to hosted CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
