andygrove commented on issue #6006: URL: https://github.com/apache/datafusion-comet/issues/6006#issuecomment-6507214136
I reproduced the per-row `AND` case from the audit list, and it is a real mismatch. `OR` and `IN` behave the same way. `CASE WHEN`, `IF`, `COALESCE` and `nvl2` already match Spark, so they can serve as regression controls. Spark evaluates `AND`, `OR` and `IN` lazily, so a guard operand keeps the other operand from running on rows it rules out. Comet's native evaluation runs every operand over the whole batch, so under ANSI mode an operation that can raise throws on a row Spark never evaluates. Setup: Spark 4.1, a release native build from `apache/main` at 7f73d86fb8, ANSI on, no operator involved. The `Spark` and `Comet` columns show what each engine does for the query. ```sql CREATE TABLE sc USING parquet AS SELECT cast(id % 4 AS int) AS v FROM range(1000); ``` | Expression | Spark | Comet | | --- | --- | --- | | `v != 0 AND 100 / v > 1` | returns rows | `DIVIDE_BY_ZERO` | | `v = 0 OR 100 / v > 1` | returns rows | `DIVIDE_BY_ZERO` | | `v IN (0, 100 / v)` | returns rows | `DIVIDE_BY_ZERO` | | `CASE WHEN v = 0 THEN false ELSE 100 / v > 1 END` | returns rows | returns rows | | `if(v = 0, false, 100 / v > 1)` | returns rows | returns rows | | `coalesce(v, 100 / v) > 1` | returns rows | returns rows | Two notes for the "Work to track" list: - `IN` is not named there. Its list elements are lazy in Spark, so it belongs in the per-row audit with `AND` and `OR`. - This affects any lambda body that Comet starts to evaluate natively. I am adding a native path for `filter(arr, x -> ...)`. Until the per-row case is resolved, that path keeps a body that combines `AND`, `OR` or `IN` with an operation that can raise on the JVM dispatcher under ANSI mode. The dispatcher runs Spark's own code, so it matches Spark. Without that guard, `filter(arr, x -> x != 0 AND 100 / x > 1)` would raise, where it returned the right rows through the dispatcher before. On the approach: conditional evaluation in the native `AND`, `OR` and `IN` kernels, which means evaluating the right-hand operand only for rows the left-hand side has not already decided, would fix this without a fallback. That fits the "conditional evaluation in native kernels" option already listed. A scoped fallback would send every ANSI query that has an `AND` over a division or an addition back to Spark, which is a large footprint, so I would measure that before choosing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
