comphead opened a new pull request, #7538: URL: https://github.com/apache/datafusion-comet/pull/7538
## Which issue does this PR close? Part of #6485, which stays open. ## Rationale for this change When no branch of a `CASE WHEN` can fail, Comet evaluates every branch over the whole batch and picks each row's value. Otherwise it filters the batch for each branch and merges the results, which is much slower. `is_infallible` did not list `GetStructField`, so a `CASE` over struct fields always took the slow path. `AND`, `OR` and `IN` use the same check to decide whether to mask their later operands, so filters on struct fields paid for it too. A local benchmark (not part of this PR), 1M rows, Spark 4.1, one task, best time: | Query over the fields of struct `s` | Before | After | | --- | --- | --- | | `CASE WHEN s.x > 500 THEN s.y * 2 ELSE s.z END` | 22.4 ms | 16.4 ms | | `CASE` with 3 branches | 27.6 ms | 16.5 ms | | `SELECT s.x * s.y + s.z, s.w - s.x ... WHERE s.x > 300 AND s.y < 700` | 35.3 ms | 18.0 ms | The same queries over top-level columns take 15 to 17 ms. For the first query, the projection's own time drops from 10.4 ms to 1.0 ms. ## What changes are included in this PR? `infallible` in `case_when.rs` accepts `GetStructField` when its child is infallible. A struct field cannot fail and does not depend on which rows it sees. ## How are these changes tested? `case_when_struct_field.sql` runs `CASE WHEN`, `IF`, `COALESCE`, `AND`, `OR` and `IN` over boolean, integer, double, decimal, date, timestamp, string and binary struct fields, with ANSI mode off and on. The rows include a NULL struct, a NULL inner struct, NULL fields, NaN, ±0.0, ±Infinity, and integer and decimal bounds. Two queries divide by a field that the `WHEN` or the left operand of `AND` rules out as zero, which must still run only for the rows that reach it. The results are the same with and without this change, so the test checks the eager path's results over struct fields rather than which path ran. It passes locally on Spark 4.1. I did not run other suites. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
