kazantsev-maksim commented on PR #4744:
URL:
https://github.com/apache/datafusion-comet/pull/4744#issuecomment-5983044495
All tests are now passing across both native and fallback suites! @sunchao
### Summary of Final Updates:
1. **Stateful Expressions & Null-Operand Evaluation:**
- Addressed the ordinary equality regression with `x =
monotonically_increasing_id()` on `[NULL, 0]`. Because DataFusion's vectorized
batch evaluation does not replicate Spark's per-element null-operand
short-circuiting, stateful expressions with mutable counters
(`MonotonicallyIncreasingID`, `Rand`, `Randn`) cleanly degrade to JVM codegen
dispatch.
- Added the corresponding regression test `filter(a, x -> x =
monotonically_increasing_id())` on `[NULL, 0L]`, verifying that it produces
`[0]`.
2. **Clean Test Suite Separation:**
- **`array_filter_ansi.sql` (Native with `codegen.enabled=false`):**
strictly asserts Comet native execution for `EmptyBatchGuardExpr`,
`ShortCircuitBinaryExpr` (3VL, masked selection), and empty/null array boundary
cases without fallback.
- **`array_filter_fallback.sql` (Fallback with `codegen.enabled=true`):**
asserts correct JVM codegen dispatch and exact Spark results for stateful
expressions (`monotonically_increasing_id`, `rand`), conditionals (`CASE WHEN`,
`IF`, `COALESCE`), and nested complex captures (`ArrayType`, `MapType`,
`StructType`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
