sunchao opened a new pull request, #7724: URL: https://github.com/apache/datafusion-comet/pull/7724
## Which issue does this PR close? Follow-up to #5781 and #5580. This PR depends on #5867 and starts at its `11c725a` head. The [incremental diff](https://github.com/apache/datafusion-comet/compare/11c725a0a60e89b101db8478175e4e8b6271d167...365133e17de8e5b6e738b0504ef4729bab0d4696) contains the changes for review. ## Rationale for this change The native null guard can evaluate a stateful operand twice. Dispatching the whole expression, as in #5867, fixes the original reproducer. Compiling a subtree on its own can also change which rows evaluate an argument. For example, Spark evaluates both inputs of `arrays_zip` inside `collect_list` through `eval`. Generated code skips the second input when the first is null, which can suppress an ANSI error. Conversely, compiling a map lookup by itself can move a repeated division ahead of a null check and introduce an ANSI error. ## What changes are included in this PR? - Keep `size` native when legacy null sizing or a non-nullable child lets it evaluate the operand once. For guards that duplicate operands, check the full subtree for nondeterminism, including the receiver of Spark 4's `make_valid_utf8`. - Track interpreted evaluation through imperative aggregate inputs and expressions that use `CodegenFallback`. Treat generator inputs conservatively because a generator may run outside whole-stage codegen. Decline dispatch for `arrays_zip` and Spark 3.x `array_append` when a NULL can change whether a later computed argument runs. Ordinary generated projections keep dispatching. - Compare the subexpressions shared by the operator and the standalone kernel in both directions. Keep the lookup in Spark when that changes which rows evaluate its key. - Keep the operator in Spark when separate kernels could split a user function's state, or when Spark would generate an undeclared null flag for an array `element_at` inside the kernel. ## How are these changes tested? SQL fixtures compare the affected queries with Spark and check native execution, dispatch, fallback reasons, and ANSI errors. A Scala test projects two calls to the same counting UDF inside separate `arrays_zip` expressions. Formatted the changed Scala files with scalafmt 3.6.1 and ran `git diff --check`. Runtime validation is delegated to hosted CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
