sunchao opened a new pull request, #7724:
URL: https://github.com/apache/datafusion-comet/pull/7724

   ## Which issue does this PR close?
   
   Follow-up to #5781 and #5580. This PR depends on #5867 and starts at its 
`11c725a` head. The [incremental 
diff](https://github.com/apache/datafusion-comet/compare/11c725a0a60e89b101db8478175e4e8b6271d167...365133e17de8e5b6e738b0504ef4729bab0d4696)
 contains the changes for review.
   
   ## Rationale for this change
   
   The native null guard can evaluate a stateful operand twice. Dispatching the 
whole expression, as in #5867, fixes the original reproducer. Compiling a 
subtree on its own can also change which rows evaluate an argument. For 
example, Spark evaluates both inputs of `arrays_zip` inside `collect_list` 
through `eval`. Generated code skips the second input when the first is null, 
which can suppress an ANSI error. Conversely, compiling a map lookup by itself 
can move a repeated division ahead of a null check and introduce an ANSI error.
   
   ## What changes are included in this PR?
   
   - Keep `size` native when legacy null sizing or a non-nullable child lets it 
evaluate the operand once. For guards that duplicate operands, check the full 
subtree for nondeterminism, including the receiver of Spark 4's 
`make_valid_utf8`.
   - Track interpreted evaluation through imperative aggregate inputs and 
expressions that use `CodegenFallback`. Treat generator inputs conservatively 
because a generator may run outside whole-stage codegen. Decline dispatch for 
`arrays_zip` and Spark 3.x `array_append` when a NULL can change whether a 
later computed argument runs. Ordinary generated projections keep dispatching.
   - Compare the subexpressions shared by the operator and the standalone 
kernel in both directions. Keep the lookup in Spark when that changes which 
rows evaluate its key.
   - Keep the operator in Spark when separate kernels could split a user 
function's state, or when Spark would generate an undeclared null flag for an 
array `element_at` inside the kernel.
   
   ## How are these changes tested?
   
   SQL fixtures compare the affected queries with Spark and check native 
execution, dispatch, fallback reasons, and ANSI errors. A Scala test projects 
two calls to the same counting UDF inside separate `arrays_zip` expressions. 
Formatted the changed Scala files with scalafmt 3.6.1 and ran `git diff 
--check`. Runtime validation is delegated to hosted CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to