rich7420 opened a new pull request, #5737: URL: https://github.com/apache/datafusion-comet/pull/5737
## Which issue does this PR close? Closes #5676. ## Rationale for this change Plain `posexplode` places its array expression in both the positions expression and the values projection, evaluating it twice per batch. `posexplode_outer` already materializes its wrapped array once. For a computed array, the duplicate evaluation can repeat native work or a JVM codegen-dispatch call. ## What changes are included in this PR? Apply the outer wrapper first, then materialize the array when positions are requested and the expression is not already a `Column`. Both positions and values reference the resulting column. Plain array-column inputs retain their existing single projection, and non-positional explode variants remain unchanged. Rename the intermediate column prefix to `__comet_explode_` and update the comment about eventually removing the outer wrapper: positional generators still need to share computed arrays after that migration. ## How are these changes tested? - Added a Rust planner regression that executes the generated operators with a counting array function over two input batches. It covers all eight combinations of outer/non-outer, positional/non-positional, and column/computed input, checking evaluation counts, projection counts, values, and positions for nonempty, empty, null arrays, and null elements. - The regression fails on the unmodified planner: plain positional explode evaluates the function four times for two batches. With this change it evaluates twice. - All 9 Rust tests matching `explode` passed. - Added a Parquet-backed test for `posexplode(split(...))` and `posexplode_outer(split(...))`, comparing Spark results and requiring Comet execution. All 38 tests in `CometGenerateExecSuite` passed on Spark 4.1.3 / Java 17. - Workspace `cargo clippy --all-targets --workspace -- -D warnings`, Rust formatting, Spotless, Apache RAT, and whitespace checks passed. [Fork CI](https://github.com/rich7420/datafusion-comet/actions/runs/33980633650) completed successfully for this commit. Conditional jobs were skipped according to the workflow configuration. The measured claim is removal of duplicate evaluation; no wall-clock speedup is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
