andygrove opened a new pull request, #6876: URL: https://github.com/apache/datafusion-comet/pull/6876
## Which issue does this PR close? Closes #6570. Backport of #6700 to `branch-1.1`. ## Rationale for this change The JVM codegen dispatcher initialized kernels from the Spark task partition rather than the partition computed by the native plan. Those indexes can differ under union, coalesce, and cartesian product, producing incorrect results for partition-seeded expressions and sharing random state across coalesced parent partitions. ## What changes are included in this PR? Cherry-picks #6700 with `-x`. The native planner now passes its partition index and plan ID through the UDF bridge, nondeterministic kernels are cached per native plan, and closing a plan releases its kernels. Deterministic kernels remain shared within a task. The 1.1 branch does not contain a later Parquet JVM-UDF test touched only to update a constructor call, so that inapplicable test-only hunk is omitted. The backport also adds the existing `isSpark40Plus` helper import needed by the new regression test on this branch. ## How are these changes tested? Ran: ```text make core ./mvnw test -Dtest=none -Dsuites='org.apache.comet.CometCodegenSourceSuite,org.apache.comet.CometCodegenSuite,org.apache.comet.udf.CometUdfBridgeSuite' -Dscalastyle.skip=true ``` Result: 3 suites completed, 171 tests passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
