andygrove opened a new pull request, #7642: URL: https://github.com/apache/datafusion-comet/pull/7642
## Which issue does this PR close? Backport of #7126 to `branch-1.2`. Closes #6922 on this branch. ## Rationale for this change A native left outer join building on the right, or right outer join building on the left, keeps the streamed side's order only when the serde declares it. When Comet can't verify that order, it declares none but still converts the join, while Spark has already dropped the sort above, so unmatched rows land after matched ones and the query returns wrong rows (#6922, `priority:critical`). `branch-1.2` has the same `CometHashJoin` serde and the same sorted-cache and checkpoint paths. See #7126 for the full analysis. ## What changes are included in this PR? Cherry-pick of c56f6a0209 (`git cherry-pick -x`). An outer hash join whose streamed order comes from the JVM (cache, checkpoint, Spark leaf) or from a nested loop join now falls back to Spark with a reason. Chains inside one native plan stay native. The source PR's compatibility and contributor guide updates and its tests in `CometJoinSuite` and `CometInMemoryCacheSuite` come with it. One adaptation, in test code only: the import block of `CometJoinSuite` conflicted because `main` imports `SQLExecution` from a later change. The pick keeps `branch-1.2`'s `org.apache.spark.sql.execution` import line and adds `CometHashJoin` to the `org.apache.spark.sql.comet` import line, which is the only line the source commit changes there. The production diff is identical to the source commit apart from line numbers and blob hashes. Other release branches: `branch-1.2` is the newest release branch, so there is no newer one to port to. Whether `branch-1.1` needs a backport is not part of this PR. ## How are these changes tested? On this branch, with JDK 17 and the default Spark profile: `./mvnw test-compile` succeeds. The six new tests plus the existing outer hash join tests in `CometJoinSuite` and `CometInMemoryCacheSuite` pass (20 tests, 0 failures) against a native library built from this branch. With `operators.scala` reverted to `branch-1.2`'s version (keeping only the `CometHashJoin` companion object so the tests compile), four of them fail: `outer hash joins over a checkpointed streamed side fall back and keep its order`, `an outer hash join over a native inner join over a checkpoint falls back`, `an outer hash join over a native nested loop join falls back` and `an outer hash join over a cache of an outer hash join over a native filter keeps order`. So the branch has the bug and the tests catch it. No native source changed, so the native build was not affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
