andygrove opened a new pull request, #7642:
URL: https://github.com/apache/datafusion-comet/pull/7642

   ## Which issue does this PR close?
   
   Backport of #7126 to `branch-1.2`. Closes #6922 on this branch.
   
   ## Rationale for this change
   
   A native left outer join building on the right, or right outer join building 
on the left, keeps the streamed side's order only when the serde declares it. 
When Comet can't verify that order, it declares none but still converts the 
join, while Spark has already dropped the sort above, so unmatched rows land 
after matched ones and the query returns wrong rows (#6922, 
`priority:critical`). `branch-1.2` has the same `CometHashJoin` serde and the 
same sorted-cache and checkpoint paths. See #7126 for the full analysis.
   
   ## What changes are included in this PR?
   
   Cherry-pick of c56f6a0209 (`git cherry-pick -x`). An outer hash join whose 
streamed order comes from the JVM (cache, checkpoint, Spark leaf) or from a 
nested loop join now falls back to Spark with a reason. Chains inside one 
native plan stay native. The source PR's compatibility and contributor guide 
updates and its tests in `CometJoinSuite` and `CometInMemoryCacheSuite` come 
with it.
   
   One adaptation, in test code only: the import block of `CometJoinSuite` 
conflicted because `main` imports `SQLExecution` from a later change. The pick 
keeps `branch-1.2`'s `org.apache.spark.sql.execution` import line and adds 
`CometHashJoin` to the `org.apache.spark.sql.comet` import line, which is the 
only line the source commit changes there. The production diff is identical to 
the source commit apart from line numbers and blob hashes.
   
   Other release branches: `branch-1.2` is the newest release branch, so there 
is no newer one to port to. Whether `branch-1.1` needs a backport is not part 
of this PR.
   
   ## How are these changes tested?
   
   On this branch, with JDK 17 and the default Spark profile: `./mvnw 
test-compile` succeeds. The six new tests plus the existing outer hash join 
tests in `CometJoinSuite` and `CometInMemoryCacheSuite` pass (20 tests, 0 
failures) against a native library built from this branch. With 
`operators.scala` reverted to `branch-1.2`'s version (keeping only the 
`CometHashJoin` companion object so the tests compile), four of them fail: 
`outer hash joins over a checkpointed streamed side fall back and keep its 
order`, `an outer hash join over a native inner join over a checkpoint falls 
back`, `an outer hash join over a native nested loop join falls back` and `an 
outer hash join over a cache of an outer hash join over a native filter keeps 
order`. So the branch has the bug and the tests catch it. No native source 
changed, so the native build was not affected.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to