FreakingMind opened a new pull request, #7633:
URL: https://github.com/apache/datafusion-comet/pull/7633

   ## Which issue does this PR close?
   
   Closes #7596.
   
   ## Rationale for this change
   
   Native MergeRows builds its two "not matched" masks with arrow's `and_not`. 
In arrow 59 `and_not` gives wrong values or panics when an input has a non-zero 
offset, both when the offsets differ and when they are the same 
(apache/arrow-rs#10698). The fix, apache/arrow-rs#10699, is only in arrow 60.
   
   The markers are `IsNotNull` over join output columns. On a sliced batch, 
`is_not_null` keeps the slice's offset for a column with nulls and returns 
offset 0 for a column without nulls. So the masks can end up at offsets where 
`and_not` breaks. Then the task either panics with `offset + len out of bounds` 
or applies inserts and deletes to the wrong rows. We hit the panic on Iceberg 
MERGE with `spark.comet.exec.mergeRows.enabled=true`.
   
   ## What changes are included in this PR?
   
   The two masks are now computed as `and(a, not(b))`, which gives the same 
result, nulls included. The comment above them says to go back to `and_not` 
once Comet is on arrow 60.
   
   branch-1.2 has the same code on arrow 59.3.2, so this could be a candidate 
for a 1.2 backport.
   
   ## How are these changes tested?
   
   New unit test `not_matched_masks_with_sliced_markers` in `merge_rows.rs`. It 
slices the marker columns at offsets (0, 3), (3, 0), (7, 7) and (64, 64) and 
checks the not-matched and not-matched-by-source rows separately. On main the 
first three offset pairs give wrong rows and the last one panics. With the fix 
all of them pass.
   
   I did not add a SQL test, because SQL can't control whether the batch 
reaching MergeRows is sliced. `CometMergeRowsNativeSuiteBase` already covers 
the three action groups end to end.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to