FreakingMind opened a new pull request, #7633: URL: https://github.com/apache/datafusion-comet/pull/7633
## Which issue does this PR close? Closes #7596. ## Rationale for this change Native MergeRows builds its two "not matched" masks with arrow's `and_not`. In arrow 59 `and_not` gives wrong values or panics when an input has a non-zero offset, both when the offsets differ and when they are the same (apache/arrow-rs#10698). The fix, apache/arrow-rs#10699, is only in arrow 60. The markers are `IsNotNull` over join output columns. On a sliced batch, `is_not_null` keeps the slice's offset for a column with nulls and returns offset 0 for a column without nulls. So the masks can end up at offsets where `and_not` breaks. Then the task either panics with `offset + len out of bounds` or applies inserts and deletes to the wrong rows. We hit the panic on Iceberg MERGE with `spark.comet.exec.mergeRows.enabled=true`. ## What changes are included in this PR? The two masks are now computed as `and(a, not(b))`, which gives the same result, nulls included. The comment above them says to go back to `and_not` once Comet is on arrow 60. branch-1.2 has the same code on arrow 59.3.2, so this could be a candidate for a 1.2 backport. ## How are these changes tested? New unit test `not_matched_masks_with_sliced_markers` in `merge_rows.rs`. It slices the marker columns at offsets (0, 3), (3, 0), (7, 7) and (64, 64) and checks the not-matched and not-matched-by-source rows separately. On main the first three offset pairs give wrong rows and the last one panics. With the fix all of them pass. I did not add a SQL test, because SQL can't control whether the batch reaching MergeRows is sliced. `CometMergeRowsNativeSuiteBase` already covers the three action groups end to end. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
