jiwen624 opened a new pull request, #18846: URL: https://github.com/apache/iceberg/pull/18846
When `OverwriteFiles` replaces data files and has a conflict detection filter, the check for concurrently added deletes prunes delete files by that filter using the delete files' own column stats. An equality delete whose key values fall outside the filter is skipped even when it applies to rows in a replaced file. The overwrite then commits, writing those rows into new data files whose data sequence number is higher than the equality delete's, so the delete no longer applies and the rows reappear. Spark's copy-on-write row-level operations commit through this path. This change checks new deletes for the replaced files without the conflict filter. `DeleteFileIndex` still matches each delete against the replaced file's own stats. **Repro** (core API): a file with x = 0..3. An overwrite of that file is planned with conflictDetectionFilter(x = 0). An equality delete for x = 2 commits concurrently. - Before: the overwrite commits, and the row with x = 2 is visible again. - After: the overwrite fails with Cannot commit, found new delete for replaced data file. **Test plan**: - New `testConcurrentEqualityDeletesOutsideConflictFilterForReplacedFile` (v2, v3) fails on main and passes with this change. - `FILE_DAY_2_ANOTHER_RANGE_EQ_DELETES` now declares its equality field. Without one, `testConcurrentNonConflictingEqualityDeletes` passed only because of the removed pruning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
