JTidbury2 opened a new pull request, #18675: URL: https://github.com/apache/iceberg/pull/18675
Fixes #18262. When a table's schema changes mid-checkpoint, `DynamicWriter` opens a second `TaskWriter` for it. That writer can't see the rows the first writer inserted, so re-writing or deleting one of those keys produces an equality delete with the same sequence number as the row. The delete never applies, and the old row stays visible. This PR makes all writers of one table (same branch, spec, upsert mode and equality fields) share one inserted-row tracker per partition for the checkpoint, so those re-writes become position deletes. If equality-key types differ, for example after a key column is promoted, the writers keep private trackers and log a WARN. Those cases are tracked in #18350. This is a port of @abrarsher23's [Affirm/iceberg#17](https://github.com/Affirm/iceberg/pull/17) to `main`, opened following @mxm's comment on #18262. Abrar is the commit author. It builds on #18288 and covers the items raised in its review: - key types compared by field ID, optionality and type, not `StructType.equals`; - CDC streams without upsert mode; - one tracker per partition. Core and `flink/v2.3` only. I'll follow up with the `v2.2`/`v1.20` backports. On v3 tables, two writers can now produce DVs for the same data file in one commit. `MergingSnapshotProducer` already merges those at commit time. Tests are in `TestTaskEqualityDeltaWriter`, `TestDeltaTaskWriter`, `TestDynamicWriter` and `TestWriteTarget`. The data and Flink 2.3 sink suites pass locally. Co-authored-by: Abrar Sher <[email protected]> AI disclosure: Claude Code helped port this and draft the description; I've reviewed the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
