jiwen624 opened a new pull request, #18747: URL: https://github.com/apache/iceberg/pull/18747
Flink upsert/CDC writers emit equality deletes only for the table's current partition spec, and an equality delete applies only to files of the same spec unless that spec is unpartitioned. After a spec change, updates and deletes therefore miss rows committed under older specs: upserts leave duplicate keys and deletes are lost. With this PR, the writer now also emits each equality delete for the older specs in its table metadata whose partition can be computed from the delete row, or one global delete if an older spec is unpartitioned. DeltaManifests keeps one delete manifest per spec (serializer version 3, still reads 1 and 2). This fixes deletes against rows committed in earlier checkpoints. It does not cover dynamic-sink spec changes within one checkpoint, or cached dynamic-sink writer factories with stale spec metadata (related: #18675). **Test plan:** - TestDeltaTaskWriter: upsert after bucket(2) → bucket(4), and CDC deletes of one key stored in two old partitions, return the expected rows. Both fail on main. - TestIcebergFilesCommitter: an older-spec delete keeps its spec through the committer and applies. Fails on main. - TestDynamicIcebergSink: an upsert in a later checkpoint, after the sink evolves the spec, leaves one row (two on main). - TestFlinkManifest: version 2 checkpoint state still restores. - All Flink 2.3 sink tests pass. Backports to 1.20 and 2.2 will follow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
