jiwen624 opened a new issue, #18748:
URL: https://github.com/apache/iceberg/issues/18748
### Apache Iceberg version
main (development)
### Query engine
Flink
### Please describe the bug 🐞
After a table's partition spec changes, Flink upsert and CDC writes can
silently fail to update or delete rows written before the change:
- An upsert of an existing key leaves both the old and the new version in
the table, so the primary key is duplicated.
- A CDC delete of such a row has no effect, and the row stays visible.
No error or warning is raised. Compacting the table does not fix it: the
stale rows are rewritten as ordinary live rows, and a key is only corrected if
it changes again afterwards. This affects v2 and v3 tables (v4 prohibits new
equality deletes) with `FlinkSink`, `IcebergSink` and `DynamicIcebergSink`.
Reproduction (v2 table, upsert mode, equality field id):
```
1. Upsert (1, 'aaa'), (2, 'bbb') into an unpartitioned table.
2. table.updateSpec().addField(bucket("id", 4)).commit()
-- same as Spark: ALTER TABLE t ADD PARTITION FIELD bucket(4, id)
3. Restart the Flink job and upsert (1, 'ccc').
4. SELECT * FROM t
expected: (1, 'ccc'), (2, 'bbb')
actual: (1, 'aaa'), (1, 'ccc'), (2, 'bbb')
```
The same happens after changing the bucket count, and for CDC deletes of
rows written before the change.
`DynamicIcebergSink` hits this without a manual step, because it evolves the
table's spec when records declare a new one. Three variants:
- (a) a row upserted under an unpartitioned spec and later, in another
checkpoint, under bucket(id, 10) ends up as two identical rows;
- (b) the same, but with both writes in one checkpoint;
- (c) a writer going back to an earlier spec (bucket(2) → bucket(4) →
bucket(2)) leaves the bucket(4) version behind.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
