jiwen624 opened a new pull request, #18806: URL: https://github.com/apache/iceberg/pull/18806
Flink upsert and CDC writers emit equality deletes only for the table's current partition spec. After the spec changes, updates and deletes miss rows committed under older specs. Upserts leave duplicate keys and deletes are lost, without any error. This documents the limitation and a workaround: stop the job, change the spec, rewrite all data into the new spec, then restart. Related: #18748 Verified on main with FlinkSink and Flink `RewriteDataFiles`. Without the workaround, an upsert after adding `bucket(id, 4)` returns `1/aaa, 1/ccc, 2/bbb`. With it, the same upsert returns `1/ccc`, and so do a `bucket(2)` → `bucket(4)` change and CDC. Running an upsert before the rewrite still produces the wrong result. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
