fmorillo7694 commented on issue #15846:
URL: https://github.com/apache/iceberg/issues/15846#issuecomment-5476568304
We independently reproduced this at AWS while investigating a customer
report.
I ported the three post-barrier tests from #15913 onto unpatched `main` (and
also ran them at the `apache-iceberg-1.11.0` tag and on the `flink/v2.3`
module). All 18 parameterized scenarios fail with the described missing-record
signature, across formatVersion {1, 2} and AVRO/PARQUET/ORC: a record processed
after `snapshotState(N)` is absent from the committed table after checkpoint N
fails and the job recovers. For example:
```
Expecting actual:
[Record(3, early-checkpoint2), Record(2, post-barrier-checkpoint1)]
to contain exactly in any order:
[Record(3, early-checkpoint2), Record(2, post-barrier-checkpoint1),
Record(1, early-checkpoint1)]
```
The existing aligned-checkpoint committer tests all pass on the same builds,
consistent with the analysis that the committer's assumption only breaks under
unaligned checkpoints.
Since this also reproduces on `flink/v2.3`, the upcoming 1.12 release would
ship the bug on the new Flink 2.3 support unless a fix lands before the cut.
Happy to help review and test a revival of #15913. cc @mxm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
