rionmonster commented on code in PR #2265:
URL: https://github.com/apache/fluss/pull/2265#discussion_r2653465669
##########
fluss-lake/fluss-lake-iceberg/src/test/java/org/apache/fluss/lake/iceberg/maintenance/IcebergRewriteITCase.java:
##########
@@ -186,6 +186,9 @@ void testLogTableCompaction() throws Exception {
t1, t1Bucket, ++i, true,
Collections.singletonList(row(1, "v1"))));
checkFileStatusInIcebergTable(t1, 3, false);
+ // Ensure tiering job has fully processed the previous writes
+ assertReplicaStatus(t1Bucket, i);
Review Comment:
@luoyuxia
I've been digging into this a bit further and it seems there's some
disparity between the actual files being written and retrieving the latest
offsets for those (specifically after any types of asynchronous operations,
such as compaction, etc.). I think we may need some mechanism to improve the
consistency, at least within the bounds of the tests.
I wrote a little monitor to run through several iterations of the tests to
see what the state of various bits looked like during each iteration. Here's a
sample of those:
**Successful Test (Successfully Performs Compaction as Expected)**
[MONITOR] log_table_33 - TableBucket{tableId=32, bucket=0}
| IcebergFiles | IcebergSnapshotId | LakeSnapshotId | ReplicaLakeSnapId |
ReplicaLakeLogEndOff | Timestamp |
|--------------|-------------------|----------------|-------------------|----------------------|-----------|
| 0 | -1 | -1 | -1 | -1
| 1767113560800 |
| 0 | -1 | -1 | -1 | -1
| 1767113561046 |
| 0 | -1 | -1 | -1 | -1
| 1767113561294 |
| 0 | -1 | -1 | -1 | -1
| 1767113561547 |
| 0 | -1 | -1 | -1 | -1
| 1767113561795 |
| 0 | -1 | -1 | -1 | -1
| 1767113562045 |
| 0 | -1 | -1 | -1 | -1
| 1767113562299 |
| 0 | -1 | -1 | -1 | -1
| 1767113562547 |
| 0 | -1 | -1 | -1 | -1
| 1767113562797 |
| 0 | -1 | -1 | -1 | -1
| 1767113563054 |
| 0 | -1 | -1 | -1 | -1
| 1767113563298 |
| 0 | -1 | -1 | -1 | -1
| 1767113563546 |
| 0 | -1 | -1 | -1 | -1
| 1767113563796 |
| 0 | -1 | -1 | -1 | -1
| 1767113564048 |
| 0 | -1 | -1 | -1 | -1
| 1767113564295 |
| 0 | -1 | -1 | -1 | -1
| 1767113564544 |
| 0 | -1 | -1 | -1 | -1
| 1767113564796 |
| 0 | -1 | -1 | -1 | -1
| 1767113565047 |
| 0 | -1 | -1 | -1 | -1
| 1767113565295 |
| 0 | -1 | -1 | -1 | -1
| 1767113565545 |
| 0 | -1 | -1 | -1 | -1
| 1767113565797 |
| 0 | -1 | -1 | -1 | -1
| 1767113566046 |
| 0 | -1 | -1 | -1 | -1
| 1767113566294 |
| 0 | -1 | -1 | -1 | -1
| 1767113566545 |
| 0 | -1 | -1 | -1 | -1
| 1767113566795 |
| 0 | -1 | -1 | -1 | -1
| 1767113567047 |
| 0 | -1 | -1 | -1 | -1
| 1767113567296 |
| 0 | -1 | -1 | -1 | -1
| 1767113567547 |
| 0 | -1 | -1 | -1 | -1
| 1767113567793 |
| 0 | -1 | -1 | -1 | -1
| 1767113568048 |
| 0 | -1 | -1 | -1 | -1
| 1767113568296 |
| 0 | -1 | -1 | -1 | -1
| 1767113568546 |
| 0 | -1 | -1 | -1 | -1
| 1767113568797 |
| 0 | -1 | -1 | -1 | -1
| 1767113569048 |
| 0 | -1 | -1 | -1 | -1
| 1767113569297 |
| 0 | -1 | -1 | -1 | -1
| 1767113569547 |
| 0 | -1 | -1 | -1 | -1
| 1767113569796 |
| 0 | -1 | -1 | -1 | -1
| 1767113570045 |
| 0 | -1 | -1 | -1 | -1
| 1767113570298 |
| 0 | -1 | -1 | -1 | -1
| 1767113570547 |
| 0 | -1 | -1 | -1 | -1
| 1767113570799 |
| 0 | -1 | -1 | -1 | -1
| 1767113571045 |
| 0 | -1 | -1 | -1 | -1
| 1767113571294 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113571547 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113571799 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113572047 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113572301 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113572549 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113572800 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113573044 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113573301 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113573549 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113573798 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113574048 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113574298 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113574551 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113574801 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113575047 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113575300 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113575549 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113575796 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113576051 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113576300 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113576547 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113576800 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113577048 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113577304 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113577551 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113577800 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113578046 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113578300 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113578550 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113578800 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113579050 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113579301 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113579548 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113579799 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113580048 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113580298 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113580550 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113580798 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113581056 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113581297 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113581550 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113581801 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113582049 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113582301 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113582548 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113582798 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113583046 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113583298 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113583548 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113583800 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113584054 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113584303 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113584550 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113584804 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113585051 |
| 1 | 5443797100773076340 | 5443797100773076340 |
5443797100773076340 | 1 | 1767113585294 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113585558 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113585810 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113586062 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113586310 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113586559 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113586805 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113587059 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113587307 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113587559 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113587810 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113588056 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113588310 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113588558 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113588809 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113589059 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113589310 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113589558 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113589813 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113590057 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113590309 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113590557 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113590807 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113591058 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113591311 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113591557 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113591811 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113592058 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113592310 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113592560 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113592809 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113593058 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113593310 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113593559 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113593810 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113594057 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113594312 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113594560 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113594808 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113595063 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113595309 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113595559 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113595810 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113596060 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113596316 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113596558 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113596808 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113597060 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113597312 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113597557 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113597809 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113598057 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113598307 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113598558 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113598817 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113599062 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113599311 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113599562 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113599809 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113600058 |
| 2 | 5171148310110237921 | 5171148310110237921 |
5443797100773076340 | 2 | 1767113600306 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113600560 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113600811 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113601060 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113601309 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113601560 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113601807 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113602059 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113602307 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113602560 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113602809 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113603057 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113603307 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113603561 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113603806 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113604057 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113604307 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113604553 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113604812 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113605056 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113605311 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113605559 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113605809 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113606058 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113606308 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113606559 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113606811 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113607060 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113607314 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113607557 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113607808 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113608059 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113608310 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113608561 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113608811 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113609056 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113609309 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113609560 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113609810 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113610060 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113610312 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113610560 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113610811 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113611061 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113611308 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113611554 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113611808 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113612060 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113612311 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113612561 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113612811 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113613062 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113613311 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113613561 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113613808 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113614061 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113614310 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113614560 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113614809 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113615059 |
| 3 | 5182733673261799288 | 5182733673261799288 |
5443797100773076340 | 3 | 1767113615307 |
| 2 | 2575057976625237982 | 2575057976625237982 |
5443797100773076340 | 4 | 1767113615557 |
| 2 | 2575057976625237982 | 2575057976625237982 |
5443797100773076340 | 4 | 1767113615808 |
**Failing Test (File appeared to never be properly written before expected
offset)**
[MONITOR] log_table_34 - TableBucket{tableId=33, bucket=0}
| IcebergFiles | IcebergSnapshotId | LakeSnapshotId | ReplicaLakeSnapId |
ReplicaLakeLogEndOff | Timestamp |
|--------------|-------------------|----------------|-------------------|----------------------|-----------|
| 0 | -1 | -1 | -1 | -1
| 1767113616327 |
| 0 | -1 | -1 | -1 | -1
| 1767113616573 |
| 0 | -1 | -1 | -1 | -1
| 1767113616823 |
| 0 | -1 | -1 | -1 | -1
| 1767113617072 |
| 0 | -1 | -1 | -1 | -1
| 1767113617326 |
| 0 | -1 | -1 | -1 | -1
| 1767113617569 |
| 0 | -1 | -1 | -1 | -1
| 1767113617826 |
| 0 | -1 | -1 | -1 | -1
| 1767113618068 |
| 0 | -1 | -1 | -1 | -1
| 1767113618326 |
| 0 | -1 | -1 | -1 | -1
| 1767113618574 |
| 0 | -1 | -1 | -1 | -1
| 1767113618823 |
| 0 | -1 | -1 | -1 | -1
| 1767113619071 |
| 0 | -1 | -1 | -1 | -1
| 1767113619321 |
| 0 | -1 | -1 | -1 | -1
| 1767113619572 |
| 0 | -1 | -1 | -1 | -1
| 1767113619825 |
| 0 | -1 | -1 | -1 | -1
| 1767113620069 |
| 0 | -1 | -1 | -1 | -1
| 1767113620322 |
| 0 | -1 | -1 | -1 | -1
| 1767113620576 |
| 0 | -1 | -1 | -1 | -1
| 1767113620822 |
| 0 | -1 | -1 | -1 | -1
| 1767113621069 |
| 0 | -1 | -1 | -1 | -1
| 1767113621324 |
| 0 | -1 | -1 | -1 | -1
| 1767113621572 |
| 0 | -1 | -1 | -1 | -1
| 1767113621826 |
| 0 | -1 | -1 | -1 | -1
| 1767113622072 |
| 0 | -1 | -1 | -1 | -1
| 1767113622321 |
| 0 | -1 | -1 | -1 | -1
| 1767113622574 |
| 0 | -1 | -1 | -1 | -1
| 1767113622823 |
| 0 | -1 | -1 | -1 | -1
| 1767113623069 |
| 0 | -1 | -1 | -1 | -1
| 1767113623324 |
| 0 | -1 | -1 | -1 | -1
| 1767113623573 |
| 0 | -1 | -1 | -1 | -1
| 1767113623821 |
| 0 | -1 | -1 | -1 | -1
| 1767113624070 |
| 0 | -1 | -1 | -1 | -1
| 1767113624320 |
| 0 | -1 | -1 | -1 | -1
| 1767113624569 |
| 0 | -1 | -1 | -1 | -1
| 1767113624821 |
| 0 | -1 | -1 | -1 | -1
| 1767113625073 |
| 0 | -1 | -1 | -1 | -1
| 1767113625324 |
| 0 | -1 | -1 | -1 | -1
| 1767113625573 |
| 0 | -1 | -1 | -1 | -1
| 1767113625824 |
| 0 | -1 | -1 | -1 | -1
| 1767113626071 |
| 0 | -1 | -1 | -1 | -1
| 1767113626319 |
| 0 | -1 | -1 | -1 | -1
| 1767113626573 |
| 0 | -1 | -1 | -1 | -1
| 1767113626824 |
| 0 | -1 | -1 | -1 | -1
| 1767113627070 |
| 0 | -1 | -1 | -1 | -1
| 1767113627324 |
| 0 | -1 | -1 | -1 | -1
| 1767113627574 |
| 0 | -1 | -1 | -1 | -1
| 1767113627818 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113628077 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113628325 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113628580 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113628828 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113629073 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113629330 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113629577 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113629829 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113630076 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113630324 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113630578 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113630828 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113631075 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113631326 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113631580 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113631827 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113632076 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113632326 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113632577 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113632826 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113633074 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113633325 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113633578 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113633829 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113634077 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113634327 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113634573 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113634826 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113635072 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113635326 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113635582 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113635828 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113636072 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113636327 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113636576 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113636826 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113637076 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113637327 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113637576 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113637829 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113638076 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113638326 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113638580 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113638827 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113639077 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113639327 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113639575 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113639829 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113640073 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113640326 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113640575 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113640827 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113641071 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113641323 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113641576 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113641834 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113642073 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113642330 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113642574 |
| 1 | 6752171114485570420 | 6752171114485570420 |
6752171114485570420 | 1 | 1767113642820 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113643084 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113643343 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113643586 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113643835 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113644086 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113644333 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113644589 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113644833 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113645090 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113645339 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113645588 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113645838 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113646087 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113646340 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113646591 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113646842 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113647089 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113647340 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113647593 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113647847 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113648089 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113648337 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113648593 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113648839 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113649086 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113649339 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113649593 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113649843 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113650094 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113650345 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113650597 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113650842 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113651089 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113651339 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113859238 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113859394 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113859633 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113859872 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113860134 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113860382 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113860628 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113860879 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113861129 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113861384 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113861627 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113861882 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113862135 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113862381 |
| 2 | 7273969972093574431 | 7273969972093574431 |
7273969972093574431 | 2 | 1767113862633 |
```
[ASSERTION FAILURE] Expected offset 3 but got 2 for bucket
TableBucket{tableId=33, bucket=0}
Replica Lake Snapshot ID: 7273969972093574431
Current State:
Iceberg Files: 2
Iceberg Snapshot ID: 7273969972093574431
Lake Snapshot ID (from admin): 7273969972093574431
Replica Lake Snapshot ID: 7273969972093574431
Replica Lake Log End Offset: 2
```
I'm not sure if this is more of an artifact of the tests themselves or a
legitimate issue. Any thoughts? Happy to continue digging. It _feels_ like a
race-condition due to the inconsistency, either on writing the files or reading
stale offsets from the data lake directly. I'm sure in a real-world
environment, this might be tolerable (as I suspect it's just a minor latency
spike which would eventually resolve), but in the confines of a test, it's
flaky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
