huyuanfeng2018 opened a new issue, #11138:
URL: https://github.com/apache/paimon/issues/11138

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Paimon version
   
   1.13
   
   ### Compute Engine
   
   flink 1.20 
   
   ### Minimal reproduce step
   
   1. Create a primary-key table with columns `id BIGINT` and `version INT`,
      using these options:
   
      'bucket' = '1'
      'bucket-key' = 'id'
      'merge-engine' = 'deduplicate'
      'write-only' = 'true'
      'changelog-producer' = 'none'
      'file.format' = 'parquet'
      'sink.writer-coordinator.enabled' = 'true'
   
      Do not configure `sequence.field`.
   
   2. Insert (id=1, version=3) and commit snapshot S1.
   
   3. Start a streaming job with a checkpointed, replayable source.
      Emit 20 other primary keys, followed by (id=1, version=4).
   
   4. Trigger checkpoint 1. Wait for checkpoint completion and publication
      of snapshot S2. Verify that the table returns version=4.
   
   5. Emit an update to version=5, then deliberately fail the source task
      before another checkpoint. Let Flink recover from checkpoint 1
      without restarting the entire cluster.
   
   6. Replay DELETE(id=1, version=4) followed by INSERT(id=1, version=5).
      Wait for the restored writer to process the records, then trigger
      and complete checkpoint 2.
   
   7. Query the latest table snapshot. It still returns version=4.
   
   8. Repeat in a separate control run with
      'sink.writer-coordinator.enabled' = 'false'.
      The result correctly returns version=5.
   
   Use manually triggered checkpoints and a long periodic checkpoint
   interval to prevent an unrelated checkpoint from refreshing the
   coordinator cache before the replay.
   
   
   ### What doesn't meet your expectations?
   
   Expected:
   After checkpoint recovery, the replayed update should become visible.
   The latest snapshot should return (id=1, version=5).
   
   Actual:
   With the writer coordinator enabled, the source restores its checkpointed
   progress and replays the update. The update is written and the next
   checkpoint completes, but the latest snapshot still returns version=4.
   
   The old row has internal sequence number 21, while the replayed INSERT
   receives sequence number 2. The old row therefore wins during merging.
   
   In the coordinator-disabled control run, the replayed INSERT receives
   sequence number 23 and the final result correctly returns version=5.
   
   
   ### Anything else?
   
   Reproduced with Flink 1.20.1 MiniCluster, Zulu JDK 8 (1.8.0_472),
   and a downstream Paimon 1.3 development checkout
   (project version: 1.3-shein-SNAPSHOT).
   A clean upstream release/master reproduction has not yet been run.
   
   Suspected cause:
   WriteOperatorCoordinator refreshes its cached snapshot when a checkpoint
   is triggered, before that checkpoint's table snapshot is committed.
   notifyCheckpointComplete(), resetToCheckpoint(), and subtaskReset() are
   empty, so task recovery can retain an outdated snapshot cache.
   
   The restored writer initializes its sequence number from those stale
   file metadata, causing replayed updates to receive sequence numbers
   lower than already committed rows.
   
   Refreshing the coordinator cache before creating the restored writer
   also produces the correct result in a lower-level control test.
   
   No MySQL server or binlog decoding error is required for reproduction.
   A deliberately injected source-task exception is sufficient.
   
   Two reproducer tests are available:
   - WriterCoordinatorRecoveryReproTest
   - WriterCoordinatorFlinkRecoveryReproTest
   
   Five diagnostic test invocations completed with zero failures or skips.
   The affected cases intentionally assert the observed incorrect result;
   passing these diagnostic tests means the bug was reproduced, not fixed.
   
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to