zhangyue19921010 opened a new issue, #11344:
URL: https://github.com/apache/paimon/issues/11344

   ### Search before asking
   
   - [x] I searched the issues and pull requests and found no matching 
reader-progress reset report.
   
   ### Paimon version
   
   master, `ad4f6e09ef51c6b9932cbf22f0be563c6dde683f` (2.2-SNAPSHOT).
   
   ### Compute Engine
   
   Flink 1.20.1 source connector. Reproduced with a real Paimon append table 
and consumer file through the source enumerator APIs on JDK 11; no full Flink 
job was run.
   
   ### Minimal reproduce step
   
   1. Create an ordinary append table with `consumer-id` and 
`scan.mode=from-snapshot`, `scan.snapshot-id=1`, and commit three snapshots.
   2. Complete a checkpoint while a reader is assigned snapshot 1. Its consumer 
progress is 1.
   3. Let the reader finish snapshots 1 and 2 and receive snapshot 3. An 
additional checkpoint may be captured but not completed.
   4. Reset the reader to the completed checkpoint. Flink returns only the 
later assignments to the enumerator, while the reader restores its snapshot-1 
split separately. The restored reader does not request another split while 
replaying that split.
   5. Complete a new checkpoint during replay. The current source writes 
consumer progress 3, although the restored reader still needs snapshot 1.
   
   The 
[regressions](https://github.com/zhangyue19921010/flink-table-store/blob/47063820395755e7ce9cc7c02da186c752208daa/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/ConsumerProgressFailoverTest.java)
 cover returned and empty assignments, readers that had requested another 
split, and older pending checkpoints. The two original reader-reset cases fail 
on unchanged production code with consumer progress 3 instead of 1.
   
   ### What doesn't meet your expectations?
   
   Consumer progress must not move past splits still needed by a restored 
reader. The old reader's assigned/consumed progress, pending split request and 
checkpoint minima survive `addSplitsBack`, so a later checkpoint can publish 
progress calculated before the reset.
   
   ### Anything else?
   
   Reset the affected reader's progress and pending split request, and discard 
pending checkpoint minima when assignments are returned. Fresh reader progress 
then advances the consumer normally. This is separate from the 
dedicated-compaction expired-checkpoint fallback in #9535; it does not change 
scan startup or expired-snapshot recovery policy.
   
   ### Are you willing to submit a PR?
   
   - [x] I am willing to submit a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to