JingsongLi commented on PR #9535: URL: https://github.com/apache/paimon/pull/9535#issuecomment-5748001481
Requirement fit: SUPPORTED. Implementation: FINDINGS. [P1] Restrict the expired-checkpoint fallback to the dedicated compaction recovery contract `DataTableStreamScan.restore` is shared by every streaming scan, but this PR changes it globally: whenever the checkpointed `nextSnapshotId` is below the earliest retained snapshot, it clears the checkpoint and reruns the configured starting scanner. That is not equivalent for ordinary consumers. For example, a stream configured with a latest-style startup can restore at N after N expires while N+1..M are still retained; resetting to the starting scanner resumes after M and silently skips those retained snapshots. Other startup modes can replay a full snapshot instead. The previous behavior was a stall, but replacing it with data loss or replay outside the dedicated compaction source is worse. Please scope this recovery to `ContinuousCompactorStartingScanner` / the dedicated compact job, or define an explicit generic expired-checkpoint policy that resumes from the earliest retained snapshot without applying the initial startup mode. Add an end-to-end restore test with an expired checkpoint and still-retained later snapshots, asserting exactly which snapshots are emitted. The current 13-line change has no regression test, and the Spark 2.13 jobs are still failing, so the broad behavior is not yet safe to merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
