liming30 opened a new issue, #1561: URL: https://github.com/apache/incubator-paimon/issues/1561
### Search before asking - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar. ### Paimon version 0.5-SNAPSHOT ### Compute Engine Flink 1.17.1 ### Minimal reproduce step This is a corner case. We use an asynchronous thread (`workerExecutor`) to perform the scan operation, and add the result of the scan to the `SplitAssigner` in (`coordinatorExecutor`). There is no guarantee that it will be executed in the one by one order of `workerExecutor(scan) -> coordinatorExecutor(processDiscoveredSplits) -> workerExecutor(scan)....` When the discoveryInterval is small or the GC is heavy, the execution order of `scan -> scan -> processDiscoveredSplits -> processDiscoveredSplits` may appear.The `SnapshotId` of the next scan will be updated in `processDiscoveredSplits`. If it is executed in the order of `scan -> scan -> processDiscoveredSplits -> checkpoint -> failover`, the intermediate snapshot will be lost. ### What doesn't meet your expectations? The Checkpoint made by Enumerator should not lose the historical snapshot. ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
