Guosmilesmile commented on PR #18144: URL: https://github.com/apache/iceberg/pull/18144#issuecomment-5701189440
For snapshot pinning, I was thinking we could resolve the snapshot at job submission time and pass the snapshot ID to every TaskManager, so that all caches use the same snapshot. But there is a problem with this approach. The problem is failover. Flink doesn't re-run the planner when a task restarts, so the pinned snapshot ID would remain unchanged. If that snapshot has expired by then, the cache load would fail with Cannot find snapshot with ID ... after the restart, and retries would keep failing until the job is resubmitted. Periodic refresh doesn't help here, because the failure happens during the initial cache load. We also don't have a way to persist and update the snapshot ID from the connector side, since a sync lookup join doesn't have checkpointed state. So while pinning the snapshot would ensure consistency across subtasks, it introduces a failover problem. For this PR, I therefore just added logging and a metric to report the snapshot ID being used.If you have a simpler idea, we'd be happy to take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
