zhoulii opened a new issue, #6965: URL: https://github.com/apache/paimon/issues/6965
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation In Flink lookup joins with Paimon tables, if the lookup table has frequent updates during the refresh interval, a large number of snapshots can accumulate. The current implementation refreshes the lookup table by incrementally loading each pending snapshot one by one. When there are many pending snapshots (e.g., dozens or hundreds), this incremental approach can be very slow and inefficient. ### Solution Add a new configuration option `lookup.refresh.full-load-threshold` that allows users to set a threshold for the number of pending snapshots. When the number of snapshots pending for refresh exceeds this threshold, the system will perform a full table reload (close and reopen) instead of incremental refresh. ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
