Guosmilesmile opened a new pull request, #18149: URL: https://github.com/apache/iceberg/pull/18149
Follow up https://github.com/apache/iceberg/pull/18144. Part of https://github.com/apache/iceberg/issues/18142 Add support for reloading the full cache of an Iceberg lookup join, instead of keeping the initial snapshot for the lifetime of the job. The reload is opt-in, so the existing behavior remains unchanged unless a reload option is configured. ### Changes - Support the standard Flink full cache reload options: - `lookup.full-cache.reload-strategy` (`PERIODIC` or `TIMED`) - `lookup.full-cache.periodic-reload.interval` / `.schedule-mode` - `lookup.full-cache.timed-reload.iso-time` / `.interval-in-days` - Infer the reload strategy from the configured options when it is not explicitly set. - Use the standard Flink reload triggers. - Reload from the current snapshot and atomically replace the existing cache after the new cache is fully loaded. - Skip the reload when there is no new snapshot. - Fail the job on the next lookup if a reload fails, instead of continuing to serve the previous cache. Since Flink triggers swallow reload failures, the lookup function maintains and latches the failure state. - Keep the initial cache load separate from reloads. A trigger only reloads an already-loaded cache, avoiding concurrent access to the same `Table` instance from the initial lookup and the reload thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
