Guosmilesmile commented on PR #17280: URL: https://github.com/apache/iceberg/pull/17280#issuecomment-5647067753
@talatuyarer I looked into `FullCachingLookupProvider`, but I still recommend keeping our current implementation so that we can retain the RocksDB backend for Iceberg Flink Lookup Join. In Flink 2.3, `FullCachingLookupProvider` has some limitations: 1. The FULL cache storage is hard-coded to an in-memory ConcurrentHashMap, with no extension point for a custom cache backend. https://github.com/apache/flink/blob/e01bbcacd962189c726d8324a35d66cc162712e7/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/table/lookup/fullcache/CacheLoader.java#L53 2. FULL caching currently only supports InputFormatProvider. For Iceberg, FlinkSource.Builder#buildFormat() is deprecated and planned for removal with the legacy non-FLIP-27 source. https://github.com/apache/flink/blob/e01bbcacd962189c726d8324a35d66cc162712e7/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/LookupJoinUtil.java#L576-L593 3. Using the native FULL cache would therefore prevent us from supporting RocksDB, which is important for dimension tables that do not fit in TaskManager heap memory. I also checked the Hudi and Paimon implementations. They support memory and RocksDB backends but do not use FullCachingLookupProvider either. So I suggest keeping the current implementation and aligning its behavior with FLIP-221 where possible, while retaining our pluggable cache backend. I’ll continue improving the monitoring and retry capabilities. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
