Guosmilesmile commented on PR #17280: URL: https://github.com/apache/iceberg/pull/17280#issuecomment-5632241614
> With Multiple slots per TaskManager, with this approach , [IcebergFullCachingLookupFunction.java](https://github.com/apache/iceberg/pull/17280/files/7c5e96f2e98e9b63c6e5befaff97fe371ba88179#diff-b82ee6900c7107147c72a039a738cae02e3c9a260afa1487f91d03681eff84ee) instance is not shared. > Already thought of that and not in scope of this PR ? Yes, we considered this. Since the cache is full-memory, the snapshot is the same across subtasks, so there is some duplicated memory. However, since the snapshot is periodically refreshed, cross-slot sharing would also require coordinating the refresh and lifecycle. I think keeping it per-subtask is simpler and more isolated. If memory becomes a concern, we can use a RocksDB-backed approach instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
