gaborkaszab commented on PR #14440: URL: https://github.com/apache/iceberg/pull/14440#issuecomment-4619627823
Hi @moomindani , @blcksrx , Thank you for putting efforts into this! Just some thoughts from me: > already has directional buy-in I'd say that this isn't entirely true. The change in theory could make sense, what I think the community misses, is a real world use-case to demonstrate this hurts anyone. I think, based on the discussion so far, this is kind of a speculative addition, might help someone somewhere. If you have more in-production details, would you mind sharing for further justification? As one example, long running queries were mentioned, where frequent access to the cache kept the table alive even if stale. Wouldn't issuing a refresh() solve this problem? It doesn't go through the cache, so won't give stale table state, and as in nature it refreshes the TableMetadata in-place within the TableOperations, practically refreshing the table metadata in the cache too (not sure if this holds risk with race-conditions on multiple threads holding the same table object, but this is as is for a long while now, no one complained). I took a glance on the measurements shared. This confuses me even more of the motivation here. The measurements are around average time to load a table, but the pain point here is table state staleness, right? Is this to show that `expireAfterWrite` is not much worse than `expireAfterAccess` but both are way better than turning the cache off? > Next step is a committer review pass — @gaborkaszab @ajantha-bhat None of us are committers. To get a strong buy-in, I'd go to dev@ list to get wide audience, and in case committers/PMCs show interest, I can help with the reviewing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
