grantatspothero commented on PR #16207: URL: https://github.com/apache/iceberg/pull/16207#issuecomment-4381653579
> I'm wondering if the table is changed every 30sec, then is there any point storing it in a cache. Two different definitions of "cache": 1. "Within query caching metadata caching". Within a single query's lifetime, TableMetadata must live in coordinator memory. Queries are usually short but sometimes can take hours, wasting coordinator memory for hours for long running queries. This wasted memory is exacerbated by: # of concurrent queries and # of tables per query. Compare this to the hive table model where coordinator memory is mostly bounded. 2. "Cross-query metadata caching". I believe this is what you are talking about. Trino does not support cross-query table metadata caching today, but [some engines do and have problems](https://github.com/StarRocks/starrocks/issues/47630). Essentially you must disable cross-query cache entirely because it is difficult to control caching at a fine granularity. "Cache these long lived table metadatas but not these constantly changing ones" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
