grantatspothero commented on PR #16207: URL: https://github.com/apache/iceberg/pull/16207#issuecomment-4398312431
> that the intention is to have effect on the list of snapshots returned by the REST catalog and not on the snapshot log Under this definition I agree this PR changes the behavior of REFS mode and that is not good. That said, the spec is ambiguous as written given different catalog implementations are already interpreting it differently. Taking a step back, what was the purpose of introducing REFS mode? I thought it was introduced as a performance optimization to allow clients to have a slim representation of TableMetadata for common operations while falling back to the full representation for less common operations. This PR seems to fit into that goal of REFS mode, but maybe REFS mode was introduced for a different reason? > Taking one step back, what I still don't entirely get is this: @grantatspothero you mentioned that you'd like to reduce memory pressure by not keeping entire snapshot log in REFS mode. Could you help me understand the full picture how you try to achieve that? This PR doesn't help you with that. Consider a multitenant coordinator service (ie: a query engine like trino). If the coordinator service does a read only query that doesn't access historical snapshots, then loading a slim version of table metadata in the client is enough to satisfy query planning entirely. The full table metadata must only be pulled into the client for read queries using time travel or write queries. Both these types of queries happen significantly less often than read only queries over the current snapshot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
