grantatspothero commented on PR #16207: URL: https://github.com/apache/iceberg/pull/16207#issuecomment-4373619730
> Above that amount of snapshots you're doomed anyway :) It is becoming more common to have larger snapshot retention in iceberg due to prevalence of streaming ingestion/low latency commits. See mailing list discussions: https://www.mail-archive.com/[email protected]/msg12764.html Examples: kafka-connect iceberg sink, Confluent Tableflow, Starburst streaming ingestion. This doesn't solve the full problem mentioned in that mailing list thread (writes still pay the full cost of writing snapshots/snapshotLog), but it does solve the problem for readers. And for query engine/caching usecases, reads >> writes so this could be beneficial. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
