polyzos commented on PR #2881: URL: https://github.com/apache/fluss/pull/2881#issuecomment-4532627129
@vamossagar12 Thanks for this.. I have one concen that i would like your thoughts on as well us from @luoyuxia and @MehulBatra ... Unlike Paimon, where setting `WRITE_ONLY=false` lets expiration run in-band as part of the normal commit, Iceberg's `ExpireSnapshots.commit()` is a **separate metadata commit** that: - Plans across manifests - Writes a new `TableMetadata` JSON - Synchronously deletes files (because `cleanExpiredFiles=true`) In a streaming tiering scenario, this happens on **every commit** with no throttling. This can materially increase per-commit latency and produce a metadata/delete storm on object stores. Could we: - Add an opt-in interval (e.g. `lake.iceberg.expire-snapshot.interval` defaulting to every N commits or M minutes), OR - At minimum, document this characteristic on the config option and recommend conservative `history.expire.max-snapshot-age-ms` / `history.expire.min-snapshots-to-keep` settings? I don't know whether it's a blocker, but I thought to mention. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
