vamossagar12 commented on PR #2881: URL: https://github.com/apache/fluss/pull/2881#issuecomment-4581392875
> @vamossagar12 Thanks for this.. I have one concen that i would like your thoughts on as well us from @luoyuxia and @MehulBatra ... Unlike Paimon, where setting `WRITE_ONLY=false` lets expiration run in-band as part of the normal commit, Iceberg's `ExpireSnapshots.commit()` is a **separate metadata commit** that: > > * Plans across manifests > * Writes a new `TableMetadata` JSON > * Synchronously deletes files (because `cleanExpiredFiles=true`) > > In a streaming tiering scenario, this happens on **every commit** with no throttling. This can materially increase per-commit latency and produce a metadata/delete storm on object stores. Could we: > > * Add an opt-in interval (e.g. `lake.iceberg.expire-snapshot.interval` defaulting to every N commits or M minutes), OR > * At minimum, document this characteristic on the config option and recommend conservative `history.expire.max-snapshot-age-ms` / `history.expire.min-snapshots-to-keep` settings? > > I don't know whether it's a blocker, but I thought to mention. Thanks @polyzos , it makes sense. Actually I went ahead with option 1 to allow ease of use for the users. I introduced 2 Iceberg-specific config options: ``` lake.iceberg.expire-snapshot.min-interval-commits (default: 10) — minimum commits since last expiration lake.iceberg.expire-snapshot.min-interval-ms (default: 10min) — minimum time elapsed since last expiration ``` Expiration only fires when both thresholds are satisfied simultaneously. This means: - In a fast-streaming scenario (e.g. 1 commit/sec), the 10-minute time gate acts as the primary rate-limiter — expiration runs at most once every 10 minutes regardless of commit frequency - In a slow scenario (e.g. 1 commit/hour), the count gate ensures we don't expire too frequently either This way the snapshotting can be controlled by the users based on commit counts or time based. Let me now what you think! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
