+1 in principle (non-binding). A few things I'd like to see settled before the flip, mostly around commit latency variance rather than the average.
1. Jitter, not mean latency. Merging append doesn't make every commit uniformly more expensive; it makes most commits slightly more expensive and a periodic subset much more expensive, whenever the merge threshold trips and a real rewrite happens. For a micro-batch pipeline the mean is close to irrelevant. What matters is whether a merge spike can exceed the trigger interval, because once a batch overruns you get queueing, and the pipeline recovers slowly rather than absorbing the cost locally. I agree with Russel, do we have p99 and max commit latency for a representative workload at, say, 10-30s triggers under both modes, and how far the worst spike sits above the median? 2. Correlated spikes. Related concern at the fleet level: independent pipelines that started around the same time and commit at similar intervals will tend to trip their merges in phase, concentrating load on the catalog and object store. Is it worth jittering the threshold slightly per writer (see in literature how the decorrelation jitter is implemented), or bounding the work admitted into any single commit, so the cost amortizes instead of clustering? Cheers, Gianluca ggraziadei On Thu, 30 Jul 2026, 19:11 Russell Spitzer, <[email protected]> wrote: > I think this makes sense to me. Do we have latency difference numbers for > some typical cases between the modes? Or is there a way we can trigger > merging heuristically based on the number of small manifests? I'm wondering > if setting that as the default is right vs just switching behaviors > automatically when it makes sense. > > On Thu, Jul 30, 2026 at 12:02 PM Steve <[email protected]> wrote: > >> Thanks Amogh for the changes and discussion. >> >> +1 to change default in 1.13 release so Spark streaming can switch to use >> merge append by default like Flink and Kafka connect. >> >> Thanks, >> Hongyue Zhang >> >> On Thu, Jul 30, 2026 at 8:14 AM Amogh Jahagirdar <[email protected]> >> wrote: >> >>> Hey folks, >>> >>> For context, Spark Streaming has been using the Iceberg Java library's >>> fast append, which produces a single new manifest pointing to all the new >>> files, without binpacking manifests. The current format's fast append >>> accumulates many tiny manifests in streaming cases , and require aggressive >>> maintenance (manifest rewriting) so that reads don't degrade. In V4, we're >>> improving this so that we achieve true low latency small commits without >>> the metadata bloat. >>> >>> Flink/Kafka Connect use the merging append on their path, because for >>> the streaming ingestion workloads that the format can reasonably handle >>> today, it makes sense to periodically merge the manifests on write (which >>> isn't *that* expensive). One can also reason about this as the metadata >>> analog for why it makes sense to merge DVs on write, rather than have >>> potentially many position deletes paid at read time. >>> >>> We recently made changes <https://github.com/apache/iceberg/pull/17403> >>> to add a spark write conf to change this append option for streaming, but >>> it defaults to the existing fast append behavior just to make sure in 1.12 >>> when folks upgrade there's no unexpected regression or if for whatever >>> reason clients rely on this single manifest per commit assumption, they >>> have time to adjust their logic. >>> >>> It's really not a good default option, so I propose that in 1.13 we flip >>> this flag so that new writers start using the merging append. >>> >>> I would also recommend that any other client libraries that are using >>> fast appends as a default re-evaluate their usage of it. >>> >>> Thanks, >>> Amogh Jahagirdar >>> >>
