I think this makes sense to me. Do we have latency difference numbers for some typical cases between the modes? Or is there a way we can trigger merging heuristically based on the number of small manifests? I'm wondering if setting that as the default is right vs just switching behaviors automatically when it makes sense.
On Thu, Jul 30, 2026 at 12:02 PM Steve <[email protected]> wrote: > Thanks Amogh for the changes and discussion. > > +1 to change default in 1.13 release so Spark streaming can switch to use > merge append by default like Flink and Kafka connect. > > Thanks, > Hongyue Zhang > > On Thu, Jul 30, 2026 at 8:14 AM Amogh Jahagirdar <[email protected]> wrote: > >> Hey folks, >> >> For context, Spark Streaming has been using the Iceberg Java library's >> fast append, which produces a single new manifest pointing to all the new >> files, without binpacking manifests. The current format's fast append >> accumulates many tiny manifests in streaming cases , and require aggressive >> maintenance (manifest rewriting) so that reads don't degrade. In V4, we're >> improving this so that we achieve true low latency small commits without >> the metadata bloat. >> >> Flink/Kafka Connect use the merging append on their path, because for the >> streaming ingestion workloads that the format can reasonably handle today, >> it makes sense to periodically merge the manifests on write (which isn't >> *that* expensive). One can also reason about this as the metadata analog >> for why it makes sense to merge DVs on write, rather than have potentially >> many position deletes paid at read time. >> >> We recently made changes <https://github.com/apache/iceberg/pull/17403> >> to add a spark write conf to change this append option for streaming, but >> it defaults to the existing fast append behavior just to make sure in 1.12 >> when folks upgrade there's no unexpected regression or if for whatever >> reason clients rely on this single manifest per commit assumption, they >> have time to adjust their logic. >> >> It's really not a good default option, so I propose that in 1.13 we flip >> this flag so that new writers start using the merging append. >> >> I would also recommend that any other client libraries that are using >> fast appends as a default re-evaluate their usage of it. >> >> Thanks, >> Amogh Jahagirdar >> >
