Thanks Amogh for the changes and discussion. +1 to change default in 1.13 release so Spark streaming can switch to use merge append by default like Flink and Kafka connect.
Thanks, Hongyue Zhang On Thu, Jul 30, 2026 at 8:14 AM Amogh Jahagirdar <[email protected]> wrote: > Hey folks, > > For context, Spark Streaming has been using the Iceberg Java library's > fast append, which produces a single new manifest pointing to all the new > files, without binpacking manifests. The current format's fast append > accumulates many tiny manifests in streaming cases , and require aggressive > maintenance (manifest rewriting) so that reads don't degrade. In V4, we're > improving this so that we achieve true low latency small commits without > the metadata bloat. > > Flink/Kafka Connect use the merging append on their path, because for the > streaming ingestion workloads that the format can reasonably handle today, > it makes sense to periodically merge the manifests on write (which isn't > *that* expensive). One can also reason about this as the metadata analog > for why it makes sense to merge DVs on write, rather than have potentially > many position deletes paid at read time. > > We recently made changes <https://github.com/apache/iceberg/pull/17403> > to add a spark write conf to change this append option for streaming, but > it defaults to the existing fast append behavior just to make sure in 1.12 > when folks upgrade there's no unexpected regression or if for whatever > reason clients rely on this single manifest per commit assumption, they > have time to adjust their logic. > > It's really not a good default option, so I propose that in 1.13 we flip > this flag so that new writers start using the merging append. > > I would also recommend that any other client libraries that are using fast > appends as a default re-evaluate their usage of it. > > Thanks, > Amogh Jahagirdar >
