Hey folks, For context, Spark Streaming has been using the Iceberg Java library's fast append, which produces a single new manifest pointing to all the new files, without binpacking manifests. The current format's fast append accumulates many tiny manifests in streaming cases , and require aggressive maintenance (manifest rewriting) so that reads don't degrade. In V4, we're improving this so that we achieve true low latency small commits without the metadata bloat.
Flink/Kafka Connect use the merging append on their path, because for the streaming ingestion workloads that the format can reasonably handle today, it makes sense to periodically merge the manifests on write (which isn't *that* expensive). One can also reason about this as the metadata analog for why it makes sense to merge DVs on write, rather than have potentially many position deletes paid at read time. We recently made changes <https://github.com/apache/iceberg/pull/17403> to add a spark write conf to change this append option for streaming, but it defaults to the existing fast append behavior just to make sure in 1.12 when folks upgrade there's no unexpected regression or if for whatever reason clients rely on this single manifest per commit assumption, they have time to adjust their logic. It's really not a good default option, so I propose that in 1.13 we flip this flag so that new writers start using the merging append. I would also recommend that any other client libraries that are using fast appends as a default re-evaluate their usage of it. Thanks, Amogh Jahagirdar
