Conceptually +1 as equality deletes really complicates the format and implementation.
However given the concern is around performance side. Is there a way to make the decision making more data driven — having some benchmarking on perf comparison between dv+single file commit in v4 vs equality delete in v3 could help making a call. Thanks Xin On Wed, Jul 15, 2026 at 11:01 PM Maximilian Michels <[email protected]> wrote: > I understood "deprecate equality deletes" as not forbidding engines to > write them, but rather discouraging them. IMHO this is long overdue, > but it is also a very soft transition. Perhaps too soft, because it > doesn't give engines who write them a real incentive to stop writing > equality deletes. > > Engines will likely be quicker to move away from writing equality > deletes if we disallow writing them in V4. Regardless, we will have to > support reading equality deletes for V2 and V3 tables. > > I'm leaning more towards removing equality deletes for V4 tables, but > I would like to hear what others think. > > -Max > > > On Wed, Jul 15, 2026 at 10:03 PM Steven Wu <[email protected]> wrote: > > > > > So I'd separate two things: deprecating in V4 (signal and direction, > safe to do now) versus forbidding equality-delete writes (gated on the > engine-agnostic path being ready). I'm only proposing the first for V4. > > > > I thought we wanted to forbid equality-delete writes for v4 tables, > which would really simplify the v4 adaptive metadata tree along with other > benefits that Huaxin already outlined, > > > > > deprecation path in v4 > > > > I heard the Kafka connector in the Iceberg repo doesn't produce equality > deletes. We would need to migrate the Flink sink to leverage the index to > produce DVs only in v4. > > > > On Wed, Jul 15, 2026 at 12:13 PM huaxin gao <[email protected]> > wrote: > >> > >> Thanks Max and Manu. > >> > >> Max, thanks for the added detail. It's a good point that the index in > ConvertEqualityDeletes is persisted in Flink state (RocksDB) and updated > incrementally. That strengthens the case, since it shows the key to > position index is already durable, just scoped to Flink today. And your > closing point is exactly the plan I have in mind: deprecate equality > deletes in V4, and once the spec has an index, persist that PK index into > Iceberg so it can be shared across engines. > >> > >> Manu, good question on how this works in practice. To be clear, I'm > proposing "deprecated but permitted," not removal. In V4, writers > (including Kafka Connect) could keep emitting equality deletes and readers > would keep applying them. Deprecation just declares deletion vectors the > going-forward mechanism and stops new investment in equality deletes. So V4 > would not be gated on the index landing. > >> > >> On the cleanup path today: ConvertEqualityDeletes runs against the > table, not a specific writer, so a Kafka Connect pipeline can already be > cleaned up. It keeps writing equality deletes, and a standalone Flink > ConvertEqualityDeletes job converts them to DVs. The one friction is that > the conversion runtime is Flink today, so a Kafka-only shop would have to > run Flink just for maintenance. A Spark action would be a natural follow-up > here, since Spark is the usual Iceberg maintenance engine and most batch > shops already run it. > >> > >> Longer term, the next step is to persist the key to position index into > Iceberg. Once it's shared, engines can look up positions and write DVs > directly at write time, so a writer can stop producing equality deletes > entirely, and neither the Flink job nor a Spark action needs to rebuild the > index each run. Each engine (Kafka Connect, Spark, Flink) would adopt > write-time DVs on its own schedule; until then it keeps writing equality > deletes and relies on background conversion. So V4 deprecation doesn't > require re-implementing every writer up front. > >> > >> So I'd separate two things: deprecating in V4 (signal and direction, > safe to do now) versus forbidding equality-delete writes (gated on the > engine-agnostic path being ready). I'm only proposing the first for V4. > >> > >> Thanks, > >> Huaxin > >> > >> On Wed, Jul 15, 2026 at 3:10 AM Manu Zhang <[email protected]> > wrote: > >>> > >>> Hi Huaxin, > >>> > >>> +1 for deprecating equality deletes, but how would this deprecation > work practically in V4? > >>> As Max pointed out, we still lack an engine-agnostic solution for > streaming use cases. For example, how would we handle equality deletes > written by Kafka Connect? > >>> While the index proposal looks promising, I don't see a clear path for > deprecating equality deletes in V4 before that index work actually lands. > >>> > >>> Thanks, > >>> Manu > >>> > >>> > >>> On Wed, Jul 15, 2026 at 5:30 PM Maximilian Michels <[email protected]> > wrote: > >>>> > >>>> Hi Huaxin, > >>>> > >>>> Thanks for reviving the discussion on deprecating equality deletes. > >>>> Equality deletes are the number one pain for streaming use cases. Many > >>>> users give up when they see the merge-on-read costs, or they build > >>>> custom solutions which move them further away from core Iceberg. That > >>>> said, we've made great progress since the initial conversation in > >>>> 2024. > >>>> > >>>> Just to add what you said: The index we maintain in > >>>> ConvertEqualityDeletes is not ephemeral. The index is persisted in > >>>> Flink's managed state (RocksDB). It is continuously updated as new > >>>> data arrives and checkpointed periodically. However, even though the > >>>> conversion works for data written by any engine, we currently require > >>>> Flink for the conversion itself. Storing the index directly in Iceberg > >>>> and enabling all engines access would be the next logical step towards > >>>> a fully engine-agnostic solution. > >>>> > >>>> The reality is that we don't yet have a working solution to avoid > >>>> writing equality deletes across all engines, but given the recent > >>>> progress, the proposed plan seems realistic. So +1 for deprecating > >>>> equality deletes in V4. > >>>> > >>>> Cheers, > >>>> Max > >>>> > >>>> > >>>> > >>>> On Tue, Jul 14, 2026 at 3:25 AM huaxin gao <[email protected]> > wrote: > >>>> > > >>>> > Hi all, > >>>> > > >>>> > I'd like to restart the conversation about deprecating equality > deletes, now in the context of the V4 spec. > >>>> > > >>>> > Background > >>>> > > >>>> > This isn't a new idea. Russell proposed deprecating equality > deletes in V3 and removing them from the spec in V4, back in October 2024 > in "[DISCUSS] - Deprecate Equality Deletes". The main blocker at the time > was that equality deletes served real use cases (especially Flink streaming > upserts) with no efficient alternative. Two developments since then make > the V4 removal worth acting on now. > >>>> > > >>>> > Why equality deletes are costly > >>>> > > >>>> > Equality deletes are cheap to write but expensive to read: a reader > must load the equality-delete files and join them against every candidate > row in the delete's sequence-number range. Positional deletes skip that > per-row join by marking exact positions, so they have always read faster, > and V3 deletion vectors make them faster still, one compact bitmap per data > file, applied by an O(1) position check, instead of V2's many > position-delete files. So equality deletes' only real edge is the cheap > write, and both background conversion and a write-time key-lookup index can > recover that. > >>>> > > >>>> > Beyond performance > >>>> > > >>>> > Equality deletes also block other features. CDC and row lineage are > effectively impossible while they are in use, because the true state of the > table can only be determined with a full scan. That same property means > differential structures such as materialized views and secondary indexes > have to be fully rebuilt whenever an equality delete is added, rather than > maintained incrementally. So removing equality deletes is close to a > prerequisite for the index work to stay incrementally maintainable. > >>>> > > >>>> > Evidence the alternatives are practical > >>>> > > >>>> > 1. Converting equality deletes to DVs works today. Max Michels' > ConvertEqualityDeletes maintenance task (16831, 16844, 16858, 16874, 16889, > 16948) rewrites equality deletes into deletion vectors as a background > Flink job: the writer keeps appending equality deletes to a staging branch, > and the task converts them to DVs on the target branch so reads apply > deletes by position. Notably, the task resolves each delete to a position > using a primary-key index that it builds and maintains inside the job, > demonstrating the full "key -> position -> DV" path end to end. > >>>> > > >>>> > 2. A persistent key-lookup index removes the need to write them at > all. The secondary index spec we're working on (#16961) includes a > key-lookup index mapping a key to its data file and row position. This is > essentially the persistent, catalog-managed form of the index Max's task > builds ephemerally. With it, a writer can resolve positions at write time > and emit DVs directly, without ever producing an equality delete. > >>>> > > >>>> > How these two efforts fit together > >>>> > > >>>> > They're complementary, and they cover the two things we need to > deprecate equality deletes: > >>>> > > >>>> > Migration (existing data): ConvertEqualityDeletes cleans up tables > that already contain equality deletes, and supports writers that still emit > them, converting them to DVs in the background. > >>>> > Going forward (new writes): the persistent key-lookup index lets > writers skip equality deletes entirely by looking up positions directly. > >>>> > The connection is that Max's task already proves the core mechanism > (resolve key -> position, write a DV); it just rebuilds a throwaway index > each cycle. A durable, shared index both enables write-time elimination and > removes that rebuild cost from the conversion path. > >>>> > > >>>> > > >>>> > Proposal > >>>> > > >>>> > I propose that we deprecate equality deletes in V4. The blocker > from 2024 was the lack of a viable alternative, and we now have the pieces: > background conversion to DVs works today, and the key-lookup index gives us > a path to eliminating them at write time. Deletion vectors should be the > going-forward mechanism for row-level deletes and upserts, produced by > background conversion now and directly by writers once the index is > available. Readers would continue to support equality deletes for backward > compatibility with existing V2/V3 tables. > >>>> > > >>>> > Migration path > >>>> > > >>>> > Existing tables keep working; readers continue to apply equality > deletes. > >>>> > ConvertEqualityDeletes (Flink) rewrites existing equality deletes > into DVs so tables can be cleared of them over time. > >>>> > > >>>> > > >>>> > I'd love people's thoughts, especially from those running large > streaming-upsert workloads. > >>>> > > >>>> > Thanks, > >>>> > Huaxin >
