Hi all, I'd like to restart the conversation about deprecating equality deletes, now in the context of the V4 spec.
*Background* This isn't a new idea. Russell proposed deprecating equality deletes in V3 and removing them from the spec in V4, back in October 2024 in "[DISCUSS] - Deprecate Equality Deletes <https://www.mail-archive.com/[email protected]/msg08097.html>". The main blocker at the time was that equality deletes served real use cases (especially Flink streaming upserts) with no efficient alternative. Two developments since then make the V4 removal worth acting on now. *Why equality deletes are costly* Equality deletes are cheap to write but expensive to read: a reader must load the equality-delete files and join them against every candidate row in the delete's sequence-number range. Positional deletes skip that per-row join by marking exact positions, so they have always read faster, and V3 deletion vectors make them faster still, one compact bitmap per data file, applied by an O(1) position check, instead of V2's many position-delete files. So equality deletes' only real edge is the cheap write, and both background conversion and a write-time key-lookup index can recover that. *Beyond performance* Equality deletes also block other features. CDC and row lineage are effectively impossible while they are in use, because the true state of the table can only be determined with a full scan. That same property means differential structures such as materialized views and secondary indexes have to be fully rebuilt whenever an equality delete is added, rather than maintained incrementally. So removing equality deletes is close to a prerequisite for the index work to stay incrementally maintainable. *Evidence the alternatives are practical* 1. Converting equality deletes to DVs works today. Max Michels' ConvertEqualityDeletes maintenance task (16831 <https://github.com/apache/iceberg/pull/16831>, 16844 <https://github.com/apache/iceberg/pull/16844>, 16858 <https://github.com/apache/iceberg/pull/16858>, 16874 <https://github.com/apache/iceberg/pull/16874>, 16889 <https://github.com/apache/iceberg/pull/16889>, 16948 <https://github.com/apache/iceberg/pull/16948>) rewrites equality deletes into deletion vectors as a background Flink job: the writer keeps appending equality deletes to a staging branch, and the task converts them to DVs on the target branch so reads apply deletes by position. Notably, the task resolves each delete to a position using a primary-key index that it builds and maintains inside the job, demonstrating the full "key -> position -> DV" path end to end. 2. A persistent key-lookup index removes the need to write them at all. The secondary index spec we're working on (#16961 <https://github.com/apache/iceberg/pull/16961>) includes a key-lookup index mapping a key to its data file and row position. This is essentially the persistent, catalog-managed form of the index Max's task builds ephemerally. With it, a writer can resolve positions at write time and emit DVs directly, without ever producing an equality delete. *How these two efforts fit together* They're complementary, and they cover the two things we need to deprecate equality deletes: - Migration (existing data): ConvertEqualityDeletes cleans up tables that already contain equality deletes, and supports writers that still emit them, converting them to DVs in the background. - Going forward (new writes): the persistent key-lookup index lets writers skip equality deletes entirely by looking up positions directly. - The connection is that Max's task already proves the core mechanism (resolve key -> position, write a DV); it just rebuilds a throwaway index each cycle. A durable, shared index both enables write-time elimination and removes that rebuild cost from the conversion path. *Proposal* I propose that we deprecate equality deletes in V4. The blocker from 2024 was the lack of a viable alternative, and we now have the pieces: background conversion to DVs works today, and the key-lookup index gives us a path to eliminating them at write time. Deletion vectors should be the going-forward mechanism for row-level deletes and upserts, produced by background conversion now and directly by writers once the index is available. Readers would continue to support equality deletes for backward compatibility with existing V2/V3 tables. *Migration path* - Existing tables keep working; readers continue to apply equality deletes. - ConvertEqualityDeletes (Flink) rewrites existing equality deletes into DVs so tables can be cleared of them over time. I'd love people's thoughts, especially from those running large streaming-upsert workloads. Thanks, Huaxin
