Hi everyone,
Here's an update on the work done so far: We've factored
https://github.com/apache/iceberg/pull/15996 into 6 separate commits,
which have all been merged. The feature is now complete, apart from
future bug fixes and enhancements.
Equality delete conversion is exposed as a new table maintenance task,
ConvertEqualityDeletes, and is integrated with IcebergSink. It can be
run standalone alongside any engine writing equality deletes, or
directly with Flink's writer (IcebergSink).
## What it does
Writers produce data files and equality deletes, as before.
ConvertEqualityDeletes reads them, resolves the equality deletes into
deletion vectors using a primary key index, and commits the data files
and DVs. The index is kept in Flink state and backed by RocksDB, so it
spills to disk on large tables. After conversion, the target branch
contains only data files and DVs, no equality deletes.
With the IcebergSink integration, as soon as the equality deletes are
committed, we run the conversion cycle. During the conversion, we
ensure via a Flink-maintained lock that no parallel maintenance tasks
run (if any configured).
## Equality delete staging
Equality deletes can be staged in two ways:
- Using a staging branch: Writes go to a staging branch, separate from
the main branch. Equality deletes and new data is read from the
staging branch and committed to the target branch. Readers will only
see new data once the conversion has run. The target branch is
guaranteed to be free of equality deletes.
- Same branch conversion: In addition to having a separate staging and
a target branch, we also support "in-place" conversion, where we use
the same branch for writing equality deletes and converting them. This
is the default configuration with IcebergSink; the converter runs on
the write branch instead of requiring a separate staging branch. This
allows readers to see new data immediately, even before conversion.
It's operationally simpler, but it doesn't guarantee that
merge-on-read cannot happen for new snapshots which have not been
freed of equality deletes.
## Example usage
### As a standalone maintenance task
TableMaintenance.forTable(env, tableLoader).add(
ConvertEqualityDeletes.builder()
.stagingBranch("staging")
.targetBranch("main")
.equalityFieldColumns(List.of("id"))
.scheduleOnEqDeleteFileCount(1));
Also see
https://iceberg.apache.org/docs/nightly/flink-maintenance/#convertequalitydeletes
### With the IcebergSink
IcebergSink.forRowData(stream)
.upsert(true)
.equalityFieldColumns(List.of("id"))
.convertEqualityDeletes()
.append();
Also see
https://iceberg.apache.org/docs/nightly/flink-writes/#post-commit-table-maintenance
### Via the config or SQL
Set `flink-maintenance.convert-equality-deletes.enabled = true`.
Also see
https://iceberg.apache.org/docs/nightly/flink-maintenance/#convertequalitydeletes-configuration
## Open items
The implementation is fully functional today. We decided to defer
these to after the initial milestone:
- Incremental re-indexing of external commits: Today, the index is
rebuilt fully when an external commit lands on the target branch, e.g.
table compaction or multiple writers. It would be better to refresh
the index using a delta, instead of rebuilding it from scratch. For
the initial implementation, we wanted to keep things as simple as
possible. I will put up a PR to address this soon.
- Row lineage is not yet preserved through the conversion. We are
going to fix this soon.
The long term the goal is unchanged: deprecate equality deletes in V4,
and once the spec has an index, persist the PK index into Iceberg so
it can be shared across engines.
Please let me know your thoughts.
Thanks,
Max
On Fri, Apr 17, 2026 at 3:58 PM Maximilian Michels <[email protected]> wrote:
>
> Hi Steven,
>
> Thanks for chiming in! Yes, the current solution requires V3 tables.
> I'll probably have to make that clearer in the PR description.
>
> In principle, it wouldn't be hard to make it work with V2 either,
> obviously with the tradeoffs that come along with it in terms of
> storage / lookup efficiency.
>
> Cheers,
> Max
>
> On Thu, Apr 16, 2026 at 5:30 PM Steven Wu <[email protected]> wrote:
> >
> > Sorry for the delayed response. I agree that this proposal could get us one
> > step closer to removing equality deletes. It can also verify whether the
> > index solution can scale for streaming CDC/Upsert writes.
> >
> > > For this PR, we opted to use deletion vectors over regular delete files
> > > due to their efficiency in terms of space and lookup.
> >
> > I didn't quite get this from the PR description. Does this mean it is
> > limited to V3 tables?
> >
> > Thanks for working on this, Max!
> >
> > On Thu, Apr 16, 2026 at 8:03 AM Maximilian Michels <[email protected]> wrote:
> >>
> >> Thanks everyone for the discussion and support.
> >>
> >> I've opened a PR which implements what we discussed here:
> >> https://github.com/apache/iceberg/pull/15996
> >>
> >> Just to cap:
> >>
> >> The EqualityDeleteConverter (EDC) will be running in-line with the
> >> writer, which produces the equality deletes to a staging branch. We
> >> monitor the staging branch for new commits, build a sharded primary
> >> key index in Flink state (backed by RocksDB for large tables), resolve
> >> equality deletes against the index, and commit back the resulting data
> >> files + DVs to the main branch.
> >>
> >> The PR is split into several commits, which we can break out into
> >> separate PRs for easier review. There are some limitations and
> >> follow-ups listed in the PR. The biggest gaps are preserving row
> >> lineage and lifecycle management of the staging branch. In a
> >> follow-up, we will also add integration with the Flink IcebergSink. I
> >> tried to keep the scope limited to the EDC maintenance task for the
> >> first PR.
> >>
> >> Cheers,
> >> Max
> >>
> >> On Thu, Apr 2, 2026 at 9:43 AM Márton Balassi <[email protected]> wrote:
> >> >
> >> > Thanks for raising this, Max and for the feedback Peter and Manu.
> >> >
> >> > I am supportive of this proposal, especially with the clearly defined
> >> > vision of eventually completely removing the need for equality deletes.
> >> >
> >> > Lifting the reliance on equality deletes in the Flink write path would
> >> > be a significant improvement, both in terms of read efficiency (by
> >> > moving towards delete vectors) and in terms of new capabilities, as it
> >> > would make writing upserts from Flink a viable path to explore going
> >> > forward.
> >> >
> >> > Cheers,
> >> > Marton
> >> >
> >> > On 2026/03/20 11:15:26 Maximilian Michels wrote:
> >> > > Thanks Peter and Manu for the feedback.
> >> > >
> >> > > @Peter:
> >> > >
> >> > > Good point on the end goal. The end goal should be to completely
> >> > > remove equality deletes.
> >> > >
> >> > > While the staging branch, which contains the equality deletes, is an
> >> > > internal implementation detail of the Flink writer, it will still be
> >> > > accessible via the Iceberg reader API. For the transition period, I
> >> > > think this has several advantages:
> >> > > 1. We don't need to fundamentally change the write logic of existing
> >> > > writers.
> >> > > 2. We still allow for the data to be inspected before converting it
> >> > > and merging it to the main branch. This is also helpful for
> >> > > troubleshooting.
> >> > >
> >> > > The staging branch solution is a first step towards removing equality
> >> > > deletes.
> >> > >
> >> > > In V4, we could already deprecate equality deletes. Once the spec
> >> > > includes indices, we can move the index into Iceberg, which should
> >> > > make it easier to develop an in-place resolution of equality deletes
> >> > > supporting multiple writers and conflict resolution. Admittedly, we
> >> > > haven't fully figured out the best in-place approach. I think it is a
> >> > > good idea to take it one step at a time.
> >> > >
> >> > > On row lineage: If we want to preserve the row id of updated rows, we
> >> > > will have to store the row id in the primary key index. Theoretically,
> >> > > we should be able to then add it to the corresponding new row. The
> >> > > question is how to do that efficiently, such that we don't have to
> >> > > rewrite any data files. We would need some way to map the row id of
> >> > > the newly inserted row to the row id of the deleted row. Do we already
> >> > > have such functionality in Iceberg?
> >> > >
> >> > > On concurrent writes: For the time being, I think we should not allow
> >> > > concurrent maintenance tasks, including equality delete conversion.
> >> > > Concurrent writes are still supported, as long as they go to the
> >> > > staging branch.
> >> > >
> >> > > @Manu:
> >> > >
> >> > > +1 to Peter's response. The primary key index is bounded and
> >> > > independent of the number of accumulated equality deletes, so memory
> >> > > doesn't blow up, as long as we have sufficient resources to load the
> >> > > index. We definitely cannot rely on the full index to fit into memory.
> >> > > Fortunately, Flink is already prepared for this; it supports spilling
> >> > > to disk via its RocksDB state backend.
> >> > >
> >> > > Cheers,
> >> > > Max
> >> > >
> >> > > On Fri, Mar 20, 2026 at 11:07 AM Péter Váry
> >> > > <[email protected]> wrote:
> >> > > >
> >> > > > Equality delete resolution could be made significantly more
> >> > > > efficient by using an index (e.g., backed by RocksDB) to store the
> >> > > > current mapping from primary keys to (file, position). While the
> >> > > > memory footprint would not be small, it would be bounded and
> >> > > > independent of the number of accumulated equality deletes. In
> >> > > > addition, even a blocking compaction should block for a shorter
> >> > > > period than the typical interval at which table compactions are
> >> > > > scheduled.
> >> > > >
> >> > > > Manu Zhang <[email protected]> ezt írta (időpont: 2026. márc.
> >> > > > 19., Cs, 15:57):
> >> > > >>
> >> > > >> Thanks Max for the proposal. One question here.
> >> > > >> When the convert task can not finish in time (e.g. blocked by
> >> > > >> compactions), and equality deletes accumulate on the staging
> >> > > >> branch, will we have the same issue as loading too many equality
> >> > > >> deletes and blowing up memory?
> >> > > >>
> >> > > >> Regards,
> >> > > >> Manu
> >> > > >>
> >> > > >> On Thu, Mar 19, 2026 at 2:45 PM Péter Váry
> >> > > >> <[email protected]> wrote:
> >> > > >>>
> >> > > >>> Thanks, Max, for continuing to push this forward.
> >> > > >>>
> >> > > >>> The proposal feels like a step in the right direction, but I would
> >> > > >>> like to see a clearer view of the end goal. As it stands, equality
> >> > > >>> deletes remain in the spec because the changes are committed to an
> >> > > >>> intermediate branch. Since the long‑term objective is to remove
> >> > > >>> equality deletes from the specification altogether, we should be
> >> > > >>> clear about the final solution that achieves this.
> >> > > >>>
> >> > > >>> Flink writes will also continue to have the limitation that row
> >> > > >>> lineage is not maintained correctly. This is unchanged from the
> >> > > >>> current situation, but I think it’s important to explicitly call
> >> > > >>> this out, or ideally, explore whether there’s a way to address it.
> >> > > >>>
> >> > > >>> In addition, concurrent writes and compactions would require
> >> > > >>> updating the primary key index, which could be expensive.
> >> > > >>>
> >> > > >>> That said, I don’t see a clearly better alternative at the moment,
> >> > > >>> and overall this seems like a reasonable way forward.
> >> > > >>>
> >> > > >>> Thanks again for continuing to drive the proposal.
> >> > > >>> Peter
> >> > > >>>
> >> > > >>>
> >> > > >>> On Wed, Mar 18, 2026, 16:47 Maximilian Michels <[email protected]>
> >> > > >>> wrote:
> >> > > >>>>
> >> > > >>>> Hi,
> >> > > >>>>
> >> > > >>>> I'd like to discuss resolving equality deletes in the Flink write
> >> > > >>>> path, which will get us one step closer to removing equality
> >> > > >>>> deletes
> >> > > >>>> from the spec.
> >> > > >>>>
> >> > > >>>> ## tl;dr
> >> > > >>>>
> >> > > >>>> We're planning to add an equality delete to deletion vector (DV)
> >> > > >>>> conversion to Flink. Equality deletes may remain as an internal
> >> > > >>>> intermediary format.
> >> > > >>>>
> >> > > >>>> ## Background
> >> > > >>>>
> >> > > >>>> For deletes, Flink currently produces equality delete files.
> >> > > >>>>
> >> > > >>>> Equality deletes are used to support deletes in the write path,
> >> > > >>>> which
> >> > > >>>> is a requirement for many use cases like CDC [3]. They are cheap
> >> > > >>>> for
> >> > > >>>> the writer; it only notes down the to-be-deleted values of the
> >> > > >>>> identifier fields inside so-called delete files, and leaves it up
> >> > > >>>> for
> >> > > >>>> the readers to match the values to the corresponding rows. The
> >> > > >>>> heavy
> >> > > >>>> lifting has to be done by the readers, which potentially need to
> >> > > >>>> scan
> >> > > >>>> the entire table to resolve equality deletes.
> >> > > >>>>
> >> > > >>>> Therefore, equality deletes have been criticized. There are
> >> > > >>>> discussions around deprecating / removing them [1].
> >> > > >>>>
> >> > > >>>> ## Resolving Equality Deletes
> >> > > >>>>
> >> > > >>>> Steven, Peter, and a few other contributors came up with a
> >> > > >>>> proposal to
> >> > > >>>> convert equality deletes into DV [2]. The original solution was
> >> > > >>>> quite
> >> > > >>>> complex, mainly due to the conflict handling between streaming
> >> > > >>>> writes,
> >> > > >>>> table maintenance, and equality delete resolution. The proposal is
> >> > > >>>> also blocked on index support in the Iceberg spec [5].
> >> > > >>>>
> >> > > >>>> We may need to simplify further to make some progress. The old
> >> > > >>>> table
> >> > > >>>> specs are going to be around for some time, even after we have a
> >> > > >>>> new
> >> > > >>>> spec with index support. Users have been asking for a solution to
> >> > > >>>> this
> >> > > >>>> issue for quite some time [3].
> >> > > >>>>
> >> > > >>>> The following is a modification of the original design document
> >> > > >>>> which
> >> > > >>>> adapts the ideas described under "use lock to avoid conflicts"
> >> > > >>>> [2].
> >> > > >>>>
> >> > > >>>> ## Proposed Solution
> >> > > >>>>
> >> > > >>>> The idea is to add the equality delete to deletion vector (DV)
> >> > > >>>> conversion as a Flink table maintenance task. After recent
> >> > > >>>> changes, we
> >> > > >>>> can now run the writer and the maintenance in the same Flink job
> >> > > >>>> and
> >> > > >>>> use a Flink-maintained lock to avoid conflicts between the
> >> > > >>>> maintenance
> >> > > >>>> tasks.
> >> > > >>>>
> >> > > >>>> 1. Instead of writing directly to the target branch, the writer
> >> > > >>>> commits data files + equality deletes to a staging branch.
> >> > > >>>> 2. The new "EqualityDeleteResolver" maintenance task reads from
> >> > > >>>> the
> >> > > >>>> staging branch and converts the equality deletes to DVs using a
> >> > > >>>> Flink-maintained primary key index, then commits data files + DVs
> >> > > >>>> to
> >> > > >>>> the target branch.
> >> > > >>>> 3. The existing Flink maintenance framework's lock mechanism
> >> > > >>>> ensures
> >> > > >>>> mutual exclusion between the convert task and table compaction to
> >> > > >>>> avoid conflicts.
> >> > > >>>>
> >> > > >>>> After conversion, the target branch contains only data files and
> >> > > >>>> DVs,
> >> > > >>>> no equality deletes.
> >> > > >>>>
> >> > > >>>> ## Limitations
> >> > > >>>>
> >> > > >>>> - Readers will only see new data until the conversion is complete.
> >> > > >>>> This is partially mitigated by the fact that snapshots with
> >> > > >>>> equality
> >> > > >>>> deletes cannot be read properly with Flink today [4].
> >> > > >>>> - The Flink-maintained index needs to be built initially which
> >> > > >>>> requires reading the entire table. We will use Flink's state
> >> > > >>>> backend
> >> > > >>>> which apart from heap-based storage, also supports spilling to
> >> > > >>>> disk
> >> > > >>>> via the RocksDB state backend.
> >> > > >>>>
> >> > > >>>> ## Wrapping up
> >> > > >>>>
> >> > > >>>> This solution may not be perfect because of the above
> >> > > >>>> limitations, but
> >> > > >>>> it provides a viable path to free users of the burden of equality
> >> > > >>>> deletes, which cannot be read efficiently by most engines today.
> >> > > >>>> Eventually, the Flink-maintained index can be replaced by an
> >> > > >>>> Iceberg
> >> > > >>>> index, which will allow for the index to be shared across engines.
> >> > > >>>>
> >> > > >>>> What does the community think?
> >> > > >>>>
> >> > > >>>> Thanks,
> >> > > >>>> Max
> >> > > >>>>
> >> > > >>>> [1] Deprecate equality deletes:
> >> > > >>>> https://lists.apache.org/thread/z0gvco6hn2bpgngvk4h6xqrnw8b32sw6
> >> > > >>>> [2] Design doc:
> >> > > >>>> https://docs.google.com/document/d/1Jz4Fjt-6jRmwqbgHX_u0ohuyTB9ytDzfslS7lYraIjk/edit
> >> > > >>>> [3] Upserts use case:
> >> > > >>>> https://lists.apache.org/thread/rt7dmg7l78xpzc9w3lwn090yzqq4fyyw
> >> > > >>>> [4] Handling upserts downstream:
> >> > > >>>> https://lists.apache.org/thread/bx1ntfr45c0g9rh643yw7w8znv6wtrno
> >> > > >>>> [5] V4 Index:
> >> > > >>>> https://lists.apache.org/thread/xdkzllzt4p3tvcd3ft4t7jsvyvztr41j
> >> > >
