Hi Dmitri, Yes, this is indeed a bug and we should create a GitHub issue for this to track fixing it. Ideally, we should be doing something similar to Option 1 of what Yufei suggested - but how this will be done may be harder than it looks. I can investigate this and I encourage others to look into it as well, if this is of interest.
Best, Adnan Hemani On Wed, Nov 5, 2025 at 8:29 AM Yufei Gu <[email protected]> wrote: > Hi Dmitri, > > Good catch! This is a subtle but important issue. Thanks for raising it. I > think we could handle it in a few ways: > > 1. Hold the event emission until the all multi-table commit succeeds, and > buffering per-table events until persistence confirms success. > 2. Include a transaction ID and status (e.g., pending, committed, aborted) > in emitted events so consumers can filter accordingly. This will add burden > to downstreams. I think we could figure out a way to filter out while > persisting them, so that the real consumers won't see the events with > aborted status. > > I'm not sure which way is better at this moment, we need to take a deep > look to evaluate both. > > Yufei > > > On Wed, Nov 5, 2025 at 8:06 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi All, > > > > I'd like to highlight an aspect of the current Events behaviour with > > respect to multi-table commits that Christopher and I discovered today. > The > > issue existed since the first Events implementation, AFAIK, it just did > not > > come into focus until now, AFAIK. > > > > Consider a multi-table commit request [1]. IcebergCatalogHandler will run > > each individual table through a separate commit operation on the base > > catalog [2]. The base catalog will issue events for each table separately > > [3][4]. However, the overall commit to Polaris Persistence may still > fail, > > e.g. due to concurrent updates [5]. > > > > Now, we can have a situation when both before/after events are delivered > > for a table, but the actual change that triggered the events is _not_ > > persisted, therefore does not exist in the current state of the Polaris > > catalog. > > > > Thoughts? > > > > [1] > > > > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L973 > > > > [2] > > > > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L1051 > > > > [3] > > > > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java#L1405 > > > > [4] > > > > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java#L1558 > > > > [5] > > > > > https://github.com/apache/polaris/blob/f934443114251f85d18c9a51ed61fc49a500a61a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java#L1058 > > > > Cheers, > > Dmitri. > > >
