Hi Mark, Thanks for sharing, that sounds very nice :)
I think neither REST API will return the Kafka payload directly, because they don't have the same scope or the same schema owner. 1. In the Iceberg REST event endpoint ( https://github.com/apache/iceberg/pull/12584), the schema comes from the Iceberg spec, not from Polaris. It's a pull API (continuation token) that returns Iceberg "Event" objects. The fields are - case (event-id, request-id, timestamp-ns, actor), and an "operation" typed by operation-type (create-table, update-table, rename-view, drop-namespace, ...) carries the table/view uuid and the updates/requirements. It only covers committed changes to catalog objects. So there are no BEFORE_* events, no load events, and no principal/role/grant events. It's also still under review on the Iceberg side. When we implement it, Polaris has to emit that shape as is. 2. Polaris events query API (https://github.com/apache/polaris/issues/5989) reads the JDBC events table, so it only returns what the persistence listener wrote. Kafka, CloudWatch and webhook don't write there. A row is a lossy projection (resource type, dotted resource identifier, string additional properties), so we can't rebuild the full envelope from it. The current direction on the issue is to keep the response close to the row. What we can (and I think should) do is make sure the identifiers line up, so consumers can correlate and dedupe across sinks: - event_id is the Kafka body (and "id" in the envelope) is the same event id that is stored in the events table and returned. - request_id is the same everywhere, and it's also the field Iceberg Event uses to group the events of one commit. We should reuse Polaris request id and map with event id. Regards JB On Wed, Nov 11, 2026 at 10:13 AM zzcgumk via dev <[email protected]> wrote: > Hi all, > This thread ended up in my spam for some reason so sorry for the > late input. I worked on the Kafka extension. The data included in the Kafka > payload was deliberately a super set of the CloudWatch payload. The > ultimate goal was to have a single mapper producing the same event JSON for > all publishers (sorry, I did not get a chance to do this) so this all looks > good to me. > > As a consumer of these events will the proposed REST Events API [1], [2] > provide similar payloads - ie it would be nice to be able to flip between > Kafka and REST easily. > > cheers > Mark > [1] https://github.com/apache/iceberg/pull/12584 > [2] https://github.com/apache/polaris/issues/5989 > > On Tue, Oct 27, 2026 at 3:23 PM vignesh a <[email protected]> wrote: > > > Hi all, > > > > The mapper [1] is on main. > > > > I've rebased the webhook listener [2] onto it. The POST body is the > > shared envelope, and the delivery time goes in the > > X-Polaris-Delivery-Time header. [2] is the first consumer of the > > mapper and the next PR to land. > > > > Quick recap on Kafka and CloudWatch from this thread: > > > > - Kafka already ships a flat payload in 1.7 / 1.8, so that listener > > stays on the current mapper for now. The follow-up is a > > per-listener format flag: flat payload stays the default and the > > shared envelope is opt-in, so existing consumers keep working. > > > > - CloudWatch moves to the same envelope after the webhook listener. > > > > Thanks, > > *Vignesh* > > > > [1] https://github.com/apache/polaris/pull/5717 > > [2] https://github.com/apache/polaris/pull/5084 > > >
