Hi all, Prerequisite PR is up [1].
Shared CloudEvents mapper next to the events SPI in runtime/service. CE context at the top, Kafka-parity fields in data, and lowercased type (org.apache.polaris.after_create_table) as discussed. Webhook rebase on [2] can follow after this lands. Thanks *Vignesh* [1] https://github.com/apache/polaris/pull/5717 [2] https://github.com/apache/polaris/pull/5084 On Wed, 7 Oct 2026 at 02:24, vignesh a <[email protected]> wrote: > Hi Dmitri, > > Thanks - Noted. +1 on lowercasing type.. > > I'll open the prerequisite PR next (mapper next to the SPI in > runtime/service). > > Thanks > *Vignesh* > > On Tue, 6 Oct 2026 at 21:32, Dmitri Bourlatchkov <[email protected]> wrote: > >> Hi Vignesh, >> >> The proposal LGTM. >> >> Minor comment: SInce CloudEvents apparently prefer lowercase names, shall >> we lowercase the type namne (in JSON)? >> >> Re: backward compatibility for Kafka. Having a flag to send old payloads >> works for me. However, I believe this tends to prolong adoption and users >> have to face a hard change anyway (when their admin flips the switch). >> This >> may not be suitable for rolling upgrade scenarios. That said, we can >> probably debate this later, after establishing the common format. >> >> Cheers, >> Dmitri. >> >> On Tue, Oct 6, 2026 at 8:11 AM vignesh a <[email protected]> wrote: >> >> > Hi Dmitri, >> > >> > Thanks - good questions, and yes at a high level that's the direction. >> > Wanted to answer them against what JB and I already talked through, and >> > I've added the draft v1 JSON again so we're looking at the same thing. >> > >> > 1) Same JSON for all JSON-producing listeners? >> > >> > For the transport listeners in this thread (Kafka, webhook, CloudWatch), >> > yes - same CloudEvents-shaped payload is the end goal. OpenTelemetry and >> > the persistence listener keep their own representations for now; those >> > stay out of scope for the first PR. >> > >> > 2) Adhere to CloudEvents? >> > >> > Yes. v1 means CloudEvents 1.0 context attributes at the top only, and >> the >> > Polaris fields in "data". That also keeps us honest with CE rules on >> > context attrs (lowercase alphanumerics / scalars) - so things like >> > activated_roles live in "data", not as top-level CE extensions. If a >> > newer CE version becomes relevant later we can bump then.. >> > >> > 3) Migrating existing consumers that aren't 1:1 with the new shape? >> > >> > Union / dual-publish for a few releases is one way to do it. On the list >> > JB leaned toward a per-listener format option instead (today's flat >> Kafka >> > payload as the default, shared envelope opt-in), mainly so we don't >> > double volume on the topic. That felt right to me too for 1.7 / 1.8 >> > consumers - they keep working until they flip the knob. Webhook [1] can >> > land on the shared envelope from day one since it isn't released that >> > way yet. Open to other ideas if you see a cleaner path.. LMK >> > >> > 4) Dedicated SPI module? >> > >> > Also +1 (Alex mentioned the same). Kafka depending on runtime/service >> > just for the SPI is tricky, and having the mapper live in that SPI >> > module later makes a lot of sense. For the first PR I was thinking of >> > putting the shared mapper next to the existing events SPI in >> > runtime/service, then extracting SPI + mapper as a follow-up - mostly to >> > avoid getting stuck on module cycles before the envelope lands (JB had >> > flagged that). Will do the SPI split earlier if that feels better. WDYT >> ?? >> > >> > Draft v1 JSON (CE context + today's Kafka body in "data"): >> > >> > { >> > "specversion": "1.0", >> > "id": "550e8400-e29b-41d4-a716-446655440000", >> > "type": "org.apache.polaris.AFTER_CREATE_TABLE", >> > "source": "org.apache.polaris", >> > "time": "2026-10-04T12:34:56.789Z", >> > "datacontenttype": "application/json", >> > "data": { >> > "event_type": "AFTER_CREATE_TABLE", >> > "event_id": "550e8400-e29b-41d4-a716-446655440000", >> > "timestamp": "2026-10-04T12:34:56.789Z", >> > "realm_id": "default-realm", >> > "principal": "alice", >> > "activated_roles": ["catalog_admin"], >> > "request_id": "req-123", >> > "catalog_name": "sales", >> > "namespace": ["analytics"], >> > "table_name": "orders", >> > "table_identifier": ["analytics", "orders"] >> > } >> > } >> > >> > A couple notes: >> > - "data" is basically today's Kafka EventToJsonMapper body, so id/time >> > show up both as CE context and again inside data (event_id / >> > timestamp) for Kafka parity. >> > - Type is a namespaced PolarisEventType name; source is a constant for >> > now - happy to bike-shed those if needed. >> > - Webhook delivery time stays transport metadata (HTTP header), not in >> > the shared envelope. >> > - Rename events would also carry source / destination arrays in "data", >> > same as Kafka today; omitted above because this example is >> > AFTER_CREATE_TABLE. >> > >> > If this looks right, I'll open the prerequisite PR on that basis... >> > >> > Thanks >> > *Vignesh* >> > >> > [1] https://github.com/apache/polaris/pull/5084 >> > >> > On Tue, 6 Oct 2026 at 05:53, Dmitri Bourlatchkov <[email protected]> >> wrote: >> > >> > > Hi Vgnesh, >> > > >> > > Your proposal sounds good to be at high level. >> > > >> > > Just to clarify: is our end goal here to use the same JSON payload for >> > all >> > > event listeners that produce JSON? >> > > >> > > Are we going to adhere to the CloudEvents spec for this JSON? >> Obviously >> > > starting with 1.0 and migrating to newer version as the become >> > available... >> > > Is that the plan? >> > > >> > > How are we going to migrare existing use cases that are not 1:1 with >> > > the ideal JSON schema? Will be publish a union of attributes for a few >> > > releases to allow consumers to switch to the new schema? >> > > >> > > +1 to adding a dedicated SPI module. I suppose it can also contain the >> > > common java-to-JSON mapping code. >> > > >> > > Thanks, >> > > Dmitri. >> > > >> > > On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <[email protected]> wrote: >> > > >> > > > Hi all, >> > > > >> > > > +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1. >> > > > >> > > > I also welcome the idea of isolating the Events SPI in its own >> module. >> > > > Having the Kafka module depend on runtime/service just for the SPI >> is >> > > > concerning. >> > > > >> > > > Looking forward to the first PR! >> > > > >> > > > Thanks, >> > > > Alex >> > > > >> > > > On Mon, Oct 5, 2026 at 10:02 AM vignesh a <[email protected]> >> > > wrote: >> > > > > >> > > > > Hi JB, >> > > > > >> > > > > Thanks - this is really useful, and I'm with you on the sequencing >> > > > > (shared envelope first, then rebase the webhook). >> > > > > >> > > > > Your points on shape make sense to me too: >> > > > > 1. CloudEvents context attrs at the top only, Polaris payload in >> > "data" >> > > > > (so things like activated_roles stay valid) >> > > > > 2. reuse the existing EventAttributes / Kafka snake_case keys in >> > "data" >> > > > > 3. v1 at Kafka parity, not the thinner webhook set I suggested >> > earlier >> > > > > 4. mapper next to the events SPI in runtime/service for now >> > > > > 5. deliverytime as webhook HTTP metadata, not in the shared >> envelope >> > > > > 6. OpenTelemetry + persistence listeners out of scope for this >> > > > > >> > > > > Also +1 on settling Kafka compat on the list. per-listener format >> > > option >> > > > > with today's flat payload as the default (shared envelope opt-in) >> > > sounds >> > > > > better than dual-publish. >> > > > > >> > > > > Draft v1 JSON I have in mind: >> > > > > >> > > > > { >> > > > > "specversion": "1.0", >> > > > > "id": "550e8400-e29b-41d4-a716-446655440000", >> > > > > "type": "org.apache.polaris.AFTER_CREATE_TABLE", >> > > > > "source": "org.apache.polaris", >> > > > > "time": "2026-10-04T12:34:56.789Z", >> > > > > "datacontenttype": "application/json", >> > > > > "data": { >> > > > > "event_type": "AFTER_CREATE_TABLE", >> > > > > "event_id": "550e8400-e29b-41d4-a716-446655440000", >> > > > > "timestamp": "2026-10-04T12:34:56.789Z", >> > > > > "realm_id": "default-realm", >> > > > > "principal": "alice", >> > > > > "activated_roles": ["catalog_admin"], >> > > > > "request_id": "req-123", >> > > > > "catalog_name": "sales", >> > > > > "namespace": ["analytics"], >> > > > > "table_name": "orders", >> > > > > "table_identifier": ["analytics", "orders"] >> > > > > } >> > > > > } >> > > > > >> > > > > A couple notes on purpose: >> > > > > - "data" is basically today's Kafka body, so id/time show up both >> as >> > CE >> > > > > context and again inside data (event_id / timestamp) for Kafka >> > parity >> > > > > - type is a namespaced PolarisEventType name; source is a constant >> > for >> > > > > now - will change if needed.. >> > > > > - webhook would put delivery time on an HTTP header >> > > > > (e.g. X-Polaris-Delivery-Time), not in the envelope >> > > > > >> > > > > if this is roughly the right shape, I'll draft the prerequisite PR >> > > around >> > > > > it. if you'd rather nail type/source naming first, tell me which >> > forms >> > > > > you prefer. >> > > > > >> > > > > Thanks, >> > > > > *Vignesh* >> > > > > >> > > > > [1] https://github.com/apache/polaris/pull/5084 >> > > > > >> > > > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré < >> [email protected]> >> > > > wrote: >> > > > > >> > > > > > Hi Vignesh >> > > > > > >> > > > > > Thanks for starting this thread. >> > > > > > >> > > > > > I'm +1 on the sequence: shared envelope first and then rebase >> the >> > > > webhook >> > > > > > PR. I slightly prefer your proposal over B or C. >> > > > > > >> > > > > > Some general comments: >> > > > > > 1. CloudEvents requires attribute names to be lowercase >> > alphanumerics >> > > > and >> > > > > > attribute values to be scalars afair. So a flat set of >> extensions >> > > > doesn't >> > > > > > really work for us: activatedroles as an array is not valid for >> > > > > > instance. Or we have to transform somehow. I would keep only the >> > > > > > CloudEvents context attributes at the top level and put the >> Polaris >> > > > fields >> > > > > > in "data". >> > > > > > 2. The EventAttributes keys already have underscored names >> > > > (catalog_name, >> > > > > > table_identifier, ...) and they match what Kafka emits. I would >> > reuse >> > > > them >> > > > > > as is. Then the current Kafka body basically becomes "data", >> which >> > > also >> > > > > > makes the migration easier. >> > > > > > 3. I would not start from the thin set: migrating Kafka to it >> would >> > > > drop >> > > > > > fields that consumers already get (namespace, catalog, view, >> ..). >> > > Kafka >> > > > > > only has nine optional fields, so I think v1 should start at >> Kafka >> > > > parity. >> > > > > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I >> > > think >> > > > we >> > > > > > should agree on the mechanism here on the list rather than in >> the >> > PR >> > > > > > description (if it doesn't happen on the mailing list, it never >> > > happens >> > > > > > :)). I would prefer a per-listener format option, with the >> current >> > > > format >> > > > > > as default, over dual-publish (which doubles the volume on the >> > > topic). >> > > > > > 5. The events SPI and the CloudWatch listener are both in >> > > > runtime/service, >> > > > > > and the Kafka extension depends on it. A new module would >> > introduce a >> > > > cycle >> > > > > > unless we extract the SPI first, so I would put the mapper next >> to >> > > the >> > > > SPI >> > > > > > in runtime/service for now, and move it when the SPI is moved. >> > > > > > 6. deliverytime is a transport metadata, I would move it to a >> HTTP >> > > > header >> > > > > > instead of the shared envelope. We should also decide what >> "type" >> > and >> > > > > > "source" look like (today it's a bare enum name and a constant). >> > > > > > >> > > > > > Could you share the v1 JSON you have in mind on this thread >> before >> > > > opening >> > > > > > the PR? I think it will be easier to converge on an example. >> > > > > > >> > > > > > The OpenTelemetry and persistence listeners have their own >> > > > representations >> > > > > > as well. I'm fine keeping them out of scope, but it's worth >> stating >> > > it >> > > > > > explicitly. >> > > > > > >> > > > > > Regards >> > > > > > JB >> > > > > > >> > > > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a < >> [email protected]> >> > > > wrote: >> > > > > > >> > > > > > > \Hi all, >> > > > > > > >> > > > > > > Robert and EJ have both asked that we settle a shared event >> > > envelope >> > > > > > > before merging the webhook listener [1]. I agree with that >> > > sequencing >> > > > > > > and would like a quick list consensus on shape and placement >> > > before I >> > > > > > > open the prerequisite PR. >> > > > > > > >> > > > > > > What each path emits today (from current main / [1]): >> > > > > > > >> > > > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper): >> > > > > > > flat snake_case: event_type, event_id, timestamp, realm_id, >> > > > principal, >> > > > > > > activated_roles, request_id, plus a fairly rich attribute >> set >> > > > > > > (table_identifier as a string list, namespace/catalog/view >> > > fields, >> > > > > > > rename source/destination, and similar). >> > > > > > > >> > > > > > > - Webhook ([1], WebhookEventListener.toJson): >> > > > > > > CloudEvents-shaped: specversion, id, type, source, time, >> plus >> > > > > > > extensions deliverytime, realmid, tableidentifier (single >> > > string), >> > > > > > > principal, activatedroles, requestid. Thinner attribute >> > coverage >> > > > than >> > > > > > > Kafka. >> > > > > > > >> > > > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener): >> > > > > > > flat map: event_type, optional table_identifier (string), >> > > realm_id, >> > > > > > > principal, activated_roles, request_id. No event_id in the >> JSON >> > > > body; >> > > > > > > timestamp is on InputLogEvent. >> > > > > > > >> > > > > > > So this is not a mechanical "extract EventToJsonMapper." >> Naming, >> > > > > > > structure, identifier encoding, and attribute breadth already >> > > > diverge, >> > > > > > > and the shared code should not live inside the kafka >> extension. >> > > > > > > >> > > > > > > Proposal: >> > > > > > > >> > > > > > > 1) Land a small prerequisite that introduces a shared, >> versioned, >> > > > > > > CloudEvents-shaped mapper (core fields + a documented >> > extension >> > > > set), >> > > > > > > placed outside any one transport extension. >> > > > > > > >> > > > > > > 2) Rebase [1] onto that shared mapper. >> > > > > > > >> > > > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an >> explicit >> > > > > > > compatibility plan so existing consumers are not silently >> > broken >> > > > > > > (version field and/or dual-publish window; exact mechanism >> in >> > > the >> > > > > > > prerequisite PR description). >> > > > > > > >> > > > > > > 4) Attribute breadth for v1: start from the >> > webhook/CloudWatch-thin >> > > > > > > common set; add Kafka's richer attributes behind the same >> > > > versioned >> > > > > > > envelope rather than blocking v1 on full Kafka parity. >> > > > > > > >> > > > > > > If that direction is wrong, the useful alternatives seem to >> be: >> > > > > > > >> > > > > > > - B) Prefer Kafka's flat snake_case as the shared shape >> instead >> > of >> > > > > > > CloudEvents-shaped. >> > > > > > > - C) Fold the shared mapper into [1] itself (larger single >> PR). >> > > > > > > >> > > > > > > Unless someone prefers B or C, I will open the prerequisite PR >> > > along >> > > > the >> > > > > > > lines of (1)-(4) early next week. >> > > > > > > >> > > > > > > Thanks, >> > > > > > > Vignesh >> > > > > > > >> > > > > > > [1] https://github.com/apache/polaris/pull/5084 >> > > > > > > >> > > > > > >> > > > >> > > >> > >> >
