Hi Vignesh, The proposal LGTM.
Minor comment: SInce CloudEvents apparently prefer lowercase names, shall we lowercase the type namne (in JSON)? Re: backward compatibility for Kafka. Having a flag to send old payloads works for me. However, I believe this tends to prolong adoption and users have to face a hard change anyway (when their admin flips the switch). This may not be suitable for rolling upgrade scenarios. That said, we can probably debate this later, after establishing the common format. Cheers, Dmitri. On Tue, Oct 6, 2026 at 8:11 AM vignesh a <[email protected]> wrote: > Hi Dmitri, > > Thanks - good questions, and yes at a high level that's the direction. > Wanted to answer them against what JB and I already talked through, and > I've added the draft v1 JSON again so we're looking at the same thing. > > 1) Same JSON for all JSON-producing listeners? > > For the transport listeners in this thread (Kafka, webhook, CloudWatch), > yes - same CloudEvents-shaped payload is the end goal. OpenTelemetry and > the persistence listener keep their own representations for now; those > stay out of scope for the first PR. > > 2) Adhere to CloudEvents? > > Yes. v1 means CloudEvents 1.0 context attributes at the top only, and the > Polaris fields in "data". That also keeps us honest with CE rules on > context attrs (lowercase alphanumerics / scalars) - so things like > activated_roles live in "data", not as top-level CE extensions. If a > newer CE version becomes relevant later we can bump then.. > > 3) Migrating existing consumers that aren't 1:1 with the new shape? > > Union / dual-publish for a few releases is one way to do it. On the list > JB leaned toward a per-listener format option instead (today's flat Kafka > payload as the default, shared envelope opt-in), mainly so we don't > double volume on the topic. That felt right to me too for 1.7 / 1.8 > consumers - they keep working until they flip the knob. Webhook [1] can > land on the shared envelope from day one since it isn't released that > way yet. Open to other ideas if you see a cleaner path.. LMK > > 4) Dedicated SPI module? > > Also +1 (Alex mentioned the same). Kafka depending on runtime/service > just for the SPI is tricky, and having the mapper live in that SPI > module later makes a lot of sense. For the first PR I was thinking of > putting the shared mapper next to the existing events SPI in > runtime/service, then extracting SPI + mapper as a follow-up - mostly to > avoid getting stuck on module cycles before the envelope lands (JB had > flagged that). Will do the SPI split earlier if that feels better. WDYT ?? > > Draft v1 JSON (CE context + today's Kafka body in "data"): > > { > "specversion": "1.0", > "id": "550e8400-e29b-41d4-a716-446655440000", > "type": "org.apache.polaris.AFTER_CREATE_TABLE", > "source": "org.apache.polaris", > "time": "2026-10-04T12:34:56.789Z", > "datacontenttype": "application/json", > "data": { > "event_type": "AFTER_CREATE_TABLE", > "event_id": "550e8400-e29b-41d4-a716-446655440000", > "timestamp": "2026-10-04T12:34:56.789Z", > "realm_id": "default-realm", > "principal": "alice", > "activated_roles": ["catalog_admin"], > "request_id": "req-123", > "catalog_name": "sales", > "namespace": ["analytics"], > "table_name": "orders", > "table_identifier": ["analytics", "orders"] > } > } > > A couple notes: > - "data" is basically today's Kafka EventToJsonMapper body, so id/time > show up both as CE context and again inside data (event_id / > timestamp) for Kafka parity. > - Type is a namespaced PolarisEventType name; source is a constant for > now - happy to bike-shed those if needed. > - Webhook delivery time stays transport metadata (HTTP header), not in > the shared envelope. > - Rename events would also carry source / destination arrays in "data", > same as Kafka today; omitted above because this example is > AFTER_CREATE_TABLE. > > If this looks right, I'll open the prerequisite PR on that basis... > > Thanks > *Vignesh* > > [1] https://github.com/apache/polaris/pull/5084 > > On Tue, 6 Oct 2026 at 05:53, Dmitri Bourlatchkov <[email protected]> wrote: > > > Hi Vgnesh, > > > > Your proposal sounds good to be at high level. > > > > Just to clarify: is our end goal here to use the same JSON payload for > all > > event listeners that produce JSON? > > > > Are we going to adhere to the CloudEvents spec for this JSON? Obviously > > starting with 1.0 and migrating to newer version as the become > available... > > Is that the plan? > > > > How are we going to migrare existing use cases that are not 1:1 with > > the ideal JSON schema? Will be publish a union of attributes for a few > > releases to allow consumers to switch to the new schema? > > > > +1 to adding a dedicated SPI module. I suppose it can also contain the > > common java-to-JSON mapping code. > > > > Thanks, > > Dmitri. > > > > On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <[email protected]> wrote: > > > > > Hi all, > > > > > > +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1. > > > > > > I also welcome the idea of isolating the Events SPI in its own module. > > > Having the Kafka module depend on runtime/service just for the SPI is > > > concerning. > > > > > > Looking forward to the first PR! > > > > > > Thanks, > > > Alex > > > > > > On Mon, Oct 5, 2026 at 10:02 AM vignesh a <[email protected]> > > wrote: > > > > > > > > Hi JB, > > > > > > > > Thanks - this is really useful, and I'm with you on the sequencing > > > > (shared envelope first, then rebase the webhook). > > > > > > > > Your points on shape make sense to me too: > > > > 1. CloudEvents context attrs at the top only, Polaris payload in > "data" > > > > (so things like activated_roles stay valid) > > > > 2. reuse the existing EventAttributes / Kafka snake_case keys in > "data" > > > > 3. v1 at Kafka parity, not the thinner webhook set I suggested > earlier > > > > 4. mapper next to the events SPI in runtime/service for now > > > > 5. deliverytime as webhook HTTP metadata, not in the shared envelope > > > > 6. OpenTelemetry + persistence listeners out of scope for this > > > > > > > > Also +1 on settling Kafka compat on the list. per-listener format > > option > > > > with today's flat payload as the default (shared envelope opt-in) > > sounds > > > > better than dual-publish. > > > > > > > > Draft v1 JSON I have in mind: > > > > > > > > { > > > > "specversion": "1.0", > > > > "id": "550e8400-e29b-41d4-a716-446655440000", > > > > "type": "org.apache.polaris.AFTER_CREATE_TABLE", > > > > "source": "org.apache.polaris", > > > > "time": "2026-10-04T12:34:56.789Z", > > > > "datacontenttype": "application/json", > > > > "data": { > > > > "event_type": "AFTER_CREATE_TABLE", > > > > "event_id": "550e8400-e29b-41d4-a716-446655440000", > > > > "timestamp": "2026-10-04T12:34:56.789Z", > > > > "realm_id": "default-realm", > > > > "principal": "alice", > > > > "activated_roles": ["catalog_admin"], > > > > "request_id": "req-123", > > > > "catalog_name": "sales", > > > > "namespace": ["analytics"], > > > > "table_name": "orders", > > > > "table_identifier": ["analytics", "orders"] > > > > } > > > > } > > > > > > > > A couple notes on purpose: > > > > - "data" is basically today's Kafka body, so id/time show up both as > CE > > > > context and again inside data (event_id / timestamp) for Kafka > parity > > > > - type is a namespaced PolarisEventType name; source is a constant > for > > > > now - will change if needed.. > > > > - webhook would put delivery time on an HTTP header > > > > (e.g. X-Polaris-Delivery-Time), not in the envelope > > > > > > > > if this is roughly the right shape, I'll draft the prerequisite PR > > around > > > > it. if you'd rather nail type/source naming first, tell me which > forms > > > > you prefer. > > > > > > > > Thanks, > > > > *Vignesh* > > > > > > > > [1] https://github.com/apache/polaris/pull/5084 > > > > > > > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > > > > > Hi Vignesh > > > > > > > > > > Thanks for starting this thread. > > > > > > > > > > I'm +1 on the sequence: shared envelope first and then rebase the > > > webhook > > > > > PR. I slightly prefer your proposal over B or C. > > > > > > > > > > Some general comments: > > > > > 1. CloudEvents requires attribute names to be lowercase > alphanumerics > > > and > > > > > attribute values to be scalars afair. So a flat set of extensions > > > doesn't > > > > > really work for us: activatedroles as an array is not valid for > > > > > instance. Or we have to transform somehow. I would keep only the > > > > > CloudEvents context attributes at the top level and put the Polaris > > > fields > > > > > in "data". > > > > > 2. The EventAttributes keys already have underscored names > > > (catalog_name, > > > > > table_identifier, ...) and they match what Kafka emits. I would > reuse > > > them > > > > > as is. Then the current Kafka body basically becomes "data", which > > also > > > > > makes the migration easier. > > > > > 3. I would not start from the thin set: migrating Kafka to it would > > > drop > > > > > fields that consumers already get (namespace, catalog, view, ..). > > Kafka > > > > > only has nine optional fields, so I think v1 should start at Kafka > > > parity. > > > > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I > > think > > > we > > > > > should agree on the mechanism here on the list rather than in the > PR > > > > > description (if it doesn't happen on the mailing list, it never > > happens > > > > > :)). I would prefer a per-listener format option, with the current > > > format > > > > > as default, over dual-publish (which doubles the volume on the > > topic). > > > > > 5. The events SPI and the CloudWatch listener are both in > > > runtime/service, > > > > > and the Kafka extension depends on it. A new module would > introduce a > > > cycle > > > > > unless we extract the SPI first, so I would put the mapper next to > > the > > > SPI > > > > > in runtime/service for now, and move it when the SPI is moved. > > > > > 6. deliverytime is a transport metadata, I would move it to a HTTP > > > header > > > > > instead of the shared envelope. We should also decide what "type" > and > > > > > "source" look like (today it's a bare enum name and a constant). > > > > > > > > > > Could you share the v1 JSON you have in mind on this thread before > > > opening > > > > > the PR? I think it will be easier to converge on an example. > > > > > > > > > > The OpenTelemetry and persistence listeners have their own > > > representations > > > > > as well. I'm fine keeping them out of scope, but it's worth stating > > it > > > > > explicitly. > > > > > > > > > > Regards > > > > > JB > > > > > > > > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <[email protected]> > > > wrote: > > > > > > > > > > > \Hi all, > > > > > > > > > > > > Robert and EJ have both asked that we settle a shared event > > envelope > > > > > > before merging the webhook listener [1]. I agree with that > > sequencing > > > > > > and would like a quick list consensus on shape and placement > > before I > > > > > > open the prerequisite PR. > > > > > > > > > > > > What each path emits today (from current main / [1]): > > > > > > > > > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper): > > > > > > flat snake_case: event_type, event_id, timestamp, realm_id, > > > principal, > > > > > > activated_roles, request_id, plus a fairly rich attribute set > > > > > > (table_identifier as a string list, namespace/catalog/view > > fields, > > > > > > rename source/destination, and similar). > > > > > > > > > > > > - Webhook ([1], WebhookEventListener.toJson): > > > > > > CloudEvents-shaped: specversion, id, type, source, time, plus > > > > > > extensions deliverytime, realmid, tableidentifier (single > > string), > > > > > > principal, activatedroles, requestid. Thinner attribute > coverage > > > than > > > > > > Kafka. > > > > > > > > > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener): > > > > > > flat map: event_type, optional table_identifier (string), > > realm_id, > > > > > > principal, activated_roles, request_id. No event_id in the JSON > > > body; > > > > > > timestamp is on InputLogEvent. > > > > > > > > > > > > So this is not a mechanical "extract EventToJsonMapper." Naming, > > > > > > structure, identifier encoding, and attribute breadth already > > > diverge, > > > > > > and the shared code should not live inside the kafka extension. > > > > > > > > > > > > Proposal: > > > > > > > > > > > > 1) Land a small prerequisite that introduces a shared, versioned, > > > > > > CloudEvents-shaped mapper (core fields + a documented > extension > > > set), > > > > > > placed outside any one transport extension. > > > > > > > > > > > > 2) Rebase [1] onto that shared mapper. > > > > > > > > > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit > > > > > > compatibility plan so existing consumers are not silently > broken > > > > > > (version field and/or dual-publish window; exact mechanism in > > the > > > > > > prerequisite PR description). > > > > > > > > > > > > 4) Attribute breadth for v1: start from the > webhook/CloudWatch-thin > > > > > > common set; add Kafka's richer attributes behind the same > > > versioned > > > > > > envelope rather than blocking v1 on full Kafka parity. > > > > > > > > > > > > If that direction is wrong, the useful alternatives seem to be: > > > > > > > > > > > > - B) Prefer Kafka's flat snake_case as the shared shape instead > of > > > > > > CloudEvents-shaped. > > > > > > - C) Fold the shared mapper into [1] itself (larger single PR). > > > > > > > > > > > > Unless someone prefers B or C, I will open the prerequisite PR > > along > > > the > > > > > > lines of (1)-(4) early next week. > > > > > > > > > > > > Thanks, > > > > > > Vignesh > > > > > > > > > > > > [1] https://github.com/apache/polaris/pull/5084 > > > > > > > > > > > > > > > > >
