Hi Vignesh Thanks for starting this thread.
I'm +1 on the sequence: shared envelope first and then rebase the webhook PR. I slightly prefer your proposal over B or C. Some general comments: 1. CloudEvents requires attribute names to be lowercase alphanumerics and attribute values to be scalars afair. So a flat set of extensions doesn't really work for us: activatedroles as an array is not valid for instance. Or we have to transform somehow. I would keep only the CloudEvents context attributes at the top level and put the Polaris fields in "data". 2. The EventAttributes keys already have underscored names (catalog_name, table_identifier, ...) and they match what Kafka emits. I would reuse them as is. Then the current Kafka body basically becomes "data", which also makes the migration easier. 3. I would not start from the thin set: migrating Kafka to it would drop fields that consumers already get (namespace, catalog, view, ..). Kafka only has nine optional fields, so I think v1 should start at Kafka parity. 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I think we should agree on the mechanism here on the list rather than in the PR description (if it doesn't happen on the mailing list, it never happens :)). I would prefer a per-listener format option, with the current format as default, over dual-publish (which doubles the volume on the topic). 5. The events SPI and the CloudWatch listener are both in runtime/service, and the Kafka extension depends on it. A new module would introduce a cycle unless we extract the SPI first, so I would put the mapper next to the SPI in runtime/service for now, and move it when the SPI is moved. 6. deliverytime is a transport metadata, I would move it to a HTTP header instead of the shared envelope. We should also decide what "type" and "source" look like (today it's a bare enum name and a constant). Could you share the v1 JSON you have in mind on this thread before opening the PR? I think it will be easier to converge on an example. The OpenTelemetry and persistence listeners have their own representations as well. I'm fine keeping them out of scope, but it's worth stating it explicitly. Regards JB On Sun, Oct 4, 2026 at 11:47 AM vignesh a <[email protected]> wrote: > \Hi all, > > Robert and EJ have both asked that we settle a shared event envelope > before merging the webhook listener [1]. I agree with that sequencing > and would like a quick list consensus on shape and placement before I > open the prerequisite PR. > > What each path emits today (from current main / [1]): > > - Kafka (extensions/events/kafka/.../EventToJsonMapper): > flat snake_case: event_type, event_id, timestamp, realm_id, principal, > activated_roles, request_id, plus a fairly rich attribute set > (table_identifier as a string list, namespace/catalog/view fields, > rename source/destination, and similar). > > - Webhook ([1], WebhookEventListener.toJson): > CloudEvents-shaped: specversion, id, type, source, time, plus > extensions deliverytime, realmid, tableidentifier (single string), > principal, activatedroles, requestid. Thinner attribute coverage than > Kafka. > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener): > flat map: event_type, optional table_identifier (string), realm_id, > principal, activated_roles, request_id. No event_id in the JSON body; > timestamp is on InputLogEvent. > > So this is not a mechanical "extract EventToJsonMapper." Naming, > structure, identifier encoding, and attribute breadth already diverge, > and the shared code should not live inside the kafka extension. > > Proposal: > > 1) Land a small prerequisite that introduces a shared, versioned, > CloudEvents-shaped mapper (core fields + a documented extension set), > placed outside any one transport extension. > > 2) Rebase [1] onto that shared mapper. > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit > compatibility plan so existing consumers are not silently broken > (version field and/or dual-publish window; exact mechanism in the > prerequisite PR description). > > 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin > common set; add Kafka's richer attributes behind the same versioned > envelope rather than blocking v1 on full Kafka parity. > > If that direction is wrong, the useful alternatives seem to be: > > - B) Prefer Kafka's flat snake_case as the shared shape instead of > CloudEvents-shaped. > - C) Fold the shared mapper into [1] itself (larger single PR). > > Unless someone prefers B or C, I will open the prerequisite PR along the > lines of (1)-(4) early next week. > > Thanks, > Vignesh > > [1] https://github.com/apache/polaris/pull/5084 >
