Hi all, +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1.
I also welcome the idea of isolating the Events SPI in its own module. Having the Kafka module depend on runtime/service just for the SPI is concerning. Looking forward to the first PR! Thanks, Alex On Mon, Oct 5, 2026 at 10:02 AM vignesh a <[email protected]> wrote: > > Hi JB, > > Thanks - this is really useful, and I'm with you on the sequencing > (shared envelope first, then rebase the webhook). > > Your points on shape make sense to me too: > 1. CloudEvents context attrs at the top only, Polaris payload in "data" > (so things like activated_roles stay valid) > 2. reuse the existing EventAttributes / Kafka snake_case keys in "data" > 3. v1 at Kafka parity, not the thinner webhook set I suggested earlier > 4. mapper next to the events SPI in runtime/service for now > 5. deliverytime as webhook HTTP metadata, not in the shared envelope > 6. OpenTelemetry + persistence listeners out of scope for this > > Also +1 on settling Kafka compat on the list. per-listener format option > with today's flat payload as the default (shared envelope opt-in) sounds > better than dual-publish. > > Draft v1 JSON I have in mind: > > { > "specversion": "1.0", > "id": "550e8400-e29b-41d4-a716-446655440000", > "type": "org.apache.polaris.AFTER_CREATE_TABLE", > "source": "org.apache.polaris", > "time": "2026-10-04T12:34:56.789Z", > "datacontenttype": "application/json", > "data": { > "event_type": "AFTER_CREATE_TABLE", > "event_id": "550e8400-e29b-41d4-a716-446655440000", > "timestamp": "2026-10-04T12:34:56.789Z", > "realm_id": "default-realm", > "principal": "alice", > "activated_roles": ["catalog_admin"], > "request_id": "req-123", > "catalog_name": "sales", > "namespace": ["analytics"], > "table_name": "orders", > "table_identifier": ["analytics", "orders"] > } > } > > A couple notes on purpose: > - "data" is basically today's Kafka body, so id/time show up both as CE > context and again inside data (event_id / timestamp) for Kafka parity > - type is a namespaced PolarisEventType name; source is a constant for > now - will change if needed.. > - webhook would put delivery time on an HTTP header > (e.g. X-Polaris-Delivery-Time), not in the envelope > > if this is roughly the right shape, I'll draft the prerequisite PR around > it. if you'd rather nail type/source naming first, tell me which forms > you prefer. > > Thanks, > *Vignesh* > > [1] https://github.com/apache/polaris/pull/5084 > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <[email protected]> wrote: > > > Hi Vignesh > > > > Thanks for starting this thread. > > > > I'm +1 on the sequence: shared envelope first and then rebase the webhook > > PR. I slightly prefer your proposal over B or C. > > > > Some general comments: > > 1. CloudEvents requires attribute names to be lowercase alphanumerics and > > attribute values to be scalars afair. So a flat set of extensions doesn't > > really work for us: activatedroles as an array is not valid for > > instance. Or we have to transform somehow. I would keep only the > > CloudEvents context attributes at the top level and put the Polaris fields > > in "data". > > 2. The EventAttributes keys already have underscored names (catalog_name, > > table_identifier, ...) and they match what Kafka emits. I would reuse them > > as is. Then the current Kafka body basically becomes "data", which also > > makes the migration easier. > > 3. I would not start from the thin set: migrating Kafka to it would drop > > fields that consumers already get (namespace, catalog, view, ..). Kafka > > only has nine optional fields, so I think v1 should start at Kafka parity. > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I think we > > should agree on the mechanism here on the list rather than in the PR > > description (if it doesn't happen on the mailing list, it never happens > > :)). I would prefer a per-listener format option, with the current format > > as default, over dual-publish (which doubles the volume on the topic). > > 5. The events SPI and the CloudWatch listener are both in runtime/service, > > and the Kafka extension depends on it. A new module would introduce a cycle > > unless we extract the SPI first, so I would put the mapper next to the SPI > > in runtime/service for now, and move it when the SPI is moved. > > 6. deliverytime is a transport metadata, I would move it to a HTTP header > > instead of the shared envelope. We should also decide what "type" and > > "source" look like (today it's a bare enum name and a constant). > > > > Could you share the v1 JSON you have in mind on this thread before opening > > the PR? I think it will be easier to converge on an example. > > > > The OpenTelemetry and persistence listeners have their own representations > > as well. I'm fine keeping them out of scope, but it's worth stating it > > explicitly. > > > > Regards > > JB > > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <[email protected]> wrote: > > > > > \Hi all, > > > > > > Robert and EJ have both asked that we settle a shared event envelope > > > before merging the webhook listener [1]. I agree with that sequencing > > > and would like a quick list consensus on shape and placement before I > > > open the prerequisite PR. > > > > > > What each path emits today (from current main / [1]): > > > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper): > > > flat snake_case: event_type, event_id, timestamp, realm_id, principal, > > > activated_roles, request_id, plus a fairly rich attribute set > > > (table_identifier as a string list, namespace/catalog/view fields, > > > rename source/destination, and similar). > > > > > > - Webhook ([1], WebhookEventListener.toJson): > > > CloudEvents-shaped: specversion, id, type, source, time, plus > > > extensions deliverytime, realmid, tableidentifier (single string), > > > principal, activatedroles, requestid. Thinner attribute coverage than > > > Kafka. > > > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener): > > > flat map: event_type, optional table_identifier (string), realm_id, > > > principal, activated_roles, request_id. No event_id in the JSON body; > > > timestamp is on InputLogEvent. > > > > > > So this is not a mechanical "extract EventToJsonMapper." Naming, > > > structure, identifier encoding, and attribute breadth already diverge, > > > and the shared code should not live inside the kafka extension. > > > > > > Proposal: > > > > > > 1) Land a small prerequisite that introduces a shared, versioned, > > > CloudEvents-shaped mapper (core fields + a documented extension set), > > > placed outside any one transport extension. > > > > > > 2) Rebase [1] onto that shared mapper. > > > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit > > > compatibility plan so existing consumers are not silently broken > > > (version field and/or dual-publish window; exact mechanism in the > > > prerequisite PR description). > > > > > > 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin > > > common set; add Kafka's richer attributes behind the same versioned > > > envelope rather than blocking v1 on full Kafka parity. > > > > > > If that direction is wrong, the useful alternatives seem to be: > > > > > > - B) Prefer Kafka's flat snake_case as the shared shape instead of > > > CloudEvents-shaped. > > > - C) Fold the shared mapper into [1] itself (larger single PR). > > > > > > Unless someone prefers B or C, I will open the prerequisite PR along the > > > lines of (1)-(4) early next week. > > > > > > Thanks, > > > Vignesh > > > > > > [1] https://github.com/apache/polaris/pull/5084 > > > > >
