Hi Vgnesh, Your proposal sounds good to be at high level.
Just to clarify: is our end goal here to use the same JSON payload for all event listeners that produce JSON? Are we going to adhere to the CloudEvents spec for this JSON? Obviously starting with 1.0 and migrating to newer version as the become available... Is that the plan? How are we going to migrare existing use cases that are not 1:1 with the ideal JSON schema? Will be publish a union of attributes for a few releases to allow consumers to switch to the new schema? +1 to adding a dedicated SPI module. I suppose it can also contain the common java-to-JSON mapping code. Thanks, Dmitri. On Mon, Oct 5, 2026 at 4:29 AM Alex Dutra <[email protected]> wrote: > Hi all, > > +1 for the proposed v1 JSON, and +1 for having Kafka parity in v1. > > I also welcome the idea of isolating the Events SPI in its own module. > Having the Kafka module depend on runtime/service just for the SPI is > concerning. > > Looking forward to the first PR! > > Thanks, > Alex > > On Mon, Oct 5, 2026 at 10:02 AM vignesh a <[email protected]> wrote: > > > > Hi JB, > > > > Thanks - this is really useful, and I'm with you on the sequencing > > (shared envelope first, then rebase the webhook). > > > > Your points on shape make sense to me too: > > 1. CloudEvents context attrs at the top only, Polaris payload in "data" > > (so things like activated_roles stay valid) > > 2. reuse the existing EventAttributes / Kafka snake_case keys in "data" > > 3. v1 at Kafka parity, not the thinner webhook set I suggested earlier > > 4. mapper next to the events SPI in runtime/service for now > > 5. deliverytime as webhook HTTP metadata, not in the shared envelope > > 6. OpenTelemetry + persistence listeners out of scope for this > > > > Also +1 on settling Kafka compat on the list. per-listener format option > > with today's flat payload as the default (shared envelope opt-in) sounds > > better than dual-publish. > > > > Draft v1 JSON I have in mind: > > > > { > > "specversion": "1.0", > > "id": "550e8400-e29b-41d4-a716-446655440000", > > "type": "org.apache.polaris.AFTER_CREATE_TABLE", > > "source": "org.apache.polaris", > > "time": "2026-10-04T12:34:56.789Z", > > "datacontenttype": "application/json", > > "data": { > > "event_type": "AFTER_CREATE_TABLE", > > "event_id": "550e8400-e29b-41d4-a716-446655440000", > > "timestamp": "2026-10-04T12:34:56.789Z", > > "realm_id": "default-realm", > > "principal": "alice", > > "activated_roles": ["catalog_admin"], > > "request_id": "req-123", > > "catalog_name": "sales", > > "namespace": ["analytics"], > > "table_name": "orders", > > "table_identifier": ["analytics", "orders"] > > } > > } > > > > A couple notes on purpose: > > - "data" is basically today's Kafka body, so id/time show up both as CE > > context and again inside data (event_id / timestamp) for Kafka parity > > - type is a namespaced PolarisEventType name; source is a constant for > > now - will change if needed.. > > - webhook would put delivery time on an HTTP header > > (e.g. X-Polaris-Delivery-Time), not in the envelope > > > > if this is roughly the right shape, I'll draft the prerequisite PR around > > it. if you'd rather nail type/source naming first, tell me which forms > > you prefer. > > > > Thanks, > > *Vignesh* > > > > [1] https://github.com/apache/polaris/pull/5084 > > > > On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <[email protected]> > wrote: > > > > > Hi Vignesh > > > > > > Thanks for starting this thread. > > > > > > I'm +1 on the sequence: shared envelope first and then rebase the > webhook > > > PR. I slightly prefer your proposal over B or C. > > > > > > Some general comments: > > > 1. CloudEvents requires attribute names to be lowercase alphanumerics > and > > > attribute values to be scalars afair. So a flat set of extensions > doesn't > > > really work for us: activatedroles as an array is not valid for > > > instance. Or we have to transform somehow. I would keep only the > > > CloudEvents context attributes at the top level and put the Polaris > fields > > > in "data". > > > 2. The EventAttributes keys already have underscored names > (catalog_name, > > > table_identifier, ...) and they match what Kafka emits. I would reuse > them > > > as is. Then the current Kafka body basically becomes "data", which also > > > makes the migration easier. > > > 3. I would not start from the thin set: migrating Kafka to it would > drop > > > fields that consumers already get (namespace, catalog, view, ..). Kafka > > > only has nine optional fields, so I think v1 should start at Kafka > parity. > > > 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I think > we > > > should agree on the mechanism here on the list rather than in the PR > > > description (if it doesn't happen on the mailing list, it never happens > > > :)). I would prefer a per-listener format option, with the current > format > > > as default, over dual-publish (which doubles the volume on the topic). > > > 5. The events SPI and the CloudWatch listener are both in > runtime/service, > > > and the Kafka extension depends on it. A new module would introduce a > cycle > > > unless we extract the SPI first, so I would put the mapper next to the > SPI > > > in runtime/service for now, and move it when the SPI is moved. > > > 6. deliverytime is a transport metadata, I would move it to a HTTP > header > > > instead of the shared envelope. We should also decide what "type" and > > > "source" look like (today it's a bare enum name and a constant). > > > > > > Could you share the v1 JSON you have in mind on this thread before > opening > > > the PR? I think it will be easier to converge on an example. > > > > > > The OpenTelemetry and persistence listeners have their own > representations > > > as well. I'm fine keeping them out of scope, but it's worth stating it > > > explicitly. > > > > > > Regards > > > JB > > > > > > On Sun, Oct 4, 2026 at 11:47 AM vignesh a <[email protected]> > wrote: > > > > > > > \Hi all, > > > > > > > > Robert and EJ have both asked that we settle a shared event envelope > > > > before merging the webhook listener [1]. I agree with that sequencing > > > > and would like a quick list consensus on shape and placement before I > > > > open the prerequisite PR. > > > > > > > > What each path emits today (from current main / [1]): > > > > > > > > - Kafka (extensions/events/kafka/.../EventToJsonMapper): > > > > flat snake_case: event_type, event_id, timestamp, realm_id, > principal, > > > > activated_roles, request_id, plus a fairly rich attribute set > > > > (table_identifier as a string list, namespace/catalog/view fields, > > > > rename source/destination, and similar). > > > > > > > > - Webhook ([1], WebhookEventListener.toJson): > > > > CloudEvents-shaped: specversion, id, type, source, time, plus > > > > extensions deliverytime, realmid, tableidentifier (single string), > > > > principal, activatedroles, requestid. Thinner attribute coverage > than > > > > Kafka. > > > > > > > > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener): > > > > flat map: event_type, optional table_identifier (string), realm_id, > > > > principal, activated_roles, request_id. No event_id in the JSON > body; > > > > timestamp is on InputLogEvent. > > > > > > > > So this is not a mechanical "extract EventToJsonMapper." Naming, > > > > structure, identifier encoding, and attribute breadth already > diverge, > > > > and the shared code should not live inside the kafka extension. > > > > > > > > Proposal: > > > > > > > > 1) Land a small prerequisite that introduces a shared, versioned, > > > > CloudEvents-shaped mapper (core fields + a documented extension > set), > > > > placed outside any one transport extension. > > > > > > > > 2) Rebase [1] onto that shared mapper. > > > > > > > > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit > > > > compatibility plan so existing consumers are not silently broken > > > > (version field and/or dual-publish window; exact mechanism in the > > > > prerequisite PR description). > > > > > > > > 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin > > > > common set; add Kafka's richer attributes behind the same > versioned > > > > envelope rather than blocking v1 on full Kafka parity. > > > > > > > > If that direction is wrong, the useful alternatives seem to be: > > > > > > > > - B) Prefer Kafka's flat snake_case as the shared shape instead of > > > > CloudEvents-shaped. > > > > - C) Fold the shared mapper into [1] itself (larger single PR). > > > > > > > > Unless someone prefers B or C, I will open the prerequisite PR along > the > > > > lines of (1)-(4) early next week. > > > > > > > > Thanks, > > > > Vignesh > > > > > > > > [1] https://github.com/apache/polaris/pull/5084 > > > > > > > >
