Hi JB,
Thanks - this is really useful, and I'm with you on the sequencing
(shared envelope first, then rebase the webhook).
Your points on shape make sense to me too:
1. CloudEvents context attrs at the top only, Polaris payload in "data"
(so things like activated_roles stay valid)
2. reuse the existing EventAttributes / Kafka snake_case keys in "data"
3. v1 at Kafka parity, not the thinner webhook set I suggested earlier
4. mapper next to the events SPI in runtime/service for now
5. deliverytime as webhook HTTP metadata, not in the shared envelope
6. OpenTelemetry + persistence listeners out of scope for this
Also +1 on settling Kafka compat on the list. per-listener format option
with today's flat payload as the default (shared envelope opt-in) sounds
better than dual-publish.
Draft v1 JSON I have in mind:
{
"specversion": "1.0",
"id": "550e8400-e29b-41d4-a716-446655440000",
"type": "org.apache.polaris.AFTER_CREATE_TABLE",
"source": "org.apache.polaris",
"time": "2026-10-04T12:34:56.789Z",
"datacontenttype": "application/json",
"data": {
"event_type": "AFTER_CREATE_TABLE",
"event_id": "550e8400-e29b-41d4-a716-446655440000",
"timestamp": "2026-10-04T12:34:56.789Z",
"realm_id": "default-realm",
"principal": "alice",
"activated_roles": ["catalog_admin"],
"request_id": "req-123",
"catalog_name": "sales",
"namespace": ["analytics"],
"table_name": "orders",
"table_identifier": ["analytics", "orders"]
}
}
A couple notes on purpose:
- "data" is basically today's Kafka body, so id/time show up both as CE
context and again inside data (event_id / timestamp) for Kafka parity
- type is a namespaced PolarisEventType name; source is a constant for
now - will change if needed..
- webhook would put delivery time on an HTTP header
(e.g. X-Polaris-Delivery-Time), not in the envelope
if this is roughly the right shape, I'll draft the prerequisite PR around
it. if you'd rather nail type/source naming first, tell me which forms
you prefer.
Thanks,
*Vignesh*
[1] https://github.com/apache/polaris/pull/5084
On Mon, 5 Oct 2026 at 11:03, Jean-Baptiste Onofré <[email protected]> wrote:
> Hi Vignesh
>
> Thanks for starting this thread.
>
> I'm +1 on the sequence: shared envelope first and then rebase the webhook
> PR. I slightly prefer your proposal over B or C.
>
> Some general comments:
> 1. CloudEvents requires attribute names to be lowercase alphanumerics and
> attribute values to be scalars afair. So a flat set of extensions doesn't
> really work for us: activatedroles as an array is not valid for
> instance. Or we have to transform somehow. I would keep only the
> CloudEvents context attributes at the top level and put the Polaris fields
> in "data".
> 2. The EventAttributes keys already have underscored names (catalog_name,
> table_identifier, ...) and they match what Kafka emits. I would reuse them
> as is. Then the current Kafka body basically becomes "data", which also
> makes the migration easier.
> 3. I would not start from the thin set: migrating Kafka to it would drop
> fields that consumers already get (namespace, catalog, view, ..). Kafka
> only has nine optional fields, so I think v1 should start at Kafka parity.
> 4. The Kafka payload is already in Polaris 1.7.0 and 1.8.0, so I think we
> should agree on the mechanism here on the list rather than in the PR
> description (if it doesn't happen on the mailing list, it never happens
> :)). I would prefer a per-listener format option, with the current format
> as default, over dual-publish (which doubles the volume on the topic).
> 5. The events SPI and the CloudWatch listener are both in runtime/service,
> and the Kafka extension depends on it. A new module would introduce a cycle
> unless we extract the SPI first, so I would put the mapper next to the SPI
> in runtime/service for now, and move it when the SPI is moved.
> 6. deliverytime is a transport metadata, I would move it to a HTTP header
> instead of the shared envelope. We should also decide what "type" and
> "source" look like (today it's a bare enum name and a constant).
>
> Could you share the v1 JSON you have in mind on this thread before opening
> the PR? I think it will be easier to converge on an example.
>
> The OpenTelemetry and persistence listeners have their own representations
> as well. I'm fine keeping them out of scope, but it's worth stating it
> explicitly.
>
> Regards
> JB
>
> On Sun, Oct 4, 2026 at 11:47 AM vignesh a <[email protected]> wrote:
>
> > \Hi all,
> >
> > Robert and EJ have both asked that we settle a shared event envelope
> > before merging the webhook listener [1]. I agree with that sequencing
> > and would like a quick list consensus on shape and placement before I
> > open the prerequisite PR.
> >
> > What each path emits today (from current main / [1]):
> >
> > - Kafka (extensions/events/kafka/.../EventToJsonMapper):
> > flat snake_case: event_type, event_id, timestamp, realm_id, principal,
> > activated_roles, request_id, plus a fairly rich attribute set
> > (table_identifier as a string list, namespace/catalog/view fields,
> > rename source/destination, and similar).
> >
> > - Webhook ([1], WebhookEventListener.toJson):
> > CloudEvents-shaped: specversion, id, type, source, time, plus
> > extensions deliverytime, realmid, tableidentifier (single string),
> > principal, activatedroles, requestid. Thinner attribute coverage than
> > Kafka.
> >
> > - CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
> > flat map: event_type, optional table_identifier (string), realm_id,
> > principal, activated_roles, request_id. No event_id in the JSON body;
> > timestamp is on InputLogEvent.
> >
> > So this is not a mechanical "extract EventToJsonMapper." Naming,
> > structure, identifier encoding, and attribute breadth already diverge,
> > and the shared code should not live inside the kafka extension.
> >
> > Proposal:
> >
> > 1) Land a small prerequisite that introduces a shared, versioned,
> > CloudEvents-shaped mapper (core fields + a documented extension set),
> > placed outside any one transport extension.
> >
> > 2) Rebase [1] onto that shared mapper.
> >
> > 3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
> > compatibility plan so existing consumers are not silently broken
> > (version field and/or dual-publish window; exact mechanism in the
> > prerequisite PR description).
> >
> > 4) Attribute breadth for v1: start from the webhook/CloudWatch-thin
> > common set; add Kafka's richer attributes behind the same versioned
> > envelope rather than blocking v1 on full Kafka parity.
> >
> > If that direction is wrong, the useful alternatives seem to be:
> >
> > - B) Prefer Kafka's flat snake_case as the shared shape instead of
> > CloudEvents-shaped.
> > - C) Fold the shared mapper into [1] itself (larger single PR).
> >
> > Unless someone prefers B or C, I will open the prerequisite PR along the
> > lines of (1)-(4) early next week.
> >
> > Thanks,
> > Vignesh
> >
> > [1] https://github.com/apache/polaris/pull/5084
> >
>
