\Hi all,
Robert and EJ have both asked that we settle a shared event envelope
before merging the webhook listener [1]. I agree with that sequencing
and would like a quick list consensus on shape and placement before I
open the prerequisite PR.
What each path emits today (from current main / [1]):
- Kafka (extensions/events/kafka/.../EventToJsonMapper):
flat snake_case: event_type, event_id, timestamp, realm_id, principal,
activated_roles, request_id, plus a fairly rich attribute set
(table_identifier as a string list, namespace/catalog/view fields,
rename source/destination, and similar).
- Webhook ([1], WebhookEventListener.toJson):
CloudEvents-shaped: specversion, id, type, source, time, plus
extensions deliverytime, realmid, tableidentifier (single string),
principal, activatedroles, requestid. Thinner attribute coverage than
Kafka.
- CloudWatch (runtime/service/.../AwsCloudWatchEventListener):
flat map: event_type, optional table_identifier (string), realm_id,
principal, activated_roles, request_id. No event_id in the JSON body;
timestamp is on InputLogEvent.
So this is not a mechanical "extract EventToJsonMapper." Naming,
structure, identifier encoding, and attribute breadth already diverge,
and the shared code should not live inside the kafka extension.
Proposal:
1) Land a small prerequisite that introduces a shared, versioned,
CloudEvents-shaped mapper (core fields + a documented extension set),
placed outside any one transport extension.
2) Rebase [1] onto that shared mapper.
3) Migrate Kafka and CloudWatch in follow-ups, with an explicit
compatibility plan so existing consumers are not silently broken
(version field and/or dual-publish window; exact mechanism in the
prerequisite PR description).
4) Attribute breadth for v1: start from the webhook/CloudWatch-thin
common set; add Kafka's richer attributes behind the same versioned
envelope rather than blocking v1 on full Kafka parity.
If that direction is wrong, the useful alternatives seem to be:
- B) Prefer Kafka's flat snake_case as the shared shape instead of
CloudEvents-shaped.
- C) Fold the shared mapper into [1] itself (larger single PR).
Unless someone prefers B or C, I will open the prerequisite PR along the
lines of (1)-(4) early next week.
Thanks,
Vignesh
[1] https://github.com/apache/polaris/pull/5084
