Hi Nándor, > it seems that the transformation of most service events to event entities is > missing from PolarisPersistenceEventListener
Yes, unfortunately this has never been addressed, and yet it makes the whole events feature pretty unusable. There is an old issue for that: [1]. An external contributor tried to tackle it, but their PR has been blocked for "security concerns" [2], and it's probably too old now. I think we need to make this configurable for each listener. > The event flattening approach in [4] doesn't seem to help much here, as it > replaces roughly 150 classes with about 150 switch branches. That wasn't the intent. The idea was rather to define transformation rules per event attribute, e.g. if an event has the TABLE_METADATA attribute then we would apply some specific transformation rule to "prune" the attribute from sensitive data or things like that. This idea has received a PoC [3] a while ago, but I'm afraid the PoC is obsolete by now. iirc Adnan was supposed to provide us with a design proposal. > I am considering using the Quarkus event-bus [5] for the > PolarisEventListener implementations. Very good idea :-) My hot take here is that we will need multiple listeners ASAP, because the JDBC listener will become kind of mandatory now, and should probably be "on" by default. This old ML thread is relevant: [4]. This PR also outlines a few good ideas: [5]. Lastly, you can also have a look at what Nessie did [6] (although Nessie has a complex delivery logic). So I'd suggest to tackle these 3 problems first: - Configurable event types per listener - The "transformation" layer (it could start simple, just "no transformation at all") - Multiple listeners We could probably parallelize the efforts to some extent. Thanks, Alex [1]: https://github.com/apache/polaris/issues/2630 [2]: https://github.com/apache/polaris/pull/2962 [3]: https://github.com/apache/polaris/pull/3217 [4]: https://lists.apache.org/thread/wj14coln1k4f9l8dmm21ktj2ql787gvc [5]: https://github.com/apache/polaris/pull/3442 [6]: https://github.com/projectnessie/nessie/blob/fe7fbb3cf2c0b573acd2d773f2d62ae67fef153d/events/quarkus/src/main/java/org/projectnessie/events/quarkus/QuarkusEventService.java On Thu, Mar 5, 2026 at 1:37 PM Romain Manni-Bucau <[email protected]> wrote: > > Hi, > > My 2cts would be that some industry tried this webhook like implementation > and it works while not that adopted. > Since iceberg is quite closely bound to eventing for ingestion in general > it can make sense to bypass REST (which doesnt scale by design until you do > adopt more relevant design like JSON-RPC which has bulk built-in) an just > go asyncapi and support messaging by default - there I totally agree with > JB that internal can be external in a lot of cases. > Would also enable to use message API (MP) instead of an ad-hoc API which > doesn't integrate with anything existing - even plain local CDI bus - > instead of quarkus one which can stay in a niche in terms of > ecosystem/adoption/end user knowledge. > > In terms of mappig I would just go model -> JSON/Avro with the schema > exposed and documented with every release (optionally sync-ed in a schema > registry) and be it, will enable external case as well as internal one with > a database which does support JSON column type (almost all modern ones). > > So overall keep it simple. > > Just my 2 cts > > Romain Manni-Bucau > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > <https://dotnetbirdie.github.io/> | Blog <https://rmannibucau.github.io/> | > Old > Blog <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > <https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064> > Javaccino founder (Java/.NET service - contact via linkedin) > > > Le jeu. 5 mars 2026 à 12:53, Jean-Baptiste Onofré <[email protected]> a > écrit : > > > Hi Nandor, > > > > I will take a look. Generally speaking, I wonder if we should implement a > > kind of internal event bus that supports "event dispatching." > > > > For example, I previously created a framework called Apache Karaf Decanter > > ( > > https://github.com/apache/karaf-decanter) based on this concept. It allows > > for multiple event appenders, which could provide a flexible way to > > collect, process, and dispatch events. > > > > Just a thought. > > > > Regards, > > JB > > > > On Thu, Mar 5, 2026 at 6:04 AM Nándor Kollár <[email protected]> > > wrote: > > > > > Hi All, > > > > > > I recently reviewed how Polaris events are persisted, which is a > > > prerequisite for implementing both the Iceberg event proposal [1] and the > > > event API in Polaris [2]. I identified two areas for improvement: it > > > appears that we only persist two types of events, and Polaris allows > > only a > > > single event listener. Because of this limitation, we cannot, for > > example, > > > persist events in the database *and* send them to CloudWatch at the same > > > time. > > > > > > Regarding the first problem, it seems that the transformation of most > > > service events to event entities is missing from > > > PolarisPersistenceEventListener [3]. Supporting each service event would > > > likely require implementing a transformation for every event type, which > > > could result in more than 150 separate methods or switch cases. The event > > > flattening approach in [4] doesn't seem to help much here, as it replaces > > > roughly 150 classes with about 150 switch branches. At the moment, I do > > not > > > yet have a good idea how we could simplify this transformation. In the > > > worst case, we would need to implement a large number of branches. > > > > > > As for the second problem, I am considering using the Quarkus event-bus > > [5] > > > for the PolarisEventListener implementations. This would hopefully keep > > the > > > listeners configurable, allowing individual listeners to be enabled or > > > disabled while also making it possible for multiple listeners to consume > > > Polaris events simultaneously. > > > > > > What do you think? > > > > > > Nandor > > > > > > [1] > > > > > > > > https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?tab=t.0 > > > [2] https://github.com/apache/polaris/pull/3924 > > > [3] > > > > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/events/listeners/PolarisPersistenceEventListener.java#L39 > > > [4] https://lists.apache.org/thread/xonxwf9b38t9cxo841r0hn1b34plf7og > > > [5] https://quarkus.io/guides/reactive-event-bus > > > > >
