Hi all, It appears to me that everybody is waiting / blocked by the limitation of one listener max. I think we need to unlock multiple listeners as a preliminary step.
For that, I still think that the Quarkus event bus is a good fit. I don't think we need to implement our own bus. For the transformation layer: I think having different rules for each listener is not going to fly: the configuration is going to be very complicated. I suggest starting with rules that apply equally regardless of the listener. But again, this area deserves a whole design doc by itself. Thanks, Alex On Mon, Mar 9, 2026 at 8:47 AM Nándor Kollár <[email protected]> wrote: > > Thanks all for your responses, they are very helpful! > > Thanks Alex for pointing out the history around these topics. Maybe we > should revive the conversation on the PR related to the transformation > layer. I’ll take a look at the PRs and the discussion you mentioned - it > seems these topics already have some history. > > As both Adnan and Alex mentioned, the Quarkus event bus was already > discussed before, and the conclusion was that it’s probably a reasonable > approach. I'm happy to work on that. JB, do you think we should implement > our own event bus? What could be the advantage of it over using the Quarkus > event bus, which is already available? > > Using InMemoryBufferEventListener sounds reasonable for me too. I think we > can handle the event REST API of the proposal < > https://github.com/apache/polaris/pull/3924#issuecomment-4000072265> > relatively separated, although to provide a fully functional and useful > API, we'll need a solid persistence layer for each event type. > > Thanks, > Nandor > > Anand Kumar Sankaran via dev <[email protected]> ezt írta (időpont: > 2026. márc. 6., P, 19:02): > > > Hi Dmitri > > > > Sorry I meant local implementation. I did not raise a PR yet. I need the > > REST API first (so we can retrieve auditing data). Nandor said he will > > tackle the OSS implementation. > > > > Noted, about batch size. I will pull that in. > > > > — > > Anand > > > > From: Dmitri Bourlatchkov <[email protected]> > > Date: Friday, March 6, 2026 at 9:54 AM > > To: [email protected] <[email protected]> > > Cc: Adnan Hemani <[email protected]>, Anand Kumar Sankaran < > > [email protected]> > > Subject: Re: [DISCUSS] Polaris event persistence > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious< > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYf-ONIUIcNix9BMki6DiT8DnwNj1doA9UrGenSDVmx0ESnkYg5ZH78wqpx1wf9tDd5RgF9Gz2MCKiEqXLLNFUK8jCMQCQT-_5XTMiwmMXKLDGhnKtEGB$ > > > > > > > Hi Anand, > > > > When you wrote "implemented", did you mean a new metrics persistence PR or > > your local implementation? > > > > Using InMemoryBufferEventListener sounds quite valid to me. > > > > maxBufferSize 5 might be too low, I guess. I'd imagine under high load > > we'd want larger write batches for more efficiency at the Persistence > > layer. How about 100? (but I do not have any data to back it up). > > > > Cheers, > > Dmitri. > > > > On Fri, Mar 6, 2026 at 10:01 AM Anand Kumar Sankaran via dev < > > [email protected]<mailto:[email protected]>> wrote: > > Hi all, > > > > As discussed in this proposal< > > https://github.com/apache/polaris/pull/3924#issuecomment-4000072265< > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924*issuecomment-4000072265__;Iw!!Iz9xO38YGHZK!_PSDnN1pdaQTIY0tSTF4LxZX6-eMrbsL8MN108V54zwOQWY8W4iNnQNC9R3rZV8f-Uyy7wAa_TOwG8ezXBBPfh0-hLg3jpw$>>, > > for our auditing purposes, I implemented event persistence like this: > > > > It uses an in-memory buffering strategy provided by Apache Polaris' > > InMemoryBufferEventListener to batch events before flushing. > > > > events: > > persistence: > > enabled: true > > bufferTime: "5000ms" # Flush after 5 seconds > > maxBufferSize: 5 # Or after 5 events > > > > I implemented an audit event listener that extends > > InMemoryBufferEventListener, listens for many events, creates PolarisEvents > > and calls InMemoryBufferEventListener.processEvents() to buffer them. > > > > Is there a problem in doing this, while I wait for the fix discussed here? > > > > - > > Anand > > > > From: Adnan Hemani via dev <[email protected]<mailto: > > [email protected]>> > > Date: Thursday, March 5, 2026 at 3:44 PM > > To: [email protected]<mailto:[email protected]> < > > [email protected]<mailto:[email protected]>> > > Cc: Adnan Hemani <[email protected]<mailto: > > [email protected]>> > > Subject: Re: [DISCUSS] Polaris event persistence > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious< > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGomgiHL3-p1SKrEFd2u0L0PbXJYIhH5QU9smTKTCzcVqeKKZNKPQEKGrvUBiAmMs6ekIb2jF-5Fj2qqxJBC7mVAYFbTWswfMi3u1i4x_1gvoli0X1wkDYijxky$ > > > > > > > > > Hi all, > > > > Thanks for reviving the conversation regarding this feature, Nándor! My > > last recollection on this conversation was that we, as a community, had > > agreed to using async PolarisEventListeners to accommodate multiple event > > listeners at the same time. Using the Quarkus event-bus seems like a > > reasonable implementation idea based on my quick research. Nándor, if you > > would like to work on this, I would be glad to help in whatever way I can - > > I just don't have the bandwidth to own this feature in the immediate month > > or so. > > > > Regarding event flattening and information redactions, I mostly agree with > > Alex: flattening the events was not a panacea for the issue, but it should > > unlock our ability to apply mass transformations across event types. The > > PoC should show how we can achieve this along with the event flattening. I > > wasn't aware that people were waiting on me for a design proposal - my > > apologies if I accidentally made this promise. If anyone else would like to > > work on this, please do feel free to. > > > > The one thing to keep in mind is that, different event listeners will need > > different transformation patterns. For example, the Events Persistence > > Listener must fit the schema we merged earlier, which closely resembles the > > (proposed) Iceberg Event API's schema. But the AWS CloudWatch one will > > require much less transformation and can be used almost transparently minus > > the light redactions for security concerns. Additionally, we must tackle > > one more workstream: storing events in a database separate from the one > > that holds the metadata. As the volume of events increases, this will > > become a big concern if we turn on the JDBC listener by default. > > > > Best, > > Adnan Hemani > > > > On Thu, Mar 5, 2026 at 5:11 AM Alexandre Dutra <[email protected]<mailto: > > [email protected]>> wrote: > > > > > Hi Nándor, > > > > > > > it seems that the transformation of most service events to event > > > entities is missing from PolarisPersistenceEventListener > > > > > > Yes, unfortunately this has never been addressed, and yet it makes the > > > whole events feature pretty unusable. There is an old issue for that: > > > [1]. An external contributor tried to tackle it, but their PR has been > > > blocked for "security concerns" [2], and it's probably too old now. I > > > think we need to make this configurable for each listener. > > > > > > > The event flattening approach in [4] doesn't seem to help much here, as > > > it replaces roughly 150 classes with about 150 switch branches. > > > > > > That wasn't the intent. The idea was rather to define transformation > > > rules per event attribute, e.g. if an event has the TABLE_METADATA > > > attribute then we would apply some specific transformation rule to > > > "prune" the attribute from sensitive data or things like that. This > > > idea has received a PoC [3] a while ago, but I'm afraid the PoC is > > > obsolete by now. iirc Adnan was supposed to provide us with a design > > > proposal. > > > > > > > I am considering using the Quarkus event-bus [5] for the > > > PolarisEventListener implementations. > > > > > > Very good idea :-) My hot take here is that we will need multiple > > > listeners ASAP, because the JDBC listener will become kind of > > > mandatory now, and should probably be "on" by default. This old ML > > > thread is relevant: [4]. This PR also outlines a few good ideas: [5]. > > > Lastly, you can also have a look at what Nessie did [6] (although > > > Nessie has a complex delivery logic). > > > > > > So I'd suggest to tackle these 3 problems first: > > > > > > - Configurable event types per listener > > > - The "transformation" layer (it could start simple, just "no > > > transformation at all") > > > - Multiple listeners > > > > > > We could probably parallelize the efforts to some extent. > > > > > > Thanks, > > > Alex > > > > > > [1]: > > https://urldefense.com/v3/__https://github.com/apache/polaris/issues/2630__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2FpIKz0oA$ > > > [2]: > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/2962__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2H9hE7Blw$ > > > [3]: > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3217__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2F5KZr6PA$ > > > [4]: > > https://urldefense.com/v3/__https://lists.apache.org/thread/wj14coln1k4f9l8dmm21ktj2ql787gvc__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2GKvqlyMQ$ > > > [5]: > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3442__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2EW2gzBmw$ > > > [6]: > > > > > https://urldefense.com/v3/__https://github.com/projectnessie/nessie/blob/fe7fbb3cf2c0b573acd2d773f2d62ae67fef153d/events/quarkus/src/main/java/org/projectnessie/events/quarkus/QuarkusEventService.java__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2GF462LDQ$ > > > > > > On Thu, Mar 5, 2026 at 1:37 PM Romain Manni-Bucau <[email protected] > > <mailto:[email protected]>> > > > wrote: > > > > > > > > Hi, > > > > > > > > My 2cts would be that some industry tried this webhook like > > > implementation > > > > and it works while not that adopted. > > > > Since iceberg is quite closely bound to eventing for ingestion in > > general > > > > it can make sense to bypass REST (which doesnt scale by design until > > you > > > do > > > > adopt more relevant design like JSON-RPC which has bulk built-in) an > > just > > > > go asyncapi and support messaging by default - there I totally agree > > with > > > > JB that internal can be external in a lot of cases. > > > > Would also enable to use message API (MP) instead of an ad-hoc API > > which > > > > doesn't integrate with anything existing - even plain local CDI bus - > > > > instead of quarkus one which can stay in a niche in terms of > > > > ecosystem/adoption/end user knowledge. > > > > > > > > In terms of mappig I would just go model -> JSON/Avro with the schema > > > > exposed and documented with every release (optionally sync-ed in a > > schema > > > > registry) and be it, will enable external case as well as internal one > > > with > > > > a database which does support JSON column type (almost all modern > > ones). > > > > > > > > So overall keep it simple. > > > > > > > > Just my 2 cts > > > > > > > > Romain Manni-Bucau > > > > @rmannibucau < > > https://urldefense.com/v3/__https://x.com/rmannibucau__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2EHnuKKkg$> > > | .NET Blog > > > > < > > https://urldefense.com/v3/__https://dotnetbirdie.github.io/__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2H7YlfyJA$> > > | Blog < > > https://urldefense.com/v3/__https://rmannibucau.github.io/__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2GnTJtaOg$ > > > > > > | Old > > > > Blog < > > https://urldefense.com/v3/__http://rmannibucau.wordpress.com__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2HBf1wPWg$> > > | Github > > > > < > > https://urldefense.com/v3/__https://github.com/rmannibucau__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2HEFYo6JA$> > > | LinkedIn > > > > < > > https://urldefense.com/v3/__https://www.linkedin.com/in/rmannibucau__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2HKOB66yw$> > > | Book > > > > < > > > > > https://urldefense.com/v3/__https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2FplzwIdg$ > > > > > > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > > > > > > > > > > Le jeu. 5 mars 2026 à 12:53, Jean-Baptiste Onofré <[email protected] > > <mailto:[email protected]>> a > > > > écrit : > > > > > > > > > Hi Nandor, > > > > > > > > > > I will take a look. Generally speaking, I wonder if we should > > > implement a > > > > > kind of internal event bus that supports "event dispatching." > > > > > > > > > > For example, I previously created a framework called Apache Karaf > > > Decanter > > > > > ( > > > > > > > https://urldefense.com/v3/__https://github.com/apache/karaf-decanter__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2HGcYY-rw$) > > based on this concept. It > > > allows > > > > > for multiple event appenders, which could provide a flexible way to > > > > > collect, process, and dispatch events. > > > > > > > > > > Just a thought. > > > > > > > > > > Regards, > > > > > JB > > > > > > > > > > On Thu, Mar 5, 2026 at 6:04 AM Nándor Kollár < > > [email protected]<mailto:[email protected]>> > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > I recently reviewed how Polaris events are persisted, which is a > > > > > > prerequisite for implementing both the Iceberg event proposal [1] > > > and the > > > > > > event API in Polaris [2]. I identified two areas for improvement: > > it > > > > > > appears that we only persist two types of events, and Polaris > > allows > > > > > only a > > > > > > single event listener. Because of this limitation, we cannot, for > > > > > example, > > > > > > persist events in the database *and* send them to CloudWatch at the > > > same > > > > > > time. > > > > > > > > > > > > Regarding the first problem, it seems that the transformation of > > most > > > > > > service events to event entities is missing from > > > > > > PolarisPersistenceEventListener [3]. Supporting each service event > > > would > > > > > > likely require implementing a transformation for every event type, > > > which > > > > > > could result in more than 150 separate methods or switch cases. The > > > event > > > > > > flattening approach in [4] doesn't seem to help much here, as it > > > replaces > > > > > > roughly 150 classes with about 150 switch branches. At the moment, > > I > > > do > > > > > not > > > > > > yet have a good idea how we could simplify this transformation. In > > > the > > > > > > worst case, we would need to implement a large number of branches. > > > > > > > > > > > > As for the second problem, I am considering using the Quarkus > > > event-bus > > > > > [5] > > > > > > for the PolarisEventListener implementations. This would hopefully > > > keep > > > > > the > > > > > > listeners configurable, allowing individual listeners to be enabled > > > or > > > > > > disabled while also making it possible for multiple listeners to > > > consume > > > > > > Polaris events simultaneously. > > > > > > > > > > > > What do you think? > > > > > > > > > > > > Nandor > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?tab=t.0__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2GIF1lYYQ$ > > > > > > [2] > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3924__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2GKnASACw$ > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/events/listeners/PolarisPersistenceEventListener.java*L39__;Iw!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2Fy-tYnPg$ > > > > > > [4] > > https://urldefense.com/v3/__https://lists.apache.org/thread/xonxwf9b38t9cxo841r0hn1b34plf7og__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2Ho3l5v9A$ > > > > > > [5] > > https://urldefense.com/v3/__https://quarkus.io/guides/reactive-event-bus__;!!Iz9xO38YGHZK!6fIlpqfh2G1mjeV0q4vV61Pf3zpbNExOynGSXZ1r9c5mbGNL5g3qTTZXXy9wV_quehU6f8b25Zd1j2G2AXuATg$ > > > > > > > > > > > > > > > > > > > > > > -- > > Dmitri Bourlatchkov > > Senior Staff Software Engineer, Dremio > > Dremio.com< > > https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_PSDnN1pdaQTIY0tSTF4LxZX6-eMrbsL8MN108V54zwOQWY8W4iNnQNC9R3rZV8f-Uyy7wAa_TOwG8ezXBBPfh0-WwCVQ3s$> > > / Follow Us on LinkedIn< > > https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_PSDnN1pdaQTIY0tSTF4LxZX6-eMrbsL8MN108V54zwOQWY8W4iNnQNC9R3rZV8f-Uyy7wAa_TOwG8ezXBBPfh0-ruNGDg8$> > > / Get Started< > > https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_PSDnN1pdaQTIY0tSTF4LxZX6-eMrbsL8MN108V54zwOQWY8W4iNnQNC9R3rZV8f-Uyy7wAa_TOwG8ezXBBPfh0-m094JcU$ > > > > > > > [https://lh3.googleusercontent.com/d/1Ta6MlHHnksAk0fuUEm2yxgiTJbWMh6YM] > > > > The Agentic Lakehouse > > The only lakehouse built for agents, managed by agents > > > > > > > > -- > Kollár Nándor
