Hi Yufei, I fully agree with your suggestions. The current Events API is structured like a listener API, but also serves the purpose of an interceptor API: introducing a new API is the best way to solve this functional mismatch. I'm looking forward to that discussion.
On the "pure" multiple listeners topic, leaving aside interceptor-like use cases, do we have agreement to proceed with my proposal? Thanks, Alex On Sat, Nov 15, 2025 at 6:47 PM Yufei Gu <[email protected]> wrote: > > To move forward, we may decouple two debates. Supporting multiple listeners > is generally uncontroversial, while whether events should act as > interception points has mixed thoughts. > > We may proceed with async multiple listeners as an enhancement. These > listeners should be notification-only, independent, and non-blocking. > > For those who rely on sync/blocking hooks, we can open a dedicated > discussion about creating a separate SPI for policy or server-interception > logic, which gives us a clean and explicit place for policy enforcement, or > general data governance use cases, without overloading the event API. > > Yufei > > > On Fri, Nov 14, 2025 at 1:43 PM Adnan Hemani > <[email protected]> wrote: > > > Hi all, > > > > +1 to Eric's statements: "Events as they stand today can and do function as > > injection points for arbitrary code" and "if we intend to change the > > functionality / scope of events we > > should do that in clear terms and with a well-considered design" > > > > I likely will not be supporting any proposal where the current synchronous > > functionality is no longer allowed as part of the events feature (unless > > there is compelling evidence to back up the claims that this functionality > > cannot be used today or will not be useful in the future), although I am > > still in heavy support of adding functionality for multiple event > > listeners. > > > > Best, > > Adnan Hemani > > > > On Thu, Nov 13, 2025 at 8:01 AM Alexandre Dutra <[email protected]> wrote: > > > > > > Why is this behavior change being coupled with a proposal to support > > > multiple event listeners, anyway? > > > > > > My proposal couples both because 1) synchronous listeners could > > > monopolize the event loop (cf. AWS CloudWatch sink), and 2) a > > > composite listener is imho not an elegant solution to the multiple > > > listeners problem. > > > > > > Thanks, > > > Alex > > > > > > On Thu, Nov 13, 2025 at 12:35 PM Eric Maynard <[email protected]> > > > wrote: > > > > > > > > It's true that the listener methods return void -- I think they should > > > > probably have always returned events -- but that doesn't mean you can't > > > do > > > > interesting things with the current (blocking) methods. Besides the > > > > already-given example of throwing an exception from within a listener, > > > some > > > > listener methods take an immutable parameter that has mutable objects > > > held > > > > within. Michael rightly points out above that even an > > accidentally-thrown > > > > exception may stop processing, which in some cases can be valuable. > > > > > > > > Why is this behavior change being coupled with a proposal to support > > > > multiple event listeners, anyway? Yufei brought the issue of multiple > > > > listeners up on the original PR > > > > <https://github.com/apache/polaris/pull/922#discussion_r1985409710> > > and > > > > IIRC there were some followup discussions about a way to wrap multiple > > > > listeners in one. I think it can be done, and might be useful. But I > > > > suspect that this can be done without losing functionality, and further > > > it > > > > seems that if we intend to change the functionality / scope of events > > we > > > > should do that in clear terms and with a well-considered design. > > > > > > > > --EM > > > > > > > > On Thu, Nov 13, 2025 at 4:24 AM Alexandre Dutra <[email protected]> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > > Events as they stand today can and do function as injection points > > > for > > > > > arbitrary code. > > > > > > > > > > I don't think so? How would one "hook into various parts of the > > > > > Polaris functionality" with this API? All listener methods take > > > > > immutable parameters and return nothing: this opinionated design > > > > > forbids "arbitrary code" to be injected. I cannot, for instance, > > > > > intercept some endpoint and modify the REST response to my needs. > > > > > > > > > > Whatever the original intent of this API was, it's a fact that in its > > > > > current state it is not suitable for implementing interceptors. My > > > > > proposal doesn't change the statu quo, only makes it official. > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > On Thu, Nov 13, 2025 at 2:32 AM Eric Maynard < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > > I mentioned Java and React because they're very clear examples of > > how > > > > > > the term "event listener" is generally understood, and Spark only > > > because > > > > > > it was referenced by name in the initial proposal email. Spark's > > > > > > listeners *have > > > > > > to be* async due to the nature of Spark as something that runs > > across > > > > > > threads (/ VMs / processes). That said, it's still possible if not > > > common > > > > > > to implement a Spark listener that does something like cancel > > > > > long-running > > > > > > jobs. This is neither here nor there, though -- my point is not > > that > > > we > > > > > > should copy Spark or React, but rather that to say events are not > > > > > intended > > > > > > as injection points is perhaps revisionist history. > > > > > > > > > > > > The original design proposal was sent out with this passage: > > > > > > > > > > > > It would be useful to add a generic event listener interface to > > > Polaris, > > > > > > consistent with other OSS projects. Users of the project may > > require > > > > > > additional functionality that doesn't have a clear enough value > > > > > proposition > > > > > > to be in OSS. Instead, there can be event listeners that let you > > hook > > > > > into > > > > > > various parts of the Polaris functionality (i.e. "before table > > > commit") > > > > > > without OSS prescribing the limits of the extra functionality. > > > > > > > > > > > > > > > > > > Events as they stand today can and do function as injection points > > > for > > > > > > arbitrary code. It's my understanding that they were designed to > > > serve > > > > > this > > > > > > role and that this aspect of the design was in part a reaction to > > > debates > > > > > > happening at the time around API stability, refactors, and > > extension > > > > > > points. We were still dreaming about killing CallContext. The > > Iceberg > > > > > > events proposal did not yet exist. And at least my own endorsement > > of > > > > > > the proposal / PR was contingent on this functionality. > > > > > > > > > > > > If we as a community align on removing this functionality from the > > > events > > > > > > framework, we should be intentional about that. Currently the > > > (re)design > > > > > of > > > > > > events seems to be taking place across many email threads without > > > clear > > > > > > arguments about what the framework should support or why. > > > > > > > > > > > > --EM > > > > > > > > > > > > On Wed, Nov 12, 2025 at 1:08 PM Alexandre Dutra <[email protected] > > > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Several people mentioned Spark event listeners. After looking > > more > > > > > > > closely into this feature, I think it actually looks very similar > > > to > > > > > > > what I'm proposing: > > > > > > > > > > > > > > - The primary intended use cases that I could find are : > > monitoring > > > > > > > job progress, tracking stages and task completion, gathering > > > metrics > > > > > > > [1]. > > > > > > > > > > > > > > - Events are distributed asynchronously via an internal component > > > > > > > called SparkListenerBus [2], which manages an event bus and a > > > > > > > single-threaded event queue. > > > > > > > > > > > > > > - The API consists solely of methods that return void without > > > checked > > > > > > > exceptions: IOW, the API wasn't designed to allow for listeners > > to > > > > > > > interact with the server (other than throwing an unchecked > > > exception, > > > > > > > of course). > > > > > > > > > > > > > > - Afaict if a listener throws, the bus catches the exception and > > > moves > > > > > on. > > > > > > > > > > > > > > I'm not an expert in this topic so I might be wrong here, but the > > > > > > > suggestion that Spark event listeners were designed to allow > > > listeners > > > > > > > to modify Spark's behavior doesn't look accurate to me. > > > > > > > > > > > > > > Thanks, > > > > > > > Alex > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > https://www.ibm.com/think/insights/apache-spark-monitoring-using-listeners-and-data-quality-libraries > > > > > > > [2]: > > > > > > > > > > > > > > > > > https://spark.apache.org/docs/latest/api/java/org/apache/spark/scheduler/SparkListenerBus.html > > > > > > > > > > > > > > On Tue, Nov 11, 2025 at 1:30 PM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Eric, > > > > > > > > > > > > > > > > I would very much prefer not to use the event listener SPI as a > > > > > means to > > > > > > > > control the operation of the Polaris Server. > > > > > > > > > > > > > > > > More specifically, I believe that any error / exception in an > > > event > > > > > > > > listener should not affect the processing of the request. > > > > > > > > > > > > > > > > If we need custom callbacks to control some aspects of the > > server > > > > > > > > behaviour, let's define a dedicated SPI for that, but, IMHO, it > > > > > should be > > > > > > > > outside the scope of events. WDYT? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 8:55 PM Eric Maynard < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > In fact, shouldn’t it be exclusively a listener’s decision on > > > > > whether > > > > > > > an > > > > > > > > > event is handled in a blocking way or not? As was noted in a > > > past > > > > > > > thread on > > > > > > > > > events, much of the utility of the event framework comes from > > > the > > > > > > > ability > > > > > > > > > to introduce custom logic and hooks into the normal operation > > > of > > > > > > > Polaris. > > > > > > > > > > > > > > > > > > If you wish, for example, to prevent the creation of more > > than > > > 1k > > > > > > > tables > > > > > > > > > with some given prefix, you can do so using a listener. If > > the > > > > > event > > > > > > > which > > > > > > > > > might trigger that logic becomes non-blocking, you would no > > > longer > > > > > be > > > > > > > able > > > > > > > > > to block/fail the create table request. > > > > > > > > > > > > > > > > > > I think maybe it’s the name “event”, but we seem to keep > > > conflating > > > > > > > these > > > > > > > > > hooks with the iceberg events or auditing events when they > > are > > > not > > > > > > > exactly > > > > > > > > > the same thing. > > > > > > > > > > > > > > > > > > —EM > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 8:47 PM Adnan Hemani > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > > > > > Thanks for writing down the proposal for this! As I had > > > > > previously > > > > > > > > > > suggested this when implementing the Persistence of Polaris > > > > > Events > > > > > > > > > > <https://github.com/apache/polaris/pull/1844>, I am > > > obviously > > > > > very > > > > > > > much > > > > > > > > > in > > > > > > > > > > favor of doing this :) > > > > > > > > > > > > > > > > > > > > A few questions I have regarding your vision of how we > > should > > > > > > > implement > > > > > > > > > > this: > > > > > > > > > > * Are you envisioning anything for being able to make > > > > > dependencies > > > > > > > > > between > > > > > > > > > > event listeners? Or are we taking a set direction that > > Event > > > > > > > Listeners > > > > > > > > > > should be independent of each other? > > > > > > > > > > * In some listeners we have the ability to make events > > > emission > > > > > > > > > synchronous > > > > > > > > > > [example > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java#L186 > > > > > > > > > > >]. > > > > > > > > > > How do we plan to support/advise (or not...) that with the > > > > > > > introduction > > > > > > > > > > of @Blocking annotations. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 11:29 AM Yufei Gu < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. It's overall a good idea to have > > > async > > > > > event > > > > > > > > > > > listeners so that they are not blocking each other. > > > > > > > > > > > > > > > > > > > > > > One downside of the async ones is that event order isn't > > > > > > > deterministic. > > > > > > > > > > > For example, event listeners of Spark need the order to > > > > > understand > > > > > > > the > > > > > > > > > > > execution semantics. I think Polaris is fine with that, > > > given > > > > > the > > > > > > > ts of > > > > > > > > > > > each event is generated by Polaris. The downstream can > > > still > > > > > > > figure out > > > > > > > > > > the > > > > > > > > > > > order. > > > > > > > > > > > > > > > > > > > > > > Thanks Pierre for sharing, I think any I/O-bound or > > > potentially > > > > > > > slow > > > > > > > > > > > listener should be annotated with @Blocking. That ensures > > > we > > > > > keep > > > > > > > the > > > > > > > > > > event > > > > > > > > > > > loop responsive and avoid impacting REST latency. > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 9:43 AM Alexandre Dutra < > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > Answering the questions above: > > > > > > > > > > > > > > > > > > > > > > > > > However, we can easily make sure that we use > > Quarkus's > > > > > SmallRye > > > > > > > > > Fault > > > > > > > > > > > > Tolerance > > > > > > > > > > > > > > > > > > > > > > > > Yes, that was my idea. It's not so much the bus itself > > > that > > > > > > > needs to > > > > > > > > > > > > be fault tolerant, but the receiving end, that is, the > > > > > > > listeners. A > > > > > > > > > > > > listener can fail for a variety of reasons (e.g. remote > > > > > broker > > > > > > > > > > > > unavailable), it would be nice to be able to backoff > > and > > > > > retry > > > > > > > > > > > > automatically. > > > > > > > > > > > > > > > > > > > > > > > > > Since the Vert.x event bus runs on event-loop threads > > > [...] > > > > > > > could > > > > > > > > > > > > blocking or slow event listeners potentially stall REST > > > > > requests > > > > > > > and > > > > > > > > > > > impact > > > > > > > > > > > > latency? > > > > > > > > > > > > > > > > > > > > > > > > What Pierre said: this could indeed happen, but it's > > > > > possible to > > > > > > > > > > > > annotate the receiving end with @Blocking, in which > > > case, the > > > > > > > > > listener > > > > > > > > > > > > will be invoked in a separate pool. > > > > > > > > > > > > > > > > > > > > > > > > > With asynchronous event listeners, is there a > > > guarantee of > > > > > > > delivery > > > > > > > > > > to > > > > > > > > > > > > all listeners for a given event? > > > > > > > > > > > > > > > > > > > > > > > > If I understand the question correctly: with > > asynchronous > > > > > > > delivery, a > > > > > > > > > > > > slow or failing listener wouldn't impact the delivery > > of > > > the > > > > > same > > > > > > > > > > > > event to other listeners. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 10:12 AM Pierre Laporte < > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the proposal, Alex. This sounds like a > > > great > > > > > > > > > improvement. > > > > > > > > > > > > > > > > > > > > > > > > > > @Yufei As per Quarkus documentation, slow event > > > listeners > > > > > > > should be > > > > > > > > > > > > marked > > > > > > > > > > > > > with @Blocking so that they are not run on the event > > > loop > > > > > > > threads. > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > Pierre > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 8, 2025 at 2:14 AM Michael Collado < > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > With asynchronous event listeners, is there a > > > guarantee > > > > > of > > > > > > > > > delivery > > > > > > > > > > > to > > > > > > > > > > > > all > > > > > > > > > > > > > > listeners for a given event? The downside of > > > synchronous > > > > > > > > > listeners > > > > > > > > > > is > > > > > > > > > > > > that > > > > > > > > > > > > > > everything is serial, but also if something fails, > > > > > processing > > > > > > > > > > stops. > > > > > > > > > > > > This > > > > > > > > > > > > > > feels important for auditing purposes, though less > > > > > important > > > > > > > for > > > > > > > > > > > other > > > > > > > > > > > > > > cases. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 2:28 PM Yufei Gu < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Alex and Adam. One concern I have is > > about > > > the > > > > > > > shared > > > > > > > > > > > runtime > > > > > > > > > > > > > > > thread pool. > > > > > > > > > > > > > > > Since the Vert.x event bus runs on event-loop > > > threads > > > > > that > > > > > > > are > > > > > > > > > > also > > > > > > > > > > > > used > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > Quarkus’ reactive REST endpoints, could blocking > > or > > > > > slow > > > > > > > event > > > > > > > > > > > > listeners > > > > > > > > > > > > > > > potentially stall REST requests and impact > > latency? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:25 AM Adam Christian < > > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think that this would be a great enhancement. > > > > > Thanks > > > > > > > for > > > > > > > > > > > > proposing > > > > > > > > > > > > > > it! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The only concern I would have is around > > > > > fault-tolerance. > > > > > > > From > > > > > > > > > > > what > > > > > > > > > > > > I > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > tell, from the Quarkus documentation, the > > Quarkus > > > > > event > > > > > > > bus > > > > > > > > > > uses > > > > > > > > > > > > Vert.x > > > > > > > > > > > > > > > > EventBus which does not guarantee message > > > delivery if > > > > > > > failure > > > > > > > > > > of > > > > > > > > > > > > part > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > the event bus occurs [1]. However, we can > > easily > > > make > > > > > > > sure > > > > > > > > > that > > > > > > > > > > > we > > > > > > > > > > > > use > > > > > > > > > > > > > > > > Quarkus's SmallRye Fault Tolerance [2]. Is my > > > rough > > > > > > > > > > understanding > > > > > > > > > > > > > > inline > > > > > > > > > > > > > > > > with your proposal? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Go community, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Adam > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > https://vertx.io/docs/apidocs/io/vertx/core/eventbus/EventBus.html > > > > > > > > > > > > > > > > [2]: > > > > > https://quarkus.io/guides/smallrye-fault-tolerance > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:49 AM Alexandre > > Dutra < > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to propose an enhancement to our > > > existing > > > > > > > events > > > > > > > > > > > > feature: > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > ability to support multiple listeners. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, only a single listener can be > > > active at > > > > > a > > > > > > > time, > > > > > > > > > > > which > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > quite limiting. For example, we might need to > > > > > persist > > > > > > > > > events > > > > > > > > > > > for > > > > > > > > > > > > > > audit > > > > > > > > > > > > > > > > > purposes and simultaneously send them to a > > > message > > > > > > > queue > > > > > > > > > for > > > > > > > > > > > > > > > > > optimization. With the current setup, this > > > isn't > > > > > easily > > > > > > > > > > > > achievable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > While a composite listener could be created, > > it > > > > > feels > > > > > > > like > > > > > > > > > a > > > > > > > > > > > less > > > > > > > > > > > > > > > > > elegant solution, and the delivery would be > > > > > strictly > > > > > > > > > serial, > > > > > > > > > > > > > > > > > processing one listener after another. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My suggestion is to leverage Quarkus internal > > > > > event bus > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) There will be one central event emitter > > > > > responsible > > > > > > > for > > > > > > > > > > > > publishing > > > > > > > > > > > > > > > > > events to the bus. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We will have zero to N listeners, each > > > > > independently > > > > > > > > > > > watching > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > event bus for relevant events. They will be > > > > > discovered > > > > > > > by > > > > > > > > > > CDI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3) We could apply filters to each listener, > > > e.g. > > > > > > > listener A > > > > > > > > > > > > listens > > > > > > > > > > > > > > > > > for event types X and Y, listener B only > > > listens to > > > > > > > event > > > > > > > > > > type > > > > > > > > > > > Y. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4) This approach would ensure fully > > > asynchronous > > > > > > > delivery > > > > > > > > > of > > > > > > > > > > > > events > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > all interested listeners. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5) Fault-tolerance could also be easily > > > implemented > > > > > > > (event > > > > > > > > > > > > delivery > > > > > > > > > > > > > > > > > retries, timeouts, etc.). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What do you all think? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > https://quarkus.io/guides/reactive-event-bus > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
