Hi Yufei, I have to disagree on the "interceptor" point. As I commented previously, I do believe that event listeners should not be used as interceptors. Specifically, exceptions and return values from event listeners should not affect the processing of the request in Polaris (i.e. requests should work as if no event listeners are present).
I do not mind adding a separate interceptor SPI with well-defined proceed/abort semantics. My objection relates only to overloading current event listeners with the assumption that they can influence request processing in the server. Thanks, Dmitri. On Sat, Nov 15, 2025 at 4:47 PM Yufei Gu <[email protected]> wrote: > To move forward, we may decouple two debates. Supporting multiple listeners > is generally uncontroversial, while whether events should act as > interception points has mixed thoughts. > > We may proceed with async multiple listeners as an enhancement. These > listeners should be notification-only, independent, and non-blocking. > > For those who rely on sync/blocking hooks, we can open a dedicated > discussion about creating a separate SPI for policy or server-interception > logic, which gives us a clean and explicit place for policy enforcement, or > general data governance use cases, without overloading the event API. > > Yufei > > > On Fri, Nov 14, 2025 at 1:43 PM Adnan Hemani > <[email protected]> wrote: > > > Hi all, > > > > +1 to Eric's statements: "Events as they stand today can and do function > as > > injection points for arbitrary code" and "if we intend to change the > > functionality / scope of events we > > should do that in clear terms and with a well-considered design" > > > > I likely will not be supporting any proposal where the current > synchronous > > functionality is no longer allowed as part of the events feature (unless > > there is compelling evidence to back up the claims that this > functionality > > cannot be used today or will not be useful in the future), although I am > > still in heavy support of adding functionality for multiple event > > listeners. > > > > Best, > > Adnan Hemani > > > > On Thu, Nov 13, 2025 at 8:01 AM Alexandre Dutra <[email protected]> > wrote: > > > > > > Why is this behavior change being coupled with a proposal to support > > > multiple event listeners, anyway? > > > > > > My proposal couples both because 1) synchronous listeners could > > > monopolize the event loop (cf. AWS CloudWatch sink), and 2) a > > > composite listener is imho not an elegant solution to the multiple > > > listeners problem. > > > > > > Thanks, > > > Alex > > > > > > On Thu, Nov 13, 2025 at 12:35 PM Eric Maynard < > [email protected]> > > > wrote: > > > > > > > > It's true that the listener methods return void -- I think they > should > > > > probably have always returned events -- but that doesn't mean you > can't > > > do > > > > interesting things with the current (blocking) methods. Besides the > > > > already-given example of throwing an exception from within a > listener, > > > some > > > > listener methods take an immutable parameter that has mutable objects > > > held > > > > within. Michael rightly points out above that even an > > accidentally-thrown > > > > exception may stop processing, which in some cases can be valuable. > > > > > > > > Why is this behavior change being coupled with a proposal to support > > > > multiple event listeners, anyway? Yufei brought the issue of multiple > > > > listeners up on the original PR > > > > <https://github.com/apache/polaris/pull/922#discussion_r1985409710> > > and > > > > IIRC there were some followup discussions about a way to wrap > multiple > > > > listeners in one. I think it can be done, and might be useful. But I > > > > suspect that this can be done without losing functionality, and > further > > > it > > > > seems that if we intend to change the functionality / scope of events > > we > > > > should do that in clear terms and with a well-considered design. > > > > > > > > --EM > > > > > > > > On Thu, Nov 13, 2025 at 4:24 AM Alexandre Dutra <[email protected]> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > > Events as they stand today can and do function as injection > points > > > for > > > > > arbitrary code. > > > > > > > > > > I don't think so? How would one "hook into various parts of the > > > > > Polaris functionality" with this API? All listener methods take > > > > > immutable parameters and return nothing: this opinionated design > > > > > forbids "arbitrary code" to be injected. I cannot, for instance, > > > > > intercept some endpoint and modify the REST response to my needs. > > > > > > > > > > Whatever the original intent of this API was, it's a fact that in > its > > > > > current state it is not suitable for implementing interceptors. My > > > > > proposal doesn't change the statu quo, only makes it official. > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > On Thu, Nov 13, 2025 at 2:32 AM Eric Maynard < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > > I mentioned Java and React because they're very clear examples of > > how > > > > > > the term "event listener" is generally understood, and Spark only > > > because > > > > > > it was referenced by name in the initial proposal email. Spark's > > > > > > listeners *have > > > > > > to be* async due to the nature of Spark as something that runs > > across > > > > > > threads (/ VMs / processes). That said, it's still possible if > not > > > common > > > > > > to implement a Spark listener that does something like cancel > > > > > long-running > > > > > > jobs. This is neither here nor there, though -- my point is not > > that > > > we > > > > > > should copy Spark or React, but rather that to say events are not > > > > > intended > > > > > > as injection points is perhaps revisionist history. > > > > > > > > > > > > The original design proposal was sent out with this passage: > > > > > > > > > > > > It would be useful to add a generic event listener interface to > > > Polaris, > > > > > > consistent with other OSS projects. Users of the project may > > require > > > > > > additional functionality that doesn't have a clear enough value > > > > > proposition > > > > > > to be in OSS. Instead, there can be event listeners that let you > > hook > > > > > into > > > > > > various parts of the Polaris functionality (i.e. "before table > > > commit") > > > > > > without OSS prescribing the limits of the extra functionality. > > > > > > > > > > > > > > > > > > Events as they stand today can and do function as injection > points > > > for > > > > > > arbitrary code. It's my understanding that they were designed to > > > serve > > > > > this > > > > > > role and that this aspect of the design was in part a reaction to > > > debates > > > > > > happening at the time around API stability, refactors, and > > extension > > > > > > points. We were still dreaming about killing CallContext. The > > Iceberg > > > > > > events proposal did not yet exist. And at least my own > endorsement > > of > > > > > > the proposal / PR was contingent on this functionality. > > > > > > > > > > > > If we as a community align on removing this functionality from > the > > > events > > > > > > framework, we should be intentional about that. Currently the > > > (re)design > > > > > of > > > > > > events seems to be taking place across many email threads without > > > clear > > > > > > arguments about what the framework should support or why. > > > > > > > > > > > > --EM > > > > > > > > > > > > On Wed, Nov 12, 2025 at 1:08 PM Alexandre Dutra < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Several people mentioned Spark event listeners. After looking > > more > > > > > > > closely into this feature, I think it actually looks very > similar > > > to > > > > > > > what I'm proposing: > > > > > > > > > > > > > > - The primary intended use cases that I could find are : > > monitoring > > > > > > > job progress, tracking stages and task completion, gathering > > > metrics > > > > > > > [1]. > > > > > > > > > > > > > > - Events are distributed asynchronously via an internal > component > > > > > > > called SparkListenerBus [2], which manages an event bus and a > > > > > > > single-threaded event queue. > > > > > > > > > > > > > > - The API consists solely of methods that return void without > > > checked > > > > > > > exceptions: IOW, the API wasn't designed to allow for listeners > > to > > > > > > > interact with the server (other than throwing an unchecked > > > exception, > > > > > > > of course). > > > > > > > > > > > > > > - Afaict if a listener throws, the bus catches the exception > and > > > moves > > > > > on. > > > > > > > > > > > > > > I'm not an expert in this topic so I might be wrong here, but > the > > > > > > > suggestion that Spark event listeners were designed to allow > > > listeners > > > > > > > to modify Spark's behavior doesn't look accurate to me. > > > > > > > > > > > > > > Thanks, > > > > > > > Alex > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > https://www.ibm.com/think/insights/apache-spark-monitoring-using-listeners-and-data-quality-libraries > > > > > > > [2]: > > > > > > > > > > > > > > > > > > https://spark.apache.org/docs/latest/api/java/org/apache/spark/scheduler/SparkListenerBus.html > > > > > > > > > > > > > > On Tue, Nov 11, 2025 at 1:30 PM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Eric, > > > > > > > > > > > > > > > > I would very much prefer not to use the event listener SPI > as a > > > > > means to > > > > > > > > control the operation of the Polaris Server. > > > > > > > > > > > > > > > > More specifically, I believe that any error / exception in an > > > event > > > > > > > > listener should not affect the processing of the request. > > > > > > > > > > > > > > > > If we need custom callbacks to control some aspects of the > > server > > > > > > > > behaviour, let's define a dedicated SPI for that, but, IMHO, > it > > > > > should be > > > > > > > > outside the scope of events. WDYT? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 8:55 PM Eric Maynard < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > In fact, shouldn’t it be exclusively a listener’s decision > on > > > > > whether > > > > > > > an > > > > > > > > > event is handled in a blocking way or not? As was noted in > a > > > past > > > > > > > thread on > > > > > > > > > events, much of the utility of the event framework comes > from > > > the > > > > > > > ability > > > > > > > > > to introduce custom logic and hooks into the normal > operation > > > of > > > > > > > Polaris. > > > > > > > > > > > > > > > > > > If you wish, for example, to prevent the creation of more > > than > > > 1k > > > > > > > tables > > > > > > > > > with some given prefix, you can do so using a listener. If > > the > > > > > event > > > > > > > which > > > > > > > > > might trigger that logic becomes non-blocking, you would no > > > longer > > > > > be > > > > > > > able > > > > > > > > > to block/fail the create table request. > > > > > > > > > > > > > > > > > > I think maybe it’s the name “event”, but we seem to keep > > > conflating > > > > > > > these > > > > > > > > > hooks with the iceberg events or auditing events when they > > are > > > not > > > > > > > exactly > > > > > > > > > the same thing. > > > > > > > > > > > > > > > > > > —EM > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 8:47 PM Adnan Hemani > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > > > > > Thanks for writing down the proposal for this! As I had > > > > > previously > > > > > > > > > > suggested this when implementing the Persistence of > Polaris > > > > > Events > > > > > > > > > > <https://github.com/apache/polaris/pull/1844>, I am > > > obviously > > > > > very > > > > > > > much > > > > > > > > > in > > > > > > > > > > favor of doing this :) > > > > > > > > > > > > > > > > > > > > A few questions I have regarding your vision of how we > > should > > > > > > > implement > > > > > > > > > > this: > > > > > > > > > > * Are you envisioning anything for being able to make > > > > > dependencies > > > > > > > > > between > > > > > > > > > > event listeners? Or are we taking a set direction that > > Event > > > > > > > Listeners > > > > > > > > > > should be independent of each other? > > > > > > > > > > * In some listeners we have the ability to make events > > > emission > > > > > > > > > synchronous > > > > > > > > > > [example > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java#L186 > > > > > > > > > > >]. > > > > > > > > > > How do we plan to support/advise (or not...) that with > the > > > > > > > introduction > > > > > > > > > > of @Blocking annotations. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 11:29 AM Yufei Gu < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. It's overall a good idea to have > > > async > > > > > event > > > > > > > > > > > listeners so that they are not blocking each other. > > > > > > > > > > > > > > > > > > > > > > One downside of the async ones is that event order > isn't > > > > > > > deterministic. > > > > > > > > > > > For example, event listeners of Spark need the order to > > > > > understand > > > > > > > the > > > > > > > > > > > execution semantics. I think Polaris is fine with that, > > > given > > > > > the > > > > > > > ts of > > > > > > > > > > > each event is generated by Polaris. The downstream can > > > still > > > > > > > figure out > > > > > > > > > > the > > > > > > > > > > > order. > > > > > > > > > > > > > > > > > > > > > > Thanks Pierre for sharing, I think any I/O-bound or > > > potentially > > > > > > > slow > > > > > > > > > > > listener should be annotated with @Blocking. That > ensures > > > we > > > > > keep > > > > > > > the > > > > > > > > > > event > > > > > > > > > > > loop responsive and avoid impacting REST latency. > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 9:43 AM Alexandre Dutra < > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > Answering the questions above: > > > > > > > > > > > > > > > > > > > > > > > > > However, we can easily make sure that we use > > Quarkus's > > > > > SmallRye > > > > > > > > > Fault > > > > > > > > > > > > Tolerance > > > > > > > > > > > > > > > > > > > > > > > > Yes, that was my idea. It's not so much the bus > itself > > > that > > > > > > > needs to > > > > > > > > > > > > be fault tolerant, but the receiving end, that is, > the > > > > > > > listeners. A > > > > > > > > > > > > listener can fail for a variety of reasons (e.g. > remote > > > > > broker > > > > > > > > > > > > unavailable), it would be nice to be able to backoff > > and > > > > > retry > > > > > > > > > > > > automatically. > > > > > > > > > > > > > > > > > > > > > > > > > Since the Vert.x event bus runs on event-loop > threads > > > [...] > > > > > > > could > > > > > > > > > > > > blocking or slow event listeners potentially stall > REST > > > > > requests > > > > > > > and > > > > > > > > > > > impact > > > > > > > > > > > > latency? > > > > > > > > > > > > > > > > > > > > > > > > What Pierre said: this could indeed happen, but it's > > > > > possible to > > > > > > > > > > > > annotate the receiving end with @Blocking, in which > > > case, the > > > > > > > > > listener > > > > > > > > > > > > will be invoked in a separate pool. > > > > > > > > > > > > > > > > > > > > > > > > > With asynchronous event listeners, is there a > > > guarantee of > > > > > > > delivery > > > > > > > > > > to > > > > > > > > > > > > all listeners for a given event? > > > > > > > > > > > > > > > > > > > > > > > > If I understand the question correctly: with > > asynchronous > > > > > > > delivery, a > > > > > > > > > > > > slow or failing listener wouldn't impact the delivery > > of > > > the > > > > > same > > > > > > > > > > > > event to other listeners. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 10:12 AM Pierre Laporte < > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the proposal, Alex. This sounds like a > > > great > > > > > > > > > improvement. > > > > > > > > > > > > > > > > > > > > > > > > > > @Yufei As per Quarkus documentation, slow event > > > listeners > > > > > > > should be > > > > > > > > > > > > marked > > > > > > > > > > > > > with @Blocking so that they are not run on the > event > > > loop > > > > > > > threads. > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > Pierre > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 8, 2025 at 2:14 AM Michael Collado < > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > With asynchronous event listeners, is there a > > > guarantee > > > > > of > > > > > > > > > delivery > > > > > > > > > > > to > > > > > > > > > > > > all > > > > > > > > > > > > > > listeners for a given event? The downside of > > > synchronous > > > > > > > > > listeners > > > > > > > > > > is > > > > > > > > > > > > that > > > > > > > > > > > > > > everything is serial, but also if something > fails, > > > > > processing > > > > > > > > > > stops. > > > > > > > > > > > > This > > > > > > > > > > > > > > feels important for auditing purposes, though > less > > > > > important > > > > > > > for > > > > > > > > > > > other > > > > > > > > > > > > > > cases. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 2:28 PM Yufei Gu < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Alex and Adam. One concern I have is > > about > > > the > > > > > > > shared > > > > > > > > > > > runtime > > > > > > > > > > > > > > > thread pool. > > > > > > > > > > > > > > > Since the Vert.x event bus runs on event-loop > > > threads > > > > > that > > > > > > > are > > > > > > > > > > also > > > > > > > > > > > > used > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > Quarkus’ reactive REST endpoints, could > blocking > > or > > > > > slow > > > > > > > event > > > > > > > > > > > > listeners > > > > > > > > > > > > > > > potentially stall REST requests and impact > > latency? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:25 AM Adam Christian > < > > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think that this would be a great > enhancement. > > > > > Thanks > > > > > > > for > > > > > > > > > > > > proposing > > > > > > > > > > > > > > it! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The only concern I would have is around > > > > > fault-tolerance. > > > > > > > From > > > > > > > > > > > what > > > > > > > > > > > > I > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > tell, from the Quarkus documentation, the > > Quarkus > > > > > event > > > > > > > bus > > > > > > > > > > uses > > > > > > > > > > > > Vert.x > > > > > > > > > > > > > > > > EventBus which does not guarantee message > > > delivery if > > > > > > > failure > > > > > > > > > > of > > > > > > > > > > > > part > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > the event bus occurs [1]. However, we can > > easily > > > make > > > > > > > sure > > > > > > > > > that > > > > > > > > > > > we > > > > > > > > > > > > use > > > > > > > > > > > > > > > > Quarkus's SmallRye Fault Tolerance [2]. Is my > > > rough > > > > > > > > > > understanding > > > > > > > > > > > > > > inline > > > > > > > > > > > > > > > > with your proposal? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Go community, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Adam > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > https://vertx.io/docs/apidocs/io/vertx/core/eventbus/EventBus.html > > > > > > > > > > > > > > > > [2]: > > > > > https://quarkus.io/guides/smallrye-fault-tolerance > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:49 AM Alexandre > > Dutra < > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to propose an enhancement to our > > > existing > > > > > > > events > > > > > > > > > > > > feature: > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > ability to support multiple listeners. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, only a single listener can be > > > active at > > > > > a > > > > > > > time, > > > > > > > > > > > which > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > quite limiting. For example, we might need > to > > > > > persist > > > > > > > > > events > > > > > > > > > > > for > > > > > > > > > > > > > > audit > > > > > > > > > > > > > > > > > purposes and simultaneously send them to a > > > message > > > > > > > queue > > > > > > > > > for > > > > > > > > > > > > > > > > > optimization. With the current setup, this > > > isn't > > > > > easily > > > > > > > > > > > > achievable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > While a composite listener could be > created, > > it > > > > > feels > > > > > > > like > > > > > > > > > a > > > > > > > > > > > less > > > > > > > > > > > > > > > > > elegant solution, and the delivery would be > > > > > strictly > > > > > > > > > serial, > > > > > > > > > > > > > > > > > processing one listener after another. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My suggestion is to leverage Quarkus > internal > > > > > event bus > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) There will be one central event emitter > > > > > responsible > > > > > > > for > > > > > > > > > > > > publishing > > > > > > > > > > > > > > > > > events to the bus. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We will have zero to N listeners, each > > > > > independently > > > > > > > > > > > watching > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > event bus for relevant events. They will be > > > > > discovered > > > > > > > by > > > > > > > > > > CDI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3) We could apply filters to each listener, > > > e.g. > > > > > > > listener A > > > > > > > > > > > > listens > > > > > > > > > > > > > > > > > for event types X and Y, listener B only > > > listens to > > > > > > > event > > > > > > > > > > type > > > > > > > > > > > Y. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4) This approach would ensure fully > > > asynchronous > > > > > > > delivery > > > > > > > > > of > > > > > > > > > > > > events > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > all interested listeners. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5) Fault-tolerance could also be easily > > > implemented > > > > > > > (event > > > > > > > > > > > > delivery > > > > > > > > > > > > > > > > > retries, timeouts, etc.). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What do you all think? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > https://quarkus.io/guides/reactive-event-bus > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
