Hi all, +1 to Eric's statements: "Events as they stand today can and do function as injection points for arbitrary code" and "if we intend to change the functionality / scope of events we should do that in clear terms and with a well-considered design"
I likely will not be supporting any proposal where the current synchronous functionality is no longer allowed as part of the events feature (unless there is compelling evidence to back up the claims that this functionality cannot be used today or will not be useful in the future), although I am still in heavy support of adding functionality for multiple event listeners. Best, Adnan Hemani On Thu, Nov 13, 2025 at 8:01 AM Alexandre Dutra <[email protected]> wrote: > > Why is this behavior change being coupled with a proposal to support > multiple event listeners, anyway? > > My proposal couples both because 1) synchronous listeners could > monopolize the event loop (cf. AWS CloudWatch sink), and 2) a > composite listener is imho not an elegant solution to the multiple > listeners problem. > > Thanks, > Alex > > On Thu, Nov 13, 2025 at 12:35 PM Eric Maynard <[email protected]> > wrote: > > > > It's true that the listener methods return void -- I think they should > > probably have always returned events -- but that doesn't mean you can't > do > > interesting things with the current (blocking) methods. Besides the > > already-given example of throwing an exception from within a listener, > some > > listener methods take an immutable parameter that has mutable objects > held > > within. Michael rightly points out above that even an accidentally-thrown > > exception may stop processing, which in some cases can be valuable. > > > > Why is this behavior change being coupled with a proposal to support > > multiple event listeners, anyway? Yufei brought the issue of multiple > > listeners up on the original PR > > <https://github.com/apache/polaris/pull/922#discussion_r1985409710> and > > IIRC there were some followup discussions about a way to wrap multiple > > listeners in one. I think it can be done, and might be useful. But I > > suspect that this can be done without losing functionality, and further > it > > seems that if we intend to change the functionality / scope of events we > > should do that in clear terms and with a well-considered design. > > > > --EM > > > > On Thu, Nov 13, 2025 at 4:24 AM Alexandre Dutra <[email protected]> > wrote: > > > > > Hi all, > > > > > > > Events as they stand today can and do function as injection points > for > > > arbitrary code. > > > > > > I don't think so? How would one "hook into various parts of the > > > Polaris functionality" with this API? All listener methods take > > > immutable parameters and return nothing: this opinionated design > > > forbids "arbitrary code" to be injected. I cannot, for instance, > > > intercept some endpoint and modify the REST response to my needs. > > > > > > Whatever the original intent of this API was, it's a fact that in its > > > current state it is not suitable for implementing interceptors. My > > > proposal doesn't change the statu quo, only makes it official. > > > > > > Thanks, > > > Alex > > > > > > On Thu, Nov 13, 2025 at 2:32 AM Eric Maynard <[email protected] > > > > > wrote: > > > > > > > > I mentioned Java and React because they're very clear examples of how > > > > the term "event listener" is generally understood, and Spark only > because > > > > it was referenced by name in the initial proposal email. Spark's > > > > listeners *have > > > > to be* async due to the nature of Spark as something that runs across > > > > threads (/ VMs / processes). That said, it's still possible if not > common > > > > to implement a Spark listener that does something like cancel > > > long-running > > > > jobs. This is neither here nor there, though -- my point is not that > we > > > > should copy Spark or React, but rather that to say events are not > > > intended > > > > as injection points is perhaps revisionist history. > > > > > > > > The original design proposal was sent out with this passage: > > > > > > > > It would be useful to add a generic event listener interface to > Polaris, > > > > consistent with other OSS projects. Users of the project may require > > > > additional functionality that doesn't have a clear enough value > > > proposition > > > > to be in OSS. Instead, there can be event listeners that let you hook > > > into > > > > various parts of the Polaris functionality (i.e. "before table > commit") > > > > without OSS prescribing the limits of the extra functionality. > > > > > > > > > > > > Events as they stand today can and do function as injection points > for > > > > arbitrary code. It's my understanding that they were designed to > serve > > > this > > > > role and that this aspect of the design was in part a reaction to > debates > > > > happening at the time around API stability, refactors, and extension > > > > points. We were still dreaming about killing CallContext. The Iceberg > > > > events proposal did not yet exist. And at least my own endorsement of > > > > the proposal / PR was contingent on this functionality. > > > > > > > > If we as a community align on removing this functionality from the > events > > > > framework, we should be intentional about that. Currently the > (re)design > > > of > > > > events seems to be taking place across many email threads without > clear > > > > arguments about what the framework should support or why. > > > > > > > > --EM > > > > > > > > On Wed, Nov 12, 2025 at 1:08 PM Alexandre Dutra <[email protected]> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Several people mentioned Spark event listeners. After looking more > > > > > closely into this feature, I think it actually looks very similar > to > > > > > what I'm proposing: > > > > > > > > > > - The primary intended use cases that I could find are : monitoring > > > > > job progress, tracking stages and task completion, gathering > metrics > > > > > [1]. > > > > > > > > > > - Events are distributed asynchronously via an internal component > > > > > called SparkListenerBus [2], which manages an event bus and a > > > > > single-threaded event queue. > > > > > > > > > > - The API consists solely of methods that return void without > checked > > > > > exceptions: IOW, the API wasn't designed to allow for listeners to > > > > > interact with the server (other than throwing an unchecked > exception, > > > > > of course). > > > > > > > > > > - Afaict if a listener throws, the bus catches the exception and > moves > > > on. > > > > > > > > > > I'm not an expert in this topic so I might be wrong here, but the > > > > > suggestion that Spark event listeners were designed to allow > listeners > > > > > to modify Spark's behavior doesn't look accurate to me. > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > [1]: > > > > > > > > > https://www.ibm.com/think/insights/apache-spark-monitoring-using-listeners-and-data-quality-libraries > > > > > [2]: > > > > > > > > > https://spark.apache.org/docs/latest/api/java/org/apache/spark/scheduler/SparkListenerBus.html > > > > > > > > > > On Tue, Nov 11, 2025 at 1:30 PM Dmitri Bourlatchkov < > [email protected]> > > > > > wrote: > > > > > > > > > > > > Hi Eric, > > > > > > > > > > > > I would very much prefer not to use the event listener SPI as a > > > means to > > > > > > control the operation of the Polaris Server. > > > > > > > > > > > > More specifically, I believe that any error / exception in an > event > > > > > > listener should not affect the processing of the request. > > > > > > > > > > > > If we need custom callbacks to control some aspects of the server > > > > > > behaviour, let's define a dedicated SPI for that, but, IMHO, it > > > should be > > > > > > outside the scope of events. WDYT? > > > > > > > > > > > > Thanks, > > > > > > Dmitri. > > > > > > > > > > > > On Mon, Nov 10, 2025 at 8:55 PM Eric Maynard < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > In fact, shouldn’t it be exclusively a listener’s decision on > > > whether > > > > > an > > > > > > > event is handled in a blocking way or not? As was noted in a > past > > > > > thread on > > > > > > > events, much of the utility of the event framework comes from > the > > > > > ability > > > > > > > to introduce custom logic and hooks into the normal operation > of > > > > > Polaris. > > > > > > > > > > > > > > If you wish, for example, to prevent the creation of more than > 1k > > > > > tables > > > > > > > with some given prefix, you can do so using a listener. If the > > > event > > > > > which > > > > > > > might trigger that logic becomes non-blocking, you would no > longer > > > be > > > > > able > > > > > > > to block/fail the create table request. > > > > > > > > > > > > > > I think maybe it’s the name “event”, but we seem to keep > conflating > > > > > these > > > > > > > hooks with the iceberg events or auditing events when they are > not > > > > > exactly > > > > > > > the same thing. > > > > > > > > > > > > > > —EM > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 8:47 PM Adnan Hemani > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > Thanks for writing down the proposal for this! As I had > > > previously > > > > > > > > suggested this when implementing the Persistence of Polaris > > > Events > > > > > > > > <https://github.com/apache/polaris/pull/1844>, I am > obviously > > > very > > > > > much > > > > > > > in > > > > > > > > favor of doing this :) > > > > > > > > > > > > > > > > A few questions I have regarding your vision of how we should > > > > > implement > > > > > > > > this: > > > > > > > > * Are you envisioning anything for being able to make > > > dependencies > > > > > > > between > > > > > > > > event listeners? Or are we taking a set direction that Event > > > > > Listeners > > > > > > > > should be independent of each other? > > > > > > > > * In some listeners we have the ability to make events > emission > > > > > > > synchronous > > > > > > > > [example > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java#L186 > > > > > > > > >]. > > > > > > > > How do we plan to support/advise (or not...) that with the > > > > > introduction > > > > > > > > of @Blocking annotations. > > > > > > > > > > > > > > > > Best, > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 11:29 AM Yufei Gu < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks for the reply. It's overall a good idea to have > async > > > event > > > > > > > > > listeners so that they are not blocking each other. > > > > > > > > > > > > > > > > > > One downside of the async ones is that event order isn't > > > > > deterministic. > > > > > > > > > For example, event listeners of Spark need the order to > > > understand > > > > > the > > > > > > > > > execution semantics. I think Polaris is fine with that, > given > > > the > > > > > ts of > > > > > > > > > each event is generated by Polaris. The downstream can > still > > > > > figure out > > > > > > > > the > > > > > > > > > order. > > > > > > > > > > > > > > > > > > Thanks Pierre for sharing, I think any I/O-bound or > potentially > > > > > slow > > > > > > > > > listener should be annotated with @Blocking. That ensures > we > > > keep > > > > > the > > > > > > > > event > > > > > > > > > loop responsive and avoid impacting REST latency. > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 9:43 AM Alexandre Dutra < > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > Answering the questions above: > > > > > > > > > > > > > > > > > > > > > However, we can easily make sure that we use Quarkus's > > > SmallRye > > > > > > > Fault > > > > > > > > > > Tolerance > > > > > > > > > > > > > > > > > > > > Yes, that was my idea. It's not so much the bus itself > that > > > > > needs to > > > > > > > > > > be fault tolerant, but the receiving end, that is, the > > > > > listeners. A > > > > > > > > > > listener can fail for a variety of reasons (e.g. remote > > > broker > > > > > > > > > > unavailable), it would be nice to be able to backoff and > > > retry > > > > > > > > > > automatically. > > > > > > > > > > > > > > > > > > > > > Since the Vert.x event bus runs on event-loop threads > [...] > > > > > could > > > > > > > > > > blocking or slow event listeners potentially stall REST > > > requests > > > > > and > > > > > > > > > impact > > > > > > > > > > latency? > > > > > > > > > > > > > > > > > > > > What Pierre said: this could indeed happen, but it's > > > possible to > > > > > > > > > > annotate the receiving end with @Blocking, in which > case, the > > > > > > > listener > > > > > > > > > > will be invoked in a separate pool. > > > > > > > > > > > > > > > > > > > > > With asynchronous event listeners, is there a > guarantee of > > > > > delivery > > > > > > > > to > > > > > > > > > > all listeners for a given event? > > > > > > > > > > > > > > > > > > > > If I understand the question correctly: with asynchronous > > > > > delivery, a > > > > > > > > > > slow or failing listener wouldn't impact the delivery of > the > > > same > > > > > > > > > > event to other listeners. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 10:12 AM Pierre Laporte < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Thanks for the proposal, Alex. This sounds like a > great > > > > > > > improvement. > > > > > > > > > > > > > > > > > > > > > > @Yufei As per Quarkus documentation, slow event > listeners > > > > > should be > > > > > > > > > > marked > > > > > > > > > > > with @Blocking so that they are not run on the event > loop > > > > > threads. > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > Pierre > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 8, 2025 at 2:14 AM Michael Collado < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > With asynchronous event listeners, is there a > guarantee > > > of > > > > > > > delivery > > > > > > > > > to > > > > > > > > > > all > > > > > > > > > > > > listeners for a given event? The downside of > synchronous > > > > > > > listeners > > > > > > > > is > > > > > > > > > > that > > > > > > > > > > > > everything is serial, but also if something fails, > > > processing > > > > > > > > stops. > > > > > > > > > > This > > > > > > > > > > > > feels important for auditing purposes, though less > > > important > > > > > for > > > > > > > > > other > > > > > > > > > > > > cases. > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 2:28 PM Yufei Gu < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Alex and Adam. One concern I have is about > the > > > > > shared > > > > > > > > > runtime > > > > > > > > > > > > > thread pool. > > > > > > > > > > > > > Since the Vert.x event bus runs on event-loop > threads > > > that > > > > > are > > > > > > > > also > > > > > > > > > > used > > > > > > > > > > > > by > > > > > > > > > > > > > Quarkus’ reactive REST endpoints, could blocking or > > > slow > > > > > event > > > > > > > > > > listeners > > > > > > > > > > > > > potentially stall REST requests and impact latency? > > > > > > > > > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:25 AM Adam Christian < > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I think that this would be a great enhancement. > > > Thanks > > > > > for > > > > > > > > > > proposing > > > > > > > > > > > > it! > > > > > > > > > > > > > > > > > > > > > > > > > > > > The only concern I would have is around > > > fault-tolerance. > > > > > From > > > > > > > > > what > > > > > > > > > > I > > > > > > > > > > > > can > > > > > > > > > > > > > > tell, from the Quarkus documentation, the Quarkus > > > event > > > > > bus > > > > > > > > uses > > > > > > > > > > Vert.x > > > > > > > > > > > > > > EventBus which does not guarantee message > delivery if > > > > > failure > > > > > > > > of > > > > > > > > > > part > > > > > > > > > > > > of > > > > > > > > > > > > > > the event bus occurs [1]. However, we can easily > make > > > > > sure > > > > > > > that > > > > > > > > > we > > > > > > > > > > use > > > > > > > > > > > > > > Quarkus's SmallRye Fault Tolerance [2]. Is my > rough > > > > > > > > understanding > > > > > > > > > > > > inline > > > > > > > > > > > > > > with your proposal? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Go community, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Adam > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > https://vertx.io/docs/apidocs/io/vertx/core/eventbus/EventBus.html > > > > > > > > > > > > > > [2]: > > > https://quarkus.io/guides/smallrye-fault-tolerance > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:49 AM Alexandre Dutra < > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to propose an enhancement to our > existing > > > > > events > > > > > > > > > > feature: > > > > > > > > > > > > the > > > > > > > > > > > > > > > ability to support multiple listeners. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, only a single listener can be > active at > > > a > > > > > time, > > > > > > > > > which > > > > > > > > > > is > > > > > > > > > > > > > > > quite limiting. For example, we might need to > > > persist > > > > > > > events > > > > > > > > > for > > > > > > > > > > > > audit > > > > > > > > > > > > > > > purposes and simultaneously send them to a > message > > > > > queue > > > > > > > for > > > > > > > > > > > > > > > optimization. With the current setup, this > isn't > > > easily > > > > > > > > > > achievable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > While a composite listener could be created, it > > > feels > > > > > like > > > > > > > a > > > > > > > > > less > > > > > > > > > > > > > > > elegant solution, and the delivery would be > > > strictly > > > > > > > serial, > > > > > > > > > > > > > > > processing one listener after another. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My suggestion is to leverage Quarkus internal > > > event bus > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) There will be one central event emitter > > > responsible > > > > > for > > > > > > > > > > publishing > > > > > > > > > > > > > > > events to the bus. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We will have zero to N listeners, each > > > independently > > > > > > > > > watching > > > > > > > > > > the > > > > > > > > > > > > > > > event bus for relevant events. They will be > > > discovered > > > > > by > > > > > > > > CDI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3) We could apply filters to each listener, > e.g. > > > > > listener A > > > > > > > > > > listens > > > > > > > > > > > > > > > for event types X and Y, listener B only > listens to > > > > > event > > > > > > > > type > > > > > > > > > Y. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4) This approach would ensure fully > asynchronous > > > > > delivery > > > > > > > of > > > > > > > > > > events > > > > > > > > > > > > to > > > > > > > > > > > > > > > all interested listeners. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5) Fault-tolerance could also be easily > implemented > > > > > (event > > > > > > > > > > delivery > > > > > > > > > > > > > > > retries, timeouts, etc.). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What do you all think? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > https://quarkus.io/guides/reactive-event-bus > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
