I mentioned Java and React because they're very clear examples of how the term "event listener" is generally understood, and Spark only because it was referenced by name in the initial proposal email. Spark's listeners *have to be* async due to the nature of Spark as something that runs across threads (/ VMs / processes). That said, it's still possible if not common to implement a Spark listener that does something like cancel long-running jobs. This is neither here nor there, though -- my point is not that we should copy Spark or React, but rather that to say events are not intended as injection points is perhaps revisionist history.
The original design proposal was sent out with this passage: It would be useful to add a generic event listener interface to Polaris, consistent with other OSS projects. Users of the project may require additional functionality that doesn't have a clear enough value proposition to be in OSS. Instead, there can be event listeners that let you hook into various parts of the Polaris functionality (i.e. "before table commit") without OSS prescribing the limits of the extra functionality. Events as they stand today can and do function as injection points for arbitrary code. It's my understanding that they were designed to serve this role and that this aspect of the design was in part a reaction to debates happening at the time around API stability, refactors, and extension points. We were still dreaming about killing CallContext. The Iceberg events proposal did not yet exist. And at least my own endorsement of the proposal / PR was contingent on this functionality. If we as a community align on removing this functionality from the events framework, we should be intentional about that. Currently the (re)design of events seems to be taking place across many email threads without clear arguments about what the framework should support or why. --EM On Wed, Nov 12, 2025 at 1:08 PM Alexandre Dutra <[email protected]> wrote: > Hi all, > > Several people mentioned Spark event listeners. After looking more > closely into this feature, I think it actually looks very similar to > what I'm proposing: > > - The primary intended use cases that I could find are : monitoring > job progress, tracking stages and task completion, gathering metrics > [1]. > > - Events are distributed asynchronously via an internal component > called SparkListenerBus [2], which manages an event bus and a > single-threaded event queue. > > - The API consists solely of methods that return void without checked > exceptions: IOW, the API wasn't designed to allow for listeners to > interact with the server (other than throwing an unchecked exception, > of course). > > - Afaict if a listener throws, the bus catches the exception and moves on. > > I'm not an expert in this topic so I might be wrong here, but the > suggestion that Spark event listeners were designed to allow listeners > to modify Spark's behavior doesn't look accurate to me. > > Thanks, > Alex > > [1]: > https://www.ibm.com/think/insights/apache-spark-monitoring-using-listeners-and-data-quality-libraries > [2]: > https://spark.apache.org/docs/latest/api/java/org/apache/spark/scheduler/SparkListenerBus.html > > On Tue, Nov 11, 2025 at 1:30 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > > Hi Eric, > > > > I would very much prefer not to use the event listener SPI as a means to > > control the operation of the Polaris Server. > > > > More specifically, I believe that any error / exception in an event > > listener should not affect the processing of the request. > > > > If we need custom callbacks to control some aspects of the server > > behaviour, let's define a dedicated SPI for that, but, IMHO, it should be > > outside the scope of events. WDYT? > > > > Thanks, > > Dmitri. > > > > On Mon, Nov 10, 2025 at 8:55 PM Eric Maynard <[email protected]> > > wrote: > > > > > In fact, shouldn’t it be exclusively a listener’s decision on whether > an > > > event is handled in a blocking way or not? As was noted in a past > thread on > > > events, much of the utility of the event framework comes from the > ability > > > to introduce custom logic and hooks into the normal operation of > Polaris. > > > > > > If you wish, for example, to prevent the creation of more than 1k > tables > > > with some given prefix, you can do so using a listener. If the event > which > > > might trigger that logic becomes non-blocking, you would no longer be > able > > > to block/fail the create table request. > > > > > > I think maybe it’s the name “event”, but we seem to keep conflating > these > > > hooks with the iceberg events or auditing events when they are not > exactly > > > the same thing. > > > > > > —EM > > > > > > On Mon, Nov 10, 2025 at 8:47 PM Adnan Hemani > > > <[email protected]> wrote: > > > > > > > Hi Alex, > > > > > > > > Thanks for writing down the proposal for this! As I had previously > > > > suggested this when implementing the Persistence of Polaris Events > > > > <https://github.com/apache/polaris/pull/1844>, I am obviously very > much > > > in > > > > favor of doing this :) > > > > > > > > A few questions I have regarding your vision of how we should > implement > > > > this: > > > > * Are you envisioning anything for being able to make dependencies > > > between > > > > event listeners? Or are we taking a set direction that Event > Listeners > > > > should be independent of each other? > > > > * In some listeners we have the ability to make events emission > > > synchronous > > > > [example > > > > < > > > > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java#L186 > > > > >]. > > > > How do we plan to support/advise (or not...) that with the > introduction > > > > of @Blocking annotations. > > > > > > > > Best, > > > > Adnan Hemani > > > > > > > > On Mon, Nov 10, 2025 at 11:29 AM Yufei Gu <[email protected]> > wrote: > > > > > > > > > Thanks for the reply. It's overall a good idea to have async event > > > > > listeners so that they are not blocking each other. > > > > > > > > > > One downside of the async ones is that event order isn't > deterministic. > > > > > For example, event listeners of Spark need the order to understand > the > > > > > execution semantics. I think Polaris is fine with that, given the > ts of > > > > > each event is generated by Polaris. The downstream can still > figure out > > > > the > > > > > order. > > > > > > > > > > Thanks Pierre for sharing, I think any I/O-bound or potentially > slow > > > > > listener should be annotated with @Blocking. That ensures we keep > the > > > > event > > > > > loop responsive and avoid impacting REST latency. > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 9:43 AM Alexandre Dutra <[email protected] > > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Answering the questions above: > > > > > > > > > > > > > However, we can easily make sure that we use Quarkus's SmallRye > > > Fault > > > > > > Tolerance > > > > > > > > > > > > Yes, that was my idea. It's not so much the bus itself that > needs to > > > > > > be fault tolerant, but the receiving end, that is, the > listeners. A > > > > > > listener can fail for a variety of reasons (e.g. remote broker > > > > > > unavailable), it would be nice to be able to backoff and retry > > > > > > automatically. > > > > > > > > > > > > > Since the Vert.x event bus runs on event-loop threads [...] > could > > > > > > blocking or slow event listeners potentially stall REST requests > and > > > > > impact > > > > > > latency? > > > > > > > > > > > > What Pierre said: this could indeed happen, but it's possible to > > > > > > annotate the receiving end with @Blocking, in which case, the > > > listener > > > > > > will be invoked in a separate pool. > > > > > > > > > > > > > With asynchronous event listeners, is there a guarantee of > delivery > > > > to > > > > > > all listeners for a given event? > > > > > > > > > > > > If I understand the question correctly: with asynchronous > delivery, a > > > > > > slow or failing listener wouldn't impact the delivery of the same > > > > > > event to other listeners. > > > > > > > > > > > > Thanks, > > > > > > Alex > > > > > > > > > > > > On Mon, Nov 10, 2025 at 10:12 AM Pierre Laporte < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Thanks for the proposal, Alex. This sounds like a great > > > improvement. > > > > > > > > > > > > > > @Yufei As per Quarkus documentation, slow event listeners > should be > > > > > > marked > > > > > > > with @Blocking so that they are not run on the event loop > threads. > > > > > > > -- > > > > > > > > > > > > > > Pierre > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 8, 2025 at 2:14 AM Michael Collado < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > With asynchronous event listeners, is there a guarantee of > > > delivery > > > > > to > > > > > > all > > > > > > > > listeners for a given event? The downside of synchronous > > > listeners > > > > is > > > > > > that > > > > > > > > everything is serial, but also if something fails, processing > > > > stops. > > > > > > This > > > > > > > > feels important for auditing purposes, though less important > for > > > > > other > > > > > > > > cases. > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 2:28 PM Yufei Gu < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks, Alex and Adam. One concern I have is about the > shared > > > > > runtime > > > > > > > > > thread pool. > > > > > > > > > Since the Vert.x event bus runs on event-loop threads that > are > > > > also > > > > > > used > > > > > > > > by > > > > > > > > > Quarkus’ reactive REST endpoints, could blocking or slow > event > > > > > > listeners > > > > > > > > > potentially stall REST requests and impact latency? > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:25 AM Adam Christian < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > I think that this would be a great enhancement. Thanks > for > > > > > > proposing > > > > > > > > it! > > > > > > > > > > > > > > > > > > > > The only concern I would have is around fault-tolerance. > From > > > > > what > > > > > > I > > > > > > > > can > > > > > > > > > > tell, from the Quarkus documentation, the Quarkus event > bus > > > > uses > > > > > > Vert.x > > > > > > > > > > EventBus which does not guarantee message delivery if > failure > > > > of > > > > > > part > > > > > > > > of > > > > > > > > > > the event bus occurs [1]. However, we can easily make > sure > > > that > > > > > we > > > > > > use > > > > > > > > > > Quarkus's SmallRye Fault Tolerance [2]. Is my rough > > > > understanding > > > > > > > > inline > > > > > > > > > > with your proposal? > > > > > > > > > > > > > > > > > > > > Go community, > > > > > > > > > > > > > > > > > > > > Adam > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > https://vertx.io/docs/apidocs/io/vertx/core/eventbus/EventBus.html > > > > > > > > > > [2]: https://quarkus.io/guides/smallrye-fault-tolerance > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:49 AM Alexandre Dutra < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > I'd like to propose an enhancement to our existing > events > > > > > > feature: > > > > > > > > the > > > > > > > > > > > ability to support multiple listeners. > > > > > > > > > > > > > > > > > > > > > > Currently, only a single listener can be active at a > time, > > > > > which > > > > > > is > > > > > > > > > > > quite limiting. For example, we might need to persist > > > events > > > > > for > > > > > > > > audit > > > > > > > > > > > purposes and simultaneously send them to a message > queue > > > for > > > > > > > > > > > optimization. With the current setup, this isn't easily > > > > > > achievable. > > > > > > > > > > > > > > > > > > > > > > While a composite listener could be created, it feels > like > > > a > > > > > less > > > > > > > > > > > elegant solution, and the delivery would be strictly > > > serial, > > > > > > > > > > > processing one listener after another. > > > > > > > > > > > > > > > > > > > > > > My suggestion is to leverage Quarkus internal event bus > > > [1]: > > > > > > > > > > > > > > > > > > > > > > 1) There will be one central event emitter responsible > for > > > > > > publishing > > > > > > > > > > > events to the bus. > > > > > > > > > > > > > > > > > > > > > > 2) We will have zero to N listeners, each independently > > > > > watching > > > > > > the > > > > > > > > > > > event bus for relevant events. They will be discovered > by > > > > CDI. > > > > > > > > > > > > > > > > > > > > > > 3) We could apply filters to each listener, e.g. > listener A > > > > > > listens > > > > > > > > > > > for event types X and Y, listener B only listens to > event > > > > type > > > > > Y. > > > > > > > > > > > > > > > > > > > > > > 4) This approach would ensure fully asynchronous > delivery > > > of > > > > > > events > > > > > > > > to > > > > > > > > > > > all interested listeners. > > > > > > > > > > > > > > > > > > > > > > 5) Fault-tolerance could also be easily implemented > (event > > > > > > delivery > > > > > > > > > > > retries, timeouts, etc.). > > > > > > > > > > > > > > > > > > > > > > What do you all think? > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > [1]: https://quarkus.io/guides/reactive-event-bus > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
