Hi All, Re: "synchronous" and "sequential" even delivery: I believe these are different concerns.
In general, given that Polaris may be running in a distributed environment, sequential event delivery cannot be guaranteed... at least I do not think Polaris should attempt that. In the case of before/after events, sequential delivery of those might be meaningful to the consumer, but I think the concern in this case is actually not the sequence of events per se, but grouping of events that belong to the same logical change (transaction). As I commented in another email, from my POV, it would be preferable to collect events related to a request and deliver them only at the end of the request as a batch. That said, synchronous delivery in one JVM is still asynchronous with respect to other Polaris JMVs. Cheers, Dmitri. On Tue, Nov 11, 2025 at 10:57 AM Alexandre Dutra <[email protected]> wrote: > Hi all, > > > Are you envisioning anything for being able to make dependencies between > event listeners? > > No, they would be completely independent. > > > In some listeners we have the ability to make events emission > synchronous. > > Indeed, this is the case today when using the AWS CloudWatch sink in > the so-called "synchronous" mode where it waits (blocks) for the > server acknowledgement. > > By doing so, this listener blocks the event loop thread serving the > REST request. **This is bad practice**. Such behavior could seriously > hurt the server responsiveness, not to mention that Quarkus may > interrupt the event loop thread that is being blocked by a slow > listener. > > My event bus proposal does not allow for this use case. All events > would be processed asynchronously, in fire-and-forget mode. They can, > however, be processed by a listener performing blocking operations > (I/O etc.) – in which case they will be executed in a dedicated thread > pool. > > > If you wish, for example, to prevent the creation of more than 1k tables > with some given prefix, you can do so using a listener. > > I don't think that's a good usage of an event listener. If we want to > do this, I would rather see a proper validation framework. Events are > not meant for that. > > In my opinion, events should never influence the course of the REST > request in any way. They are meant to notify interested parties, not > for interested parties to alter Polaris' behavior. > > Thanks, > Alex > > On Mon, Nov 10, 2025 at 10:55 PM Eric Maynard <[email protected]> > wrote: > > > > In fact, shouldn’t it be exclusively a listener’s decision on whether an > > event is handled in a blocking way or not? As was noted in a past thread > on > > events, much of the utility of the event framework comes from the ability > > to introduce custom logic and hooks into the normal operation of Polaris. > > > > If you wish, for example, to prevent the creation of more than 1k tables > > with some given prefix, you can do so using a listener. If the event > which > > might trigger that logic becomes non-blocking, you would no longer be > able > > to block/fail the create table request. > > > > I think maybe it’s the name “event”, but we seem to keep conflating these > > hooks with the iceberg events or auditing events when they are not > exactly > > the same thing. > > > > —EM > > > > On Mon, Nov 10, 2025 at 8:47 PM Adnan Hemani > > <[email protected]> wrote: > > > > > Hi Alex, > > > > > > Thanks for writing down the proposal for this! As I had previously > > > suggested this when implementing the Persistence of Polaris Events > > > <https://github.com/apache/polaris/pull/1844>, I am obviously very > much in > > > favor of doing this :) > > > > > > A few questions I have regarding your vision of how we should implement > > > this: > > > * Are you envisioning anything for being able to make dependencies > between > > > event listeners? Or are we taking a set direction that Event Listeners > > > should be independent of each other? > > > * In some listeners we have the ability to make events emission > synchronous > > > [example > > > < > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java#L186 > > > >]. > > > How do we plan to support/advise (or not...) that with the introduction > > > of @Blocking annotations. > > > > > > Best, > > > Adnan Hemani > > > > > > On Mon, Nov 10, 2025 at 11:29 AM Yufei Gu <[email protected]> > wrote: > > > > > > > Thanks for the reply. It's overall a good idea to have async event > > > > listeners so that they are not blocking each other. > > > > > > > > One downside of the async ones is that event order isn't > deterministic. > > > > For example, event listeners of Spark need the order to understand > the > > > > execution semantics. I think Polaris is fine with that, given the ts > of > > > > each event is generated by Polaris. The downstream can still figure > out > > > the > > > > order. > > > > > > > > Thanks Pierre for sharing, I think any I/O-bound or potentially slow > > > > listener should be annotated with @Blocking. That ensures we keep the > > > event > > > > loop responsive and avoid impacting REST latency. > > > > > > > > Yufei > > > > > > > > > > > > On Mon, Nov 10, 2025 at 9:43 AM Alexandre Dutra <[email protected]> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Answering the questions above: > > > > > > > > > > > However, we can easily make sure that we use Quarkus's SmallRye > Fault > > > > > Tolerance > > > > > > > > > > Yes, that was my idea. It's not so much the bus itself that needs > to > > > > > be fault tolerant, but the receiving end, that is, the listeners. A > > > > > listener can fail for a variety of reasons (e.g. remote broker > > > > > unavailable), it would be nice to be able to backoff and retry > > > > > automatically. > > > > > > > > > > > Since the Vert.x event bus runs on event-loop threads [...] could > > > > > blocking or slow event listeners potentially stall REST requests > and > > > > impact > > > > > latency? > > > > > > > > > > What Pierre said: this could indeed happen, but it's possible to > > > > > annotate the receiving end with @Blocking, in which case, the > listener > > > > > will be invoked in a separate pool. > > > > > > > > > > > With asynchronous event listeners, is there a guarantee of > delivery > > > to > > > > > all listeners for a given event? > > > > > > > > > > If I understand the question correctly: with asynchronous > delivery, a > > > > > slow or failing listener wouldn't impact the delivery of the same > > > > > event to other listeners. > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > On Mon, Nov 10, 2025 at 10:12 AM Pierre Laporte < > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > > Thanks for the proposal, Alex. This sounds like a great > improvement. > > > > > > > > > > > > @Yufei As per Quarkus documentation, slow event listeners should > be > > > > > marked > > > > > > with @Blocking so that they are not run on the event loop > threads. > > > > > > -- > > > > > > > > > > > > Pierre > > > > > > > > > > > > > > > > > > On Sat, Nov 8, 2025 at 2:14 AM Michael Collado < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > With asynchronous event listeners, is there a guarantee of > delivery > > > > to > > > > > all > > > > > > > listeners for a given event? The downside of synchronous > listeners > > > is > > > > > that > > > > > > > everything is serial, but also if something fails, processing > > > stops. > > > > > This > > > > > > > feels important for auditing purposes, though less important > for > > > > other > > > > > > > cases. > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 2:28 PM Yufei Gu <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > Thanks, Alex and Adam. One concern I have is about the shared > > > > runtime > > > > > > > > thread pool. > > > > > > > > Since the Vert.x event bus runs on event-loop threads that > are > > > also > > > > > used > > > > > > > by > > > > > > > > Quarkus’ reactive REST endpoints, could blocking or slow > event > > > > > listeners > > > > > > > > potentially stall REST requests and impact latency? > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:25 AM Adam Christian < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > I think that this would be a great enhancement. Thanks for > > > > > proposing > > > > > > > it! > > > > > > > > > > > > > > > > > > The only concern I would have is around fault-tolerance. > From > > > > what > > > > > I > > > > > > > can > > > > > > > > > tell, from the Quarkus documentation, the Quarkus event bus > > > uses > > > > > Vert.x > > > > > > > > > EventBus which does not guarantee message delivery if > failure > > > of > > > > > part > > > > > > > of > > > > > > > > > the event bus occurs [1]. However, we can easily make sure > that > > > > we > > > > > use > > > > > > > > > Quarkus's SmallRye Fault Tolerance [2]. Is my rough > > > understanding > > > > > > > inline > > > > > > > > > with your proposal? > > > > > > > > > > > > > > > > > > Go community, > > > > > > > > > > > > > > > > > > Adam > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > https://vertx.io/docs/apidocs/io/vertx/core/eventbus/EventBus.html > > > > > > > > > [2]: https://quarkus.io/guides/smallrye-fault-tolerance > > > > > > > > > > > > > > > > > > On Fri, Nov 7, 2025 at 11:49 AM Alexandre Dutra < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > I'd like to propose an enhancement to our existing events > > > > > feature: > > > > > > > the > > > > > > > > > > ability to support multiple listeners. > > > > > > > > > > > > > > > > > > > > Currently, only a single listener can be active at a > time, > > > > which > > > > > is > > > > > > > > > > quite limiting. For example, we might need to persist > events > > > > for > > > > > > > audit > > > > > > > > > > purposes and simultaneously send them to a message queue > for > > > > > > > > > > optimization. With the current setup, this isn't easily > > > > > achievable. > > > > > > > > > > > > > > > > > > > > While a composite listener could be created, it feels > like a > > > > less > > > > > > > > > > elegant solution, and the delivery would be strictly > serial, > > > > > > > > > > processing one listener after another. > > > > > > > > > > > > > > > > > > > > My suggestion is to leverage Quarkus internal event bus > [1]: > > > > > > > > > > > > > > > > > > > > 1) There will be one central event emitter responsible > for > > > > > publishing > > > > > > > > > > events to the bus. > > > > > > > > > > > > > > > > > > > > 2) We will have zero to N listeners, each independently > > > > watching > > > > > the > > > > > > > > > > event bus for relevant events. They will be discovered by > > > CDI. > > > > > > > > > > > > > > > > > > > > 3) We could apply filters to each listener, e.g. > listener A > > > > > listens > > > > > > > > > > for event types X and Y, listener B only listens to event > > > type > > > > Y. > > > > > > > > > > > > > > > > > > > > 4) This approach would ensure fully asynchronous > delivery of > > > > > events > > > > > > > to > > > > > > > > > > all interested listeners. > > > > > > > > > > > > > > > > > > > > 5) Fault-tolerance could also be easily implemented > (event > > > > > delivery > > > > > > > > > > retries, timeouts, etc.). > > > > > > > > > > > > > > > > > > > > What do you all think? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > [1]: https://quarkus.io/guides/reactive-event-bus > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
