I just looked and I think something similar to Nessie events are what I'm
looking for in this proposal. Whichever mechanism is used
(ServiceLoader/CDI), the intent is to allow specifying a class that
implements an event listener interface. Communication between catalogs is a
different feature which can co-exist but I think should be discussed
separately.
Regarding:
First off, I'm a bit concerned that this event listener interface would end
> up with dozens, maybe hundreds of rather unrelated methods: "onFooStarted",
> "onBarDropped", etc.
The intent is to allow extending behavior in many different places. I'm
wondering if this comment is more of a suggestion to carefully pick the
methods in the interface such that they're minimal but flexible, so that it
doesn't devolve into hundreds of tiny and potentially overlapping methods.
On Fri, Dec 13, 2024 at 8:13 AM Laurent Goujon <laur...@dremio.com.invalid>
wrote:
> >
> > Next, and since you mentioned Quarkus: when using Quarkus, the
> application
> > state is entirely constructed at build time, not at runtime. It's thus
> not
> > possible to just "drop a jar" somewhere with your listener
> implementation,
> > and expect it to be picked up by the running Polaris server. Instead,
> this
> > scenario requires a bit more work: one has to create an application that
> > extends the Polaris Quarkus runtime, add the listener implementation,
> build
> > the modified app, then package and deploy it. I'm not saying that this
> > isn't possible, only that it requires some work from users. In Nessie, we
> > do this e.g. to allow users to provide a subscriber for Nessie events.
> >
>
> It's a choice though? For example if the application would use the
> ServiceLoader interface instead of CDI, it would be possible to add things
> at runtime?
>
> Laurent
> On Fri, Dec 13, 2024 at 7:26 AM Alex Dutra <alex.du...@dremio.com.invalid>
> wrote:
>
> > Hi Andrew,
> >
> > I think the idea of extending / augmenting Polaris behavior is very
> > appealing, but deserves some more thinking.
> >
> > First off, I'm a bit concerned that this event listener interface would
> end
> > up with dozens, maybe hundreds of rather unrelated methods:
> "onFooStarted",
> > "onBarDropped", etc.
> >
> > Next, and since you mentioned Quarkus: when using Quarkus, the
> application
> > state is entirely constructed at build time, not at runtime. It's thus
> not
> > possible to just "drop a jar" somewhere with your listener
> implementation,
> > and expect it to be picked up by the running Polaris server. Instead,
> this
> > scenario requires a bit more work: one has to create an application that
> > extends the Polaris Quarkus runtime, add the listener implementation,
> build
> > the modified app, then package and deploy it. I'm not saying that this
> > isn't possible, only that it requires some work from users. In Nessie, we
> > do this e.g. to allow users to provide a subscriber for Nessie events.
> >
> > Speaking of Nessie events, I wonder if a better option would be to
> > implement something similar in Polaris? IOW Polaris would emit events
> > ("tableCreated", "catalogAdded", etc.) and users interested in these
> would
> > just provide a subscriber/listener that would send the events to a
> > messaging system or anything that suits their needs. This would require
> > more work, but OTOH, would achieve a better decoupling between Polaris
> > proper and how users can extend it.
> >
> > And that's where I think your proposal might step a bit into Catalog
> > Federation land. With federation, catalogs must have a way to send and
> > receive notifications. Polaris already has the "receiving end" (the
> > notifications endpoint), but nothing is implemented thus far for the
> sender
> > part. This observation has been brought
> > <https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht> to
> the
> > Iceberg mailing list already, where different approaches were discussed
> > (push vs pull based) – but no decision taken. Then some time later, JB
> > brought
> > a similar proposal
> > <https://lists.apache.org/thread/48fczg6okw9mos94o90tr347fbz9qc3b> to
> the
> > Polaris mailing list, but again, it seems the discussion stalled.
> >
> > I think my preference here would be to transfer this proposal to the
> > Catalog Federation epic, and make sure that we consider that events might
> > be consumed not only by reflecting catalogs, but by a broader audience
> > including general-purpose applications that wish to be notified of a
> > catalog's events (e.g. for the purpose of triggering table
> optimizations).
> >
> > Thanks,
> >
> > Alex
> >
> >
> > On Wed, Dec 11, 2024 at 8:00 PM Andrew Guterman <
> > andrew.guterm...@gmail.com>
> > wrote:
> >
> > > Hi folks,
> > >
> > > It would be useful to add a generic event listener interface to
> Polaris,
> > > consistent with other OSS projects. Users of the project may require
> > > additional functionality that doesn't have a clear enough value
> > proposition
> > > to be in OSS. Instead, there can be event listeners that let you hook
> > into
> > > various parts of the Polaris functionality (i.e. "before table commit")
> > > without OSS prescribing the limits of the extra functionality.
> > >
> > > Here are some examples of the shape of changes this would help:
> > >
> > > - https://github.com/apache/polaris/pull/458
> > > - https://github.com/apache/polaris/pull/155
> > > - https://github.com/apache/polaris/pull/423
> > > - Modification to application startup (i.e. by registering new
> Jersey
> > > resources)
> > > - Further observability changes
> > >
> > > In all of these cases we can (and sometimes have) introduced new ad-hoc
> > > interfaces to allow pluggable behavior, but I believe it's more
> scalable
> > to
> > > have a single event listener interface whose number of events grows
> over
> > > time. The intent is also not to reduce the amount of contributions to
> > OSS,
> > > but rather keep the changes focused and vendor-agnostic.
> > >
> > > Here are some example implementations in other OSS projects:
> > >
> > > -
> > >
> > >
> >
> https://spark.apache.org/docs/3.5.3/api/java/org/apache/spark/scheduler/SparkListener.html
> > > -
> > >
> https://heroiclabs.com/docs/nakama/guides/server-framework/using-hooks/
> > >
> > > I'm waiting for the Quarkus work to settle before suggesting a concrete
> > > implementation, as I heard Quarkus might make it easy to intercept
> method
> > > calls, but I did want to gather thoughts in the interim.
> > >
> > > Andrew
> > >
> >
>
