Hi, Imho, we should start with a SPI (and it was my initial idea) for the change notification layer.
The change history layer is not strictly required imho because: 1. it means that history should be stored by Polaris (not sure it's a good idea) 2. the history can be built externally using the change notification layer Regards JB On Tue, Oct 29, 2024 at 5:16 PM Laurent Goujon <laur...@dremio.com.invalid> wrote: > > Now I realize it wasn't mentioned if it is a SPI (something to extend the > codebase with) or a REST API (webhook). A public REST API for notifications > is cause of concern for me as it introduces some new complexity to deal > with availability issues. > > So what type of APIs are we talking about? > > Laurent > > On Tue, Oct 29, 2024 at 11:13 AM Russell Spitzer <russell.spit...@gmail.com> > wrote: > > > I know we've had a lot of discussions about this and I think we were hoping > > to get this into the Iceberg REST API as well. I think we should try to > > push the API there but I have no problem having it only within the Polaris > > spec if we can't get that in. I think it has a lot of overlap with the > > things we have been talking about in Yufei's Table Maintenance proposal as > > well. > > > > On Tue, Oct 29, 2024 at 10:58 AM Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > > > Hi folks, > > > > > > I would like to discuss a new proposal for a polling notification API. > > > The proposal actually contains two "layers": > > > - Entities change notification API (optionally with Java client to > > > poll): a client can receive notifications when an event happens on an > > > entity (containing the entity type, like table, view, namespace, > > > catalog, ...). Events can be entities created, modified, deleted. The > > > use case is basically to have both maintenance anticipation, > > > incremental index update, actions triggering. > > > - Entities change history: a client can get the list of all entities > > > changes in the past N duration (hours, minutes, seconds). It could be > > > an async or sync operation. The use case is about reconciliation and > > > auditing. > > > > > > Thoughts ? > > > > > > If there's no objections, I will work on a proposal document (with > > > much more details) that I will share on the dev mailing list (and also > > > create a GitHub Issue with "proposal" tag). > > > > > > Regards > > > JB > > > > >
