Hi Russ, Yes, I know it has been discussed in Iceberg. It would be great to have this in Iceberg REST Spec, but my proposal is to start the shape something in Polaris and share the idea within the Iceberg community.
There are overlaps with Yufei's proposal about table maintenance. I started the review the proposal and I think we should add some more details. Probably we can find a "common backbone". Regards JB On Tue, Oct 29, 2024 at 5:12 PM Russell Spitzer <russell.spit...@gmail.com> wrote: > > I know we've had a lot of discussions about this and I think we were hoping > to get this into the Iceberg REST API as well. I think we should try to > push the API there but I have no problem having it only within the Polaris > spec if we can't get that in. I think it has a lot of overlap with the > things we have been talking about in Yufei's Table Maintenance proposal as > well. > > On Tue, Oct 29, 2024 at 10:58 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Hi folks, > > > > I would like to discuss a new proposal for a polling notification API. > > The proposal actually contains two "layers": > > - Entities change notification API (optionally with Java client to > > poll): a client can receive notifications when an event happens on an > > entity (containing the entity type, like table, view, namespace, > > catalog, ...). Events can be entities created, modified, deleted. The > > use case is basically to have both maintenance anticipation, > > incremental index update, actions triggering. > > - Entities change history: a client can get the list of all entities > > changes in the past N duration (hours, minutes, seconds). It could be > > an async or sync operation. The use case is about reconciliation and > > auditing. > > > > Thoughts ? > > > > If there's no objections, I will work on a proposal document (with > > much more details) that I will share on the dev mailing list (and also > > create a GitHub Issue with "proposal" tag). > > > > Regards > > JB > >
