I think the concept is really useful. The only thing I think which would require some more investigation is how exactly we implement this API -- where the data is stored, how long it's retained, etc. We might need to consider pushing this data out into another service or at least supporting such an implementation.
I'm also glad you called out the idea of a "fine-triggered" TMS based on events. A while ago, I had started drafting a design with a similar idea: [image: Screenshot 2025-04-10 at 11.46.19 AM.png] The concept was that some service can scrape events from Polaris (or get Polaris can push events to it) and that service will persist the events so that TMS, observability service, etc. can query those events. To this end, I think it might be worth finishing the ongoing Event Listeners <https://docs.google.com/document/d/1sJiFKeMlPVlqRUj8Rv4YufMMrfCq_3ZFtDuNv_8eOQ0/edit?tab=t.0#heading=h.8d519gwzsle2> work, so we have a way to collect the kind of information that the observe API will report. This gets us canonical event types as well. On Wed, Apr 9, 2025 at 10:09 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi folks, > > I would like to discuss a proposal that I have in mind: the "observe" API. > > The purpose of this API is to return some metrics and gauges from Polaris, > like: > - what's number of entities (number of tables, views; etc) in a Polaris > catalog > - what's the number of times a entity as been accessed on a period > - optionaly, access to "polished" metrics from table (extracted++ from > the metadata) > - optionaly, provide extra details (from Parquet metrics for instance) > > In terms of use cases, this API could be helpful: > - to have a policy "activated" depending of this metrics (something > like policy A is only valid if a catalog has more than X tables, or > policy B is activated when a view has been accessed more than Y times > in the last hour, etc). We can have TMS service "fined triggered" with > these policies. > - to be leverage by a FGAC mechanism (e.g. governance depending of > these metrics) > - to be easily displayed by a UI or CLI > > I already have a few ideas in mind that I would be happy to share in a > design document. But before that, I would like to get your feedback > about this proposal. > > Thanks ! > Regards > JB >
