Hi All, Thanks all for attending the review today and providing your valuable feedback, I really appreciate it ! And apologize for the late notice for the review meeting here.
I think we had a good discussion, and agreed to 1) Move on to provide a new Spark Catalog Plugin and new set of APIs with restricted scope to table managements, but keep the doors open for future extensions includes new fields or even new APIs. 2) Update the field create_at_ms to catalog_register_at to be more clear that the timestamp is for the registration time with Polaris. We will start working on figuring out more implementation details for the plugins as well as refactoring needed for Apache Polaris. If it turns out too complicated, we can have a separate review or discussion about the implementation part also. As a follow up of the meeting, we would like to open this thread for discussion for the URI prefix for the new endpoint. As Dmitri brought up on the design doc, since this set of APIs are new and not restricted by Iceberg anymore, shall we introduce a new prefix like /polaris/catalog/v1/ instead of reuse the current /catalog/v1? I think my concern about this proposal is that it seems to diverge how the APIs path are formalized today, the APIs in Polaris today use a path that is prefixed by functionality, like /catalog/ or /management/. If we start using /polaris/catalog/, shall we change the /management also to /polaris/management ? Shall we move the new Policy Engine related APIs to /polaris/catalog also? Another choice we have is to use /polaris-catalog/v1 instead /catalog/v1 as a prefix, which aligns with the current API path better. Then we have the same question, what do we do with the Policy Engine APIs? The Policies can eventually be applied to both table entities created through the /catalog APIs and /polaris/catalog APIs. Overall, I would prefer to stay consistent with the current APIs and continue to use /catalog/v1, since regardless of Iceberg or Generic tables, they are just entities managed by Polaris, and the APIs don't have 'iceberg' in the path also. I would like to hear what people think about this? Link to the doc: https://docs.google.com/document/d/1_R9jBIwoH3CV9G7gSoRJPQVEcOsROp4IEiGoeYeQE8A/edit?pli=1&tab=t.0 Best Regards, Yun
