Hi everyone, PR #5222 [1] prompted me to look more broadly at the status of the Polaris notification API.
This is separate from the technical scope of that PR. Fixing the concurrency behavior of an existing endpoint seems reasonable, and I do not think the PR needs to wait for this discussion. Despite its name, the endpoint is effectively an inbound catalog-synchronization API. A remote catalog or sync agent can use it to create, update, validate, or drop externally managed table state in Polaris. The endpoint arrived with the initial Polaris code import and has since acquired ordering and validation semantics. The January 2025 discussion [2] classified it as a Polaris-specific API, resulting in PR #906 [3] moving it into a separate specification file. The same discussion said that it had originally been intended for contribution to Iceberg and was not expected to remain in the Polaris specification long-term. Since then, Iceberg has added an `overwrite` option to table registration [4]. The upstream discussion explicitly considered complete metadata replacement and push-based mirroring [5]. Polaris now implements that operation [6], although currently only for internal catalogs. This appears to overlap substantially with the notification endpoint's catalog-synchronization purpose. Snowflake is a documented consumer of the notification endpoint [7]. I could not find public evidence of another deployed producer, although code search is of course not usage telemetry. The endpoint is not marked experimental or beta. Under the current Polaris evolution policy [8], it therefore appears to be a stable, versioned REST API with ongoing compatibility expectations. Before we continue evolving it incrementally, should this remain a stable Polaris-specific API, or should it be deprecated in favor of standard Iceberg REST operations? If existing users rely on behavior that the standard endpoints cannot provide, it would be useful to document those requirements and the case for retaining the API. Does that match others' understanding? Robert [1] https://github.com/apache/polaris/pull/5222 [2] https://lists.apache.org/thread/1fqocs00pno0xfr4ss2p69d6dv5h8qzf [3] https://github.com/apache/polaris/pull/906 [4] https://github.com/apache/iceberg/pull/12239 [5] https://lists.apache.org/thread/kkj1f98v035p2k25jpcb5mboc9bpsk58 [6] https://github.com/apache/polaris/pull/4506 [7] https://docs.snowflake.com/en/sql-reference/functions/system_send_notifications_to_catalog [8] https://github.com/apache/polaris/blob/main/site/content/in-dev/unreleased/evolution.md
