saidixith002 opened a new pull request, #256: URL: https://github.com/apache/polaris-tools/pull/256
Add a --skip-iceberg-content flag to sync-polaris that skips walking and synchronizing Iceberg namespaces and tables for each catalog. Principals, principal-roles, catalogs, catalog-roles, and grants are still synchronized as before. Motivation: disaster recovery ------------------------------ For teams running Polaris in an active/standby (or multi-region) topology, polaris-synchronizer is a convenient way to keep the *access control and catalog structure* of a standby instance in sync with the primary so that failover is fast - principals, principal-roles, catalogs, catalog-roles, and grants all need to exist ahead of time on the standby. However, the default sync path also opens an Iceberg REST catalog session per catalog and recursively lists every namespace and table underneath it, which is the most expensive part of the sync and scales with the number of tables rather than the number of Polaris entities. For a DR-readiness sync that may run frequently (e.g. on a schedule) purely to keep entity/grant structure current, this Iceberg walk is unnecessary overhead - the actual Iceberg table metadata/content is typically already replicated by a separate, storage-level mechanism (e.g. object storage replication) rather than by this tool. --skip-iceberg-content lets operators run frequent, lightweight structural syncs for DR readiness without paying the cost of a full namespace/table walk, while still running full syncs (without the flag) when an actual data migration is needed. Known limitation ----------------- Grant synchronization is not filtered by grant type, so a TABLE- or NAMESPACE-scoped grant on a catalog role is still synced to the target even when --skip-iceberg-content prevented that table/namespace from ever being created there. Against a real Polaris server this would likely fail server-side (grant referencing an unknown resource); with the default --halt-on-failure=false this fails silently per-grant rather than aborting the run. This is called out explicitly and covered by a test documenting the current behavior - narrowing grant sync to skip content-scoped grants when this flag is set is left as follow-up work. Testing ------- Added PolarisSynchronizerSkipIcebergContentTest covering: - Iceberg REST catalog sessions are never initialized when the flag is set, while catalog-role sync still runs. - Original (Iceberg content synced) behavior is unchanged when the flag is not set. - Documents that TABLE-scoped grants are still attempted even when the underlying table was never synced (see Known limitation above). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
