Hi all, Thanks for laying out the different work streams.
I think one part of the classification needs tightening: metadata integrity is not merely adjacent work. Iceberg documents two catalog security requirements [1] for table encryption: * `encryption.key-id` must not be modified or removed during the table's lifetime; * when `metadata.json` is kept in storage vulnerable to tampering, the catalog must obtain the metadata from a trusted store or verify its integrity using information kept in separate trusted storage. These requirements apply before Polaris can safely rely on `metadata.json` for encrypted-table operations, including server-side purge. The earlier RFC discussed both requirements, but I do not currently see where the active work in #5060 and #5127 will enforce them. Could we make these two invariants explicit requirements of the combined Polaris design and identify which changes will own them? It may be useful for the authors and reviewers of #5060 and #5127 to walk through this together at the next Polaris community call (in 14 days) and converge on one plan. Cheers, Robert [1] https://iceberg.apache.org/docs/nightly/encryption/#catalog-security-requirements On Tue, Jul 21, 2026 at 10:39 AM Hiroaki Kawai <[email protected]> wrote: > Hi all, > > During the review of the following Polaris pull request, I was asked to > bring > this topic to the dev mailing list for visibility: > > https://github.com/apache/polaris/pull/5060 > > The discussion spans several layers: engine-side file writes, REST metadata > commits, KMS access, and server-side operations such as purge. Based on the > current Iceberg behavior, REST Catalog semantics, and the related > implementation > work, the responsibilities and related work can be organized as follows. > > Iceberg native encryption > ------------------------- > > Iceberg native encryption protects data files, delete files, manifest > files, > and manifest-list files. > > The table metadata JSON itself remains plaintext. It contains the > encryption > metadata and wrapped keys required to open the encrypted files. > > Engine/client-side writes > ------------------------- > > The engine or client writes the encrypted data files, manifests, and > manifest > lists. It obtains the required KMS access and commits the resulting > metadata > updates, including wrapped encryption keys, through the catalog. > > For a REST Catalog, Polaris receives and persists those metadata updates > through the normal commit path. Polaris does not write manifests and does > not > need KMS access for this path. > > The remaining Iceberg client work for performing encrypted reads and writes > through RESTCatalog is tracked in: > > https://github.com/apache/iceberg/pull/13225 > > Server-side purge > ----------------- > > Server-side purge is a separate read and delete path. > > When a table is dropped with purgeRequested=true, Polaris is responsible > for > deleting the files referenced by the table. The data and delete file > locations > are stored in manifests, and the manifest locations are obtained from > manifest > lists. > > For an encrypted table, the cleanup worker therefore needs to decrypt the > manifest lists and manifests before it can enumerate the files to delete. > This > requires KMS access and the wrapped table keys on the Polaris side. > > This does not require Polaris to write, rewrite, or re-encrypt manifests. > The > server uses encryption only to read the files required for purge and then > delegates deletion to the underlying FileIO. > > The Polaris implementation work for this path is in: > > https://github.com/apache/polaris/pull/5060 > > That PR keeps encryption handling out of the normal LocalIcebergCatalog > write > path. The encryption context is used by asynchronous cleanup tasks to > create > an encryption-aware FileIO for reading encrypted manifest lists and > manifests. > > HiveCatalog comparison > ---------------------- > > HiveCatalog can make the two paths look less distinct because the catalog > is > loaded in the same process as the engine. The same catalog implementation > can > therefore provide encryption support for engine-side writes and for purge. > > With a REST Catalog, the engine and the Polaris server are separate > processes. > The engine performs encrypted writes, while Polaris participates only in > REST > metadata commits and server-side operations assigned to the catalog > service. > > Related work > ------------ > > The overall Polaris encryption discussion is tracked in: > > https://github.com/apache/polaris/issues/2829 > > REST KMS credential vending is being discussed separately in: > > https://github.com/apache/iceberg/pull/17155 > > That work concerns credentials returned by a REST catalog to a client so > that > the client can access KMS directly. It is separate from the credentials > used > internally by a Polaris cleanup worker for server-side purge. > > Server-side scan planning is another server-side read path related to > encrypted > table metadata. Polaris does not currently implement the general REST > server-side scan-planning endpoint, so this is outside the scope of the > current > purge work. > > Metadata integrity, encryption-key retirement, and other server-side > maintenance operations are also adjacent topics recorded in the tracking > issue, but are separate from the existing purgeRequested behavior. > > These topics can therefore be tracked as separate but related work streams: > client-side encryption, server-side purge, credential vending, and possible > future server-side operations. > > If anything in this summary is incomplete or does not match your > understanding, > please point it out. >
