Thanks for suggesting that we discuss the combined plan at the community call. Unfortunately, I won’t be able to attend the upcoming call. I have sent a separate follow-up titled “Catalog security prerequisites for Iceberg table encryption in Polaris,” outlining how I think the two catalog security requirements should be addressed and how the related work fits together. I prepared the following draft patches: - #5185 pins the expected encryption.key-id, including its absence: - https://github.com/apache/polaris/pull/5185 - #5186 pins and verifies the current encrypted metadata revision and is stacked on #5185: - https://github.com/apache/polaris/pull/5186
2026年7月23日(木) 21:44 Robert Stupp <[email protected]>: > > Hi all, > > Thanks for laying out the different work streams. > > I think one part of the classification needs tightening: metadata integrity > is not merely adjacent work. > > Iceberg documents two catalog security requirements [1] for table > encryption: > > * `encryption.key-id` must not be modified or removed during the table's > lifetime; > * when `metadata.json` is kept in storage vulnerable to tampering, the > catalog > must obtain the metadata from a trusted store or verify its integrity > using > information kept in separate trusted storage. > > These requirements apply before Polaris can safely rely on `metadata.json` > for encrypted-table operations, including server-side purge. > > The earlier RFC discussed both requirements, but I do not currently see > where the active work in #5060 and #5127 will enforce them. > Could we make these two invariants explicit requirements of the combined > Polaris design and identify which changes will own them? > > It may be useful for the authors and reviewers of #5060 and #5127 to walk > through this together at the next Polaris community call (in 14 days) and > converge on one plan. > > Cheers, > Robert > > [1] > https://iceberg.apache.org/docs/nightly/encryption/#catalog-security-requirements > > On Tue, Jul 21, 2026 at 10:39 AM Hiroaki Kawai <[email protected]> > wrote: > > > Hi all, > > > > During the review of the following Polaris pull request, I was asked to > > bring > > this topic to the dev mailing list for visibility: > > > > https://github.com/apache/polaris/pull/5060 > > > > The discussion spans several layers: engine-side file writes, REST metadata > > commits, KMS access, and server-side operations such as purge. Based on the > > current Iceberg behavior, REST Catalog semantics, and the related > > implementation > > work, the responsibilities and related work can be organized as follows. > > > > Iceberg native encryption > > ------------------------- > > > > Iceberg native encryption protects data files, delete files, manifest > > files, > > and manifest-list files. > > > > The table metadata JSON itself remains plaintext. It contains the > > encryption > > metadata and wrapped keys required to open the encrypted files. > > > > Engine/client-side writes > > ------------------------- > > > > The engine or client writes the encrypted data files, manifests, and > > manifest > > lists. It obtains the required KMS access and commits the resulting > > metadata > > updates, including wrapped encryption keys, through the catalog. > > > > For a REST Catalog, Polaris receives and persists those metadata updates > > through the normal commit path. Polaris does not write manifests and does > > not > > need KMS access for this path. > > > > The remaining Iceberg client work for performing encrypted reads and writes > > through RESTCatalog is tracked in: > > > > https://github.com/apache/iceberg/pull/13225 > > > > Server-side purge > > ----------------- > > > > Server-side purge is a separate read and delete path. > > > > When a table is dropped with purgeRequested=true, Polaris is responsible > > for > > deleting the files referenced by the table. The data and delete file > > locations > > are stored in manifests, and the manifest locations are obtained from > > manifest > > lists. > > > > For an encrypted table, the cleanup worker therefore needs to decrypt the > > manifest lists and manifests before it can enumerate the files to delete. > > This > > requires KMS access and the wrapped table keys on the Polaris side. > > > > This does not require Polaris to write, rewrite, or re-encrypt manifests. > > The > > server uses encryption only to read the files required for purge and then > > delegates deletion to the underlying FileIO. > > > > The Polaris implementation work for this path is in: > > > > https://github.com/apache/polaris/pull/5060 > > > > That PR keeps encryption handling out of the normal LocalIcebergCatalog > > write > > path. The encryption context is used by asynchronous cleanup tasks to > > create > > an encryption-aware FileIO for reading encrypted manifest lists and > > manifests. > > > > HiveCatalog comparison > > ---------------------- > > > > HiveCatalog can make the two paths look less distinct because the catalog > > is > > loaded in the same process as the engine. The same catalog implementation > > can > > therefore provide encryption support for engine-side writes and for purge. > > > > With a REST Catalog, the engine and the Polaris server are separate > > processes. > > The engine performs encrypted writes, while Polaris participates only in > > REST > > metadata commits and server-side operations assigned to the catalog > > service. > > > > Related work > > ------------ > > > > The overall Polaris encryption discussion is tracked in: > > > > https://github.com/apache/polaris/issues/2829 > > > > REST KMS credential vending is being discussed separately in: > > > > https://github.com/apache/iceberg/pull/17155 > > > > That work concerns credentials returned by a REST catalog to a client so > > that > > the client can access KMS directly. It is separate from the credentials > > used > > internally by a Polaris cleanup worker for server-side purge. > > > > Server-side scan planning is another server-side read path related to > > encrypted > > table metadata. Polaris does not currently implement the general REST > > server-side scan-planning endpoint, so this is outside the scope of the > > current > > purge work. > > > > Metadata integrity, encryption-key retirement, and other server-side > > maintenance operations are also adjacent topics recorded in the tracking > > issue, but are separate from the existing purgeRequested behavior. > > > > These topics can therefore be tracked as separate but related work streams: > > client-side encryption, server-side purge, credential vending, and possible > > future server-side operations. > > > > If anything in this summary is incomplete or does not match your > > understanding, > > please point it out. > >
