Hi All, I think Fabio made great progress on the KMS work [2802] and the PR is approaching a mergeable state. Please have a look.
>From my POV, the only concern is exposing new Storage Config properties to older clients (even when they were not set explicitly). WDYT? If that is not an issue for the community, I think we can merge. [2802] https://github.com/apache/polaris/pull/2802 Thanks, Dmitri. On Mon, Nov 17, 2025 at 1:23 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > Heads up: Fabio made some progress on [2802] (thanks!) and as for me, the > PR is almost ready to merge. Please have another look and comment if you > have any concerns. > > We can certainly follow-up with additional feature PRs to enhance KMS > support later (as discussed). > > [2802] https://github.com/apache/polaris/pull/2802 > > Cheers, > Dmitri. > > On Fri, Oct 31, 2025 at 2:00 PM Michael Collado <[email protected]> > wrote: > >> I'm fine with postponing the full implementation for table-level >> overrides, >> but I think it's worth pointing out that the only real change needed is on >> one in AccessConfigProvider. My primary ask was to ensure that the API be >> only changed once so that the implementation under the hood can support >> the >> new feature without another API change and I think the current PR supports >> that. I'd be happy to build on this to add table-level support with tests. >> >> Mike >> >> On Thu, Oct 30, 2025 at 10:36 AM Dmitri Bourlatchkov <[email protected]> >> wrote: >> >> > To recap the KMS discussion in the community sync meeting today: >> > >> > Using per-catalog and per-table KMS config was discussed. IIRC, the >> general >> > agreement was that both options are worth supporting. >> > >> > However, from the practical perspective, the agreement was to implement >> > only per-catalog configuration in the current PR and work on per-table >> > support later. >> > >> > Options for per-table support (need more discussion): >> > >> > a) Use table properties received via REST API and copy them into Polaris >> > properties in table entities. Use entity properties for producing >> > StorageAccessConfig and policies associated with vended credentials. >> > >> > Option (a) appears to be easier to implement and more flexible (allows >> > users to define per-table config using existing SQL-based clients). >> > >> > b) Use entity-specific Storage Config >> > >> > Option (b) is also worth considering, but it needs more changes / >> > refactoring in Polaris code and it is not clear ATM how clients can >> provide >> > per-table config in this case. >> > >> > Cheers, >> > Dmitri. >> > >> > >> > On Thu, Oct 30, 2025 at 10:07 AM Dmitri Bourlatchkov <[email protected]> >> > wrote: >> > >> > > Heads up: I put an item on today's Community Sync agenda for KMS. >> > > >> > > On Fri, Oct 24, 2025 at 3:20 PM Dmitri Bourlatchkov <[email protected] >> > >> > > wrote: >> > > >> > >> Hi All, >> > >> >> > >> At the high level, both engines and Polaris need to declare one >> specific >> > >> KMS key ARN for writes (both data and metadata files) and have >> policies >> > >> that allow accessing any previous KMS keys for reads. >> > >> >> > >> Do we really need to have that configurable per-table? From my POV it >> > >> feels like a catalog-level concern. >> > >> >> > >> In any case, even if we support using different keys for new writes >> in >> > >> different tables, the list of previous KMS ARNs for reads is probably >> > still >> > >> a catalog-level concern. Relying on _current_ table metadata >> properties >> > to >> > >> always correctly reflect all possible previous KMS ARNs seems >> excessive >> > and >> > >> risky to me. It would also increase the binding of the table to a >> > >> particular storage location / technology. On the other hand, >> > catalog-level >> > >> config can be adjusted by the Polaris admin user at any time to >> correct >> > the >> > >> list of KMS ARNs so that clients get the right access policies >> applied >> > to >> > >> their session credentials. >> > >> >> > >> In summary, I think Fabio's example for "storageConfigInfo" goes in >> the >> > >> right direction conceptually. I'd just propose a couple minor >> changes: >> > >> * defaultKmsKey -> currentKmsKey ("current" meaning used for new >> writes) >> > >> * allowedKmsKeys -> make it optional, if not set, allow any KMS ARN >> for >> > >> reads. >> > >> >> > >> If we want to have fine-grained per-table configuration, I like >> > Michael's >> > >> proposal for per-entity (per-table) storage config overrides, but I >> also >> > >> agree that it's a bigger change (requires some refactoring in the >> core >> > >> code), so it might be best to address it in follow-up PRs. >> > >> >> > >> Inspecting table properties could be a convenience feature to allow >> for >> > >> simpler user-level workflows (e.g. setting them in SQL), but I tend >> to >> > >> think that ultimately Polaris should keep the "authoritative" data >> about >> > >> KMS in its own structures (e.g. Storage Config) and use table >> properties >> > >> only as a source of changes to that data. In that regard, I believe >> it >> > >> might be preferable to use Polaris-specific properties names (NOT >> > >> "s3.sse.key"). >> > >> >> > >> As for the "s3.sse.key" property, I do not think that Iceberg itself >> > >> loads it from table metadata properties. AFAIK, Iceberg java code >> gets >> > it >> > >> from loadTable REST API responses. >> > >> >> > >> WDYT? >> > >> >> > >> Thanks, >> > >> Dmitri. >> > >> >> > >> >> > >> On Fri, Oct 24, 2025 at 12:28 PM Rizzo Cascio, Fabio >> > >> <[email protected]> wrote: >> > >> >> > >>> Hi Michael, >> > >>> >> > >>> For the APIs are you thinking something like this? >> > >>> >> > >>> >> > >>> '{ >> > >>> "catalog": { >> > >>> "name": "quickstart_catalog", >> > >>> "type": "INTERNAL", >> > >>> "readOnly": false, >> > >>> "properties": { >> > >>> "default-base-location": "'$STORAGE_LOCATION'" >> > >>> }, >> > >>> "storageConfigInfo": { >> > >>> "storageType": "'$STORAGE_TYPE'", >> > >>> "allowedLocations": ["'$STORAGE_LOCATION'"], >> > >>> "allowedKmsKeys": ["arn:aws:kms:us-east-1:***:key/mykey"], >> > >>> "defaultKmsKey": "arn:aws:kms:us-east-1:***:key/mykey" >> > >>> } >> > >>> } >> > >>> }' >> > >>> >> > >>> >> > >>> Can you clarify the part where you are referring to the PR #2735 , >> > cause >> > >>> I’m not super familiar with that part of the code ? >> > >>> >> > >>> >> > >>> Thanks >> > >>> >> > >>> Fabio >> > >>> >> > >>> From: Michael Collado <[email protected]> >> > >>> Date: Thursday, 23 October 2025 at 18:04 >> > >>> To: [email protected] <[email protected]> >> > >>> Subject: Re: [EXTERNAL]Re: KMS Key addition for s3 >> > >>> >> > >>> Maybe I caused more confusion than I meant to 😅 >> > >>> >> > >>> In my mind, there are two ways to support table-level overrides for >> KMS >> > >>> keys: >> > >>> 1. We can support StorageConfig overrides at the table level - I >> think >> > we >> > >>> previously talked about this being the right long term approach, as >> it >> > >>> would be great to allow overrides for more than just KMS keys (for >> > >>> example, >> > >>> using a base location that's in the catalog's allowed locations, but >> > not >> > >>> the default base location). This is a bigger change, however. We >> also >> > >>> haven't defined an easy way for users to create tables with these >> > storage >> > >>> overrides. >> > >>> 2. Support existing S3FileIO table properties. This feels like a >> > >>> necessary >> > >>> change anyway, as I think this would be necessary for users to >> migrate >> > >>> existing tables into Polaris. It's also a smaller change, so I was >> > kinda >> > >>> hoping we could sneak it in here :) >> > >>> >> > >>> I think Dmitri's question of whether the metadata.json file is >> stored >> > >>> encrypted is an important one, as we'd need to know which KMS key to >> > add >> > >>> to >> > >>> the IAM policy before we can read the metadata.json file to read the >> > >>> table's properties 🤪 . Maybe we can continue with work like #2735 >> and >> > >>> add >> > >>> the TableMetadata properties to the Table Entity's properties map? >> That >> > >>> would allow us to read the encryption properties before we read the >> > file >> > >>> from storage. This supports existing CREATE TABLE syntax, as users >> can >> > >>> use >> > >>> something like this to specify a table-level override: >> > >>> >> > >>> CREATE TABLE ... TBLPROPERTIES("s3.sse.type": "kms", "s3.sse.key": >> > >>> "arn:aws:kms:us-east-1:***:key/mykey) >> > >>> >> > >>> However, that doesn't fix the registerTable issue raised. The only >> way >> > I >> > >>> can see to support registering tables that use a different KMS key >> from >> > >>> the >> > >>> catalog default is to follow the pattern we established with >> > >>> allowedLocations and defaultBaseLocation. The catalog can support a >> > list >> > >>> of >> > >>> KMS keys that can be used for decryption with one specified as the >> > >>> default. >> > >>> Again, this feels like an issue that needs to be tackled anyway as >> > there >> > >>> are definitely catalogs that use a mix of encryption keys. >> > >>> >> > >>> The encryption keys do need to be returned in the LoadTableResponse, >> > >>> which >> > >>> I think the current PR does. That should eliminate the need to >> specify >> > >>> the >> > >>> --conf spark.sql.catalog.polaris.s3.sse.* key/value pairs in the >> client >> > >>> configuration. Only cases like CREATE TABLE AS SELECT... would need >> to >> > >>> know >> > >>> about the KMS keys beforehand. Simple CREATE TABLE statements can >> use >> > the >> > >>> TBLPROPERTIES config, as in the above example. >> > >>> >> > >>> This is maybe a lot for one PR, so I'm open to hear about how we can >> > >>> split >> > >>> this up and move forward incrementally. But I think it would be good >> > for >> > >>> the APIs to be thought of up front. >> > >>> >> > >>> Mike >> > >>> >> > >>> On Thu, Oct 23, 2025 at 7:42 AM Rizzo Cascio, Fabio >> > >>> <[email protected]> wrote: >> > >>> >> > >>> > Hi Dmitri, >> > >>> > >> > >>> > You mean doing a read table data I assume, like this: >> > >>> > >> > >>> > >> > >>> >> > >> http://localhost:8181/api/catalog/v1/test_catalog/namespaces/fns/tables/t3 >> > >>> > >> > >>> > The answer is yes; the key it is used to do server-side >> encryption >> > so >> > >>> if >> > >>> > the role running the pod has the correct entitlement to use the >> key >> > to >> > >>> > decrypt the data there would be no issue. >> > >>> > >> > >>> > Btw, this is also the case using mini on docker. >> > >>> > >> > >>> > Response is: >> > >>> > { >> > >>> > "metadata-location": >> > >>> "s3://mybucket/fns/t3/metadata/a.metadata.json", >> > >>> > "metadata": { >> > >>> > "format-version": 2, >> > >>> > "table-uuid": "4d452e08-cf39-434b-b003-a21ccd6c4a42", >> > >>> > "location": "s3://mybucket/fns/t3", >> > >>> > "last-sequence-number": 1, >> > >>> > "last-updated-ms": 1761059868331, >> > >>> > "last-column-id": 2, >> > >>> > "current-schema-id": 0, >> > >>> > "schemas": [ >> > >>> > { >> > >>> > "type": "struct", >> > >>> > "schema-id": 0, >> > >>> > "fields": [ >> > >>> > { >> > >>> > "id": 1, >> > >>> > "name": "field1", >> > >>> > "required": false, >> > >>> > "type": "int" >> > >>> > }, >> > >>> > { >> > >>> > "id": 2, >> > >>> > "name": "field2", >> > >>> > "required": false, >> > >>> > "type": "int" >> > >>> > } >> > >>> > ] >> > >>> > } >> > >>> > ], >> > >>> > "default-spec-id": 0, >> > >>> > "partition-specs": [ >> > >>> > { >> > >>> > "spec-id": 0, >> > >>> > "fields": [] >> > >>> > } >> > >>> > ], >> > >>> > "last-partition-id": 999, >> > >>> > "default-sort-order-id": 0, >> > >>> > "sort-orders": [ >> > >>> > { >> > >>> > "order-id": 0, >> > >>> > "fields": [] >> > >>> > } >> > >>> > ], >> > >>> > "properties": { >> > >>> > "created-at": "2025-10-21T15:14:52.093498193Z", >> > >>> > "s3.sse.type": "kms", >> > >>> > "s3.sse.key": "arn:aws:kms:us-east-1:***:key/mykey, >> > >>> > "write.parquet.compression-codec": "zstd" >> > >>> > }, >> > >>> > "current-snapshot-id": 2031898559602022053, >> > >>> > "refs": { >> > >>> > "main": { >> > >>> > "snapshot-id": 2031898559602022053, >> > >>> > "type": "branch" >> > >>> > } >> > >>> > }, >> > >>> > "snapshots": [ >> > >>> > { >> > >>> > "sequence-number": 1, >> > >>> > "snapshot-id": 2031898559602022053, >> > >>> > "timestamp-ms": 1761059868331, >> > >>> > "summary": { >> > >>> > "operation": "append", >> > >>> > "spark.app.id": "local-1761059845206", >> > >>> > "added-data-files": "1", >> > >>> > "added-records": "1", >> > >>> > "added-files-size": "657", >> > >>> > "changed-partition-count": "1", >> > >>> > "total-records": "1", >> > >>> > "total-files-size": "657", >> > >>> > "total-data-files": "1", >> > >>> > "total-delete-files": "0", >> > >>> > "total-position-deletes": "0", >> > >>> > "total-equality-deletes": "0", >> > >>> > "engine-version": "3.5.7", >> > >>> > "app-id": "local-1761059845206", >> > >>> > "engine-name": "spark", >> > >>> > "iceberg-version": "Apache Iceberg unspecified >> > >>> (commit >> > >>> > 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)" >> > >>> > }, >> > >>> > "manifest-list": >> > >>> > "s3://mybucket/fns/t3/metadata/snap-1.avro", >> > >>> > "schema-id": 0 >> > >>> > } >> > >>> > ], >> > >>> > "statistics": [], >> > >>> > "partition-statistics": [], >> > >>> > "snapshot-log": [ >> > >>> > { >> > >>> > "timestamp-ms": 1761059868331, >> > >>> > "snapshot-id": 2031898559602022053 >> > >>> > } >> > >>> > ], >> > >>> > "metadata-log": [ >> > >>> > { >> > >>> > "timestamp-ms": 1761059692095, >> > >>> > "metadata-file": >> > >>> > "s3://mybucket/fns/t3/metadata/0.metadata.json" >> > >>> > } >> > >>> > ] >> > >>> > } >> > >>> > } >> > >>> > >> > >>> > Thanks >> > >>> > >> > >>> > Fabio >> > >>> > >> > >>> > >> > >>> > From: Dmitri Bourlatchkov <[email protected]> >> > >>> > Date: Thursday, 23 October 2025 at 15:24 >> > >>> > To: [email protected] <[email protected]> >> > >>> > Subject: Re: [EXTERNAL]Re: KMS Key addition for s3 >> > >>> > >> > >>> > Hi Fabio, >> > >>> > >> > >>> > Thanks for the detailed explanation! It helps a lot. >> > >>> > >> > >>> > A couple of things are still not clear to me, though :) >> > >>> > >> > >>> > Are table metadata JSON files ever encrypted with KMS? >> > >>> > >> > >>> > If so, can Polaris read them without knowing the KMS FileIO >> > properties >> > >>> > (e.g. if the table came via registerTable with pre-encrypted >> > metadata)? >> > >>> > >> > >>> > Sorry, if these are trivial things and I missed some doc >> somewhere... >> > >>> > please give a pointer if so. >> > >>> > >> > >>> > Thanks, >> > >>> > Dmitri. >> > >>> > >> > >>> > On Thu, Oct 23, 2025 at 5:28 AM Rizzo Cascio, Fabio >> > >>> > <[email protected]> wrote: >> > >>> > >> > >>> > > Hi Dmitri, >> > >>> > > >> > >>> > > Let me try to explain what I mean: >> > >>> > > >> > >>> > > Let’s say we’re creating a catalog using AWS S3, just using the >> > >>> FileIo >> > >>> > > Iceberg properties (see: >> > >>> > > >> > >>> >> https://iceberg.apache.org/docs/nightly/aws/#s3-server-side-encryption >> > ). >> > >>> > > >> > >>> > > First, we create the catalog with a request like this: >> > >>> > > >> > >>> > > POST http://localhost:8181/api/management/v1/catalogs >> > >>> > > >> > >>> > > { >> > >>> > > "catalog": { >> > >>> > > "name": "test_catalog", >> > >>> > > "type": "INTERNAL", >> > >>> > > "readOnly": true, >> > >>> > > "properties": { >> > >>> > > "default-base-location": "s3://mybucket" >> > >>> > > }, >> > >>> > > "storageConfigInfo": { >> > >>> > > "storageType": "S3", >> > >>> > > "roleArn": "arn:aws:iam::*****:role/myrole", >> > >>> > > "region": "us-east-1" >> > >>> > > } >> > >>> > > } >> > >>> > > } >> > >>> > > >> > >>> > > >> > >>> > > Next, we create a namespace: >> > >>> > > >> > >>> > > POST >> http://localhost:8181/api/catalog/v1/test_catalog/namespaces >> > >>> > > >> > >>> > > { >> > >>> > > "namespace": ["fns"], >> > >>> > > "properties": { >> > >>> > > "location": "s3://mybucket/fns/" >> > >>> > > } >> > >>> > > } >> > >>> > > >> > >>> > > >> > >>> > > Then, we create a table: >> > >>> > > >> > >>> > > POST >> > >>> > > >> > >>> >> > http://localhost:8181/api/catalog/v1/test_catalog/namespaces/fns/tables >> > >>> > > >> > >>> > > { >> > >>> > > "name": "t3", >> > >>> > > "schema": { >> > >>> > > "type": "struct", >> > >>> > > "fields": [ >> > >>> > > {"id": 1, "name": "field1", "type": "int", "required": >> > >>> false}, >> > >>> > > {"id": 2, "name": "field2", "type": "int", "required": >> > false} >> > >>> > > ] >> > >>> > > }, >> > >>> > > "stage-create": false >> > >>> > > } >> > >>> > > >> > >>> > > >> > >>> > > To insert data into the table using Spark with a KMS key: >> > >>> > > >> > >>> > > bin/spark-sql \ >> > >>> > > --packages >> > >>> > > >> > >>> > >> > >>> >> > >> org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0,org.apache.iceberg:iceberg-aws-bundle:1.9.0 >> > >>> > > \ >> > >>> > > --conf >> > >>> > > >> > >>> > >> > >>> >> > >> spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions >> > >>> > > \ >> > >>> > > --conf >> > >>> > spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog >> > >>> > > \ >> > >>> > > --conf spark.sql.catalog.polaris.type=rest \ >> > >>> > > --conf spark.sql.catalog.polaris.uri= >> > >>> > http://localhost:8181/api/catalog >> > >>> > > \ >> > >>> > > --conf >> spark.sql.catalog.polaris.token-refresh-enabled=false \ >> > >>> > > --conf spark.sql.catalog.polaris.warehouse=test_catalog \ >> > >>> > > --conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \ >> > >>> > > --conf >> > >>> > > >> > >>> > >> > >>> >> > >> spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation=vended-credentials >> > >>> > > \ >> > >>> > > --conf >> > >>> > > >> > >>> > >> > >>> >> > >> spark.sql.catalog.polaris.s3.sse.key=arn:aws:kms:us-east-1:****:key/mykey \ >> > >>> > > --conf spark.sql.catalog.polaris.s3.sse.type=kms \ >> > >>> > > --conf >> > >>> > > >> > >>> > >> > >>> >> > >> spark.sql.catalog.polaris.credential=38a0ebdbcc97c58f:654af690485a32f02ce17dda7cd2c36d >> > >>> > > >> > >>> > > >> > >>> > > Then run: >> > >>> > > >> > >>> > > insert into t3 values (7,9); >> > >>> > > >> > >>> > > >> > >>> > > This will create an entry in the bucket with the data encrypted >> > >>> using the >> > >>> > > specified key. >> > >>> > > >> > >>> > > If you remove these two properties from the connection command: >> > >>> > > >> > >>> > > * --conf >> > >>> > > >> > >>> >> > >> spark.sql.catalog.polaris.s3.sse.key=arn:aws:kms:us-east-1:****:key/mykey >> > >>> > > >> > >>> > > * --conf spark.sql.catalog.polaris.s3.sse.type=kms >> > >>> > > >> > >>> > > then you can insert data into the table without encryption. >> > >>> > > >> > >>> > > In Polaris, at the point of the insert, we don’t know the KMS >> key, >> > so >> > >>> > > there’s no way to add that key in the policy. This is where >> adding >> > >>> > > arn:aws:kms:%s:%s:key/* helps. >> > >>> > > >> > >>> > > Adding a new property in the storageConfigInfo for the KMS key >> will >> > >>> help >> > >>> > > with creating a more restrictive IAM policy, but we also need to >> > >>> support >> > >>> > > the existing properties defined in Iceberg. >> > >>> > > >> > >>> > > The reason for my changes—passing the properties down to >> > >>> > > AwsCredentialsStorageIntegration—is because you can pass extra >> > >>> properties >> > >>> > > to any Polaris request, and those will feed down to Iceberg. For >> > >>> example: >> > >>> > > >> > >>> > > { >> > >>> > > "name": "t3", >> > >>> > > "schema": { >> > >>> > > "type": "struct", >> > >>> > > "fields": [ >> > >>> > > {"id": 1, "name": "field1", "type": "int", "required": >> > >>> false}, >> > >>> > > {"id": 2, "name": "field2", "type": "int", "required": >> > false} >> > >>> > > ] >> > >>> > > }, >> > >>> > > "properties": { >> > >>> > > "s3.sse.type": "kms", >> > >>> > > "s3.sse.key": "arn:aws:kms:us-east-1:***:key/mykey" >> > >>> > > }, >> > >>> > > "stage-create": false >> > >>> > > } >> > >>> > > >> > >>> > > >> > >>> > > With my changes I was trying to support both ways, not sure if >> this >> > >>> is >> > >>> > the >> > >>> > > best approach or just going for the easier way of allowing all >> the >> > >>> keys >> > >>> > > under the account is better. >> > >>> > > >> > >>> > > Let me know if you need any more details. >> > >>> > > >> > >>> > > Thanks >> > >>> > > >> > >>> > > Fabio >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> > > 1. >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> > > From: Dmitri Bourlatchkov <[email protected]> >> > >>> > > Date: Wednesday, 22 October 2025 at 18:50 >> > >>> > > To: [email protected] <[email protected]> >> > >>> > > Subject: Re: [EXTERNAL]Re: KMS Key addition for s3 >> > >>> > > >> > >>> > > Hi Fabio, >> > >>> > > >> > >>> > > I agree that we seem to have some confusion here :) let's try >> and >> > >>> > clarify. >> > >>> > > >> > >>> > > What do you mean exactly by "s3 iceberg properties"? Are they >> > Iceberg >> > >>> > table >> > >>> > > metadata properties? Iceberg (REST) Catalog properties? FileIO >> > >>> > properties? >> > >>> > > >> > >>> > > Why would Polaris need to support them? What would an end-to-end >> > >>> flow be >> > >>> > > between Polaris and the client (e.g. Spark)? >> > >>> > > >> > >>> > > Thanks, >> > >>> > > Dmitri. >> > >>> > > >> > >>> > > On Wed, Oct 22, 2025 at 12:08 PM Rizzo Cascio, Fabio >> > >>> > > <[email protected]> wrote: >> > >>> > > >> > >>> > > > I feel like there’s something I’m overlooking. >> > >>> > > > Setting the kms property at the catalog level works, but it >> > doesn’t >> > >>> > > > support the s3 iceberg properties, right? >> > >>> > > > >> > >>> > > > To support those, the best approach is to let clients use the >> s3 >> > >>> > iceberg >> > >>> > > > properties directly. Then, in >> AwsCredentialsStorageIntegration, >> > we >> > >>> can >> > >>> > > > simply add a policy that allows any key within that account. >> This >> > >>> way, >> > >>> > we >> > >>> > > > don’t need to rely on metadata properties to retrieve the key >> > >>> value. >> > >>> > > > >> > >>> > > > Does that make sense? >> > >>> > > > >> > >>> > > > >> > >>> > > > Thanks >> > >>> > > > Fabio >> > >>> > > > >> > >>> > > > >> > >>> > > > From: Dmitri Bourlatchkov <[email protected]> >> > >>> > > > Date: Wednesday, 22 October 2025 at 15:42 >> > >>> > > > To: [email protected] <[email protected]> >> > >>> > > > Subject: Re: [EXTERNAL]Re: KMS Key addition for s3 >> > >>> > > > >> > >>> > > > Hi Fabio, >> > >>> > > > >> > >>> > > > Yes, that would be the preferable solution for KMS from my >> POV. >> > >>> > > > >> > >>> > > > I wonder what other people think about this too :) >> > >>> > > > >> > >>> > > > Cheers, >> > >>> > > > Dmitri. >> > >>> > > > >> > >>> > > > On Wed, Oct 22, 2025 at 9:45 AM Rizzo Cascio, Fabio >> > >>> > > > <[email protected]> wrote: >> > >>> > > > >> > >>> > > > > Hi Dmitri, >> > >>> > > > > >> > >>> > > > > If I add the kms props to AccessConfig in >> AWSCredentialStorage >> > >>> I can >> > >>> > > see >> > >>> > > > > it in the LoadTable response coming from Polaris. >> > >>> > > > > >> > >>> > > > > Fabio >> > >>> > > > > >> > >>> > > > > From: Dmitri Bourlatchkov <[email protected]> >> > >>> > > > > Date: Tuesday, 21 October 2025 at 15:23 >> > >>> > > > > To: [email protected] <[email protected]> >> > >>> > > > > Subject: Re: [EXTERNAL]Re: KMS Key addition for s3 >> > >>> > > > > >> > >>> > > > > Hi Fabio, >> > >>> > > > > >> > >>> > > > > Yes, I glimpsed that from your email. Sorry if my post >> caused >> > >>> > > confusion. >> > >>> > > > I >> > >>> > > > > just wanted to reply to the top email as what I'm proposing >> > >>> seems to >> > >>> > > be a >> > >>> > > > > key feature for KMS support. >> > >>> > > > > >> > >>> > > > > Would you be able to validate whether sending KMS FileIO >> > >>> properties >> > >>> > to >> > >>> > > > > clients from LoadTable responses work in practice (e.g. in >> > >>> Spark)? >> > >>> > > > > >> > >>> > > > > I believe this can be done by adding KMS properties as >> "extra" >> > >>> > > properties >> > >>> > > > > to AccessConfig. >> > >>> > > > > >> > >>> > > > > Thanks, >> > >>> > > > > Dmitri. >> > >>> > > > > >> > >>> > > > > On Tue, Oct 21, 2025 at 4:15 AM Rizzo Cascio, Fabio >> > >>> > > > > <[email protected]> wrote: >> > >>> > > > > >> > >>> > > > > > Hi Dmitri, >> > >>> > > > > > >> > >>> > > > > > This is what I was saying in my other email. Anyway I’m >> gonna >> > >>> > update >> > >>> > > my >> > >>> > > > > PR >> > >>> > > > > > with the changes I have made to get it working, the >> project >> > >>> won’t >> > >>> > > > build >> > >>> > > > > > because I haven’t update the tests etc, I just want to >> show >> > my >> > >>> > > changes >> > >>> > > > > and >> > >>> > > > > > see if we can agree on a direction before I make all the >> > >>> changes. >> > >>> > > > > > >> > >>> > > > > > Thanks >> > >>> > > > > > >> > >>> > > > > > Fabio >> > >>> > > > > > >> > >>> > > > > > >> > >>> > > > > > >> > >>> > > > > > >> > >>> > > > > > From: Dmitri Bourlatchkov <[email protected]> >> > >>> > > > > > Date: Monday, 20 October 2025 at 17:38 >> > >>> > > > > > To: [email protected] <[email protected]> >> > >>> > > > > > Subject: [EXTERNAL]Re: KMS Key addition for s3 >> > >>> > > > > > >> > >>> > > > > > Hi Fabio, Ashok and All, >> > >>> > > > > > >> > >>> > > > > > Apologies if I'm missing something obvious, but the two >> WIP >> > >>> KMS PRs >> > >>> > > > > [1424] >> > >>> > > > > > [2802] appear to be dealing only with AWS policies on the >> > >>> vended >> > >>> > > > > credential >> > >>> > > > > > session. They do not appear to deal with client >> configuration >> > >>> (in >> > >>> > > > > LoadTable >> > >>> > > > > > responses). >> > >>> > > > > > >> > >>> > > > > > As far as I understand, Iceberg clients need certain >> FileIO >> > >>> > > properties >> > >>> > > > to >> > >>> > > > > > be set in order to utilize KMS. >> > >>> > > > > > >> > >>> > > > > > I'd imagine that Polaris ought to provide these FileIO >> > >>> properties >> > >>> > in >> > >>> > > > > > LoadTable responses in addition to granting privileges for >> > KMS >> > >>> > access >> > >>> > > > to >> > >>> > > > > > the vended (session) credentials. >> > >>> > > > > > >> > >>> > > > > > In other words, the decision whether to use KMS rests with >> > >>> Polaris >> > >>> > > (we >> > >>> > > > > can >> > >>> > > > > > discuss how to configure that). If that is enabled, >> clients >> > >>> should >> > >>> > > not >> > >>> > > > > need >> > >>> > > > > > any extra configuration, they should get complete and >> usable >> > >>> > > > > > configuration + credentials from Polaris. >> > >>> > > > > > >> > >>> > > > > > WDYT? >> > >>> > > > > > >> > >>> > > > > > [1424] https://github.com/apache/polaris/pull/1424 >> > >>> > > > > > [2802] https://github.com/apache/polaris/pull/2802 >> > >>> > > > > > >> > >>> > > > > > Thanks, >> > >>> > > > > > Dmitri. >> > >>> > > > > > >> > >>> > > > > > >> > >>> > > > > > On Mon, Oct 13, 2025 at 3:50 AM Rizzo Cascio, Fabio >> > >>> > > > > > <[email protected]> wrote: >> > >>> > > > > > >> > >>> > > > > > > Hi guys, >> > >>> > > > > > > >> > >>> > > > > > > I have created a new PR to be able to use a kms key for >> the >> > >>> S3 >> > >>> > > > bucket, >> > >>> > > > > it >> > >>> > > > > > > is mandatory for me to use any S3 storage and hopefully >> a >> > >>> good >> > >>> > > > addition >> > >>> > > > > > for >> > >>> > > > > > > other people that want to use it. >> > >>> > > > > > > >> > >>> > > > > > > PR link: https://github.com/apache/polaris/pull/2802 >> > >>> > > > > > > >> > >>> > > > > > > Thanks >> > >>> > > > > > > >> > >>> > > > > > > Fabio >> > >>> > > > > > > >> > >>> > > > > > > This message is confidential and subject to terms at: >> > >>> > > > > > > https://www.jpmorgan.com/emaildisclaimer including on >> > >>> > > confidential, >> > >>> > > > > > > privileged or legal entity information, malicious >> content >> > and >> > >>> > > > > monitoring >> > >>> > > > > > of >> > >>> > > > > > > electronic messages. If you are not the intended >> recipient, >> > >>> > please >> > >>> > > > > delete >> > >>> > > > > > > this message and notify the sender immediately. Any >> > >>> unauthorized >> > >>> > > use >> > >>> > > > is >> > >>> > > > > > > strictly prohibited. >> > >>> > > > > > > >> > >>> > > > > > >> > >>> > > > > > This message is confidential and subject to terms at: >> > >>> > > > > > https://www.jpmorgan.com/emaildisclaimer including on >> > >>> > confidential, >> > >>> > > > > > privileged or legal entity information, malicious content >> and >> > >>> > > > monitoring >> > >>> > > > > of >> > >>> > > > > > electronic messages. If you are not the intended >> recipient, >> > >>> please >> > >>> > > > delete >> > >>> > > > > > this message and notify the sender immediately. Any >> > >>> unauthorized >> > >>> > use >> > >>> > > is >> > >>> > > > > > strictly prohibited. >> > >>> > > > > > >> > >>> > > > > >> > >>> > > > > This message is confidential and subject to terms at: >> > >>> > > > > https://www.jpmorgan.com/emaildisclaimer including on >> > >>> confidential, >> > >>> > > > > privileged or legal entity information, malicious content >> and >> > >>> > > monitoring >> > >>> > > > of >> > >>> > > > > electronic messages. If you are not the intended recipient, >> > >>> please >> > >>> > > delete >> > >>> > > > > this message and notify the sender immediately. Any >> > unauthorized >> > >>> use >> > >>> > is >> > >>> > > > > strictly prohibited. >> > >>> > > > > >> > >>> > > > >> > >>> > > > This message is confidential and subject to terms at: >> > >>> > > > https://www.jpmorgan.com/emaildisclaimer including on >> > >>> confidential, >> > >>> > > > privileged or legal entity information, malicious content and >> > >>> > monitoring >> > >>> > > of >> > >>> > > > electronic messages. If you are not the intended recipient, >> > please >> > >>> > delete >> > >>> > > > this message and notify the sender immediately. Any >> unauthorized >> > >>> use is >> > >>> > > > strictly prohibited. >> > >>> > > > >> > >>> > > >> > >>> > > This message is confidential and subject to terms at: >> > >>> > > https://www.jpmorgan.com/emaildisclaimer including on >> > confidential, >> > >>> > > privileged or legal entity information, malicious content and >> > >>> monitoring >> > >>> > of >> > >>> > > electronic messages. If you are not the intended recipient, >> please >> > >>> delete >> > >>> > > this message and notify the sender immediately. Any unauthorized >> > use >> > >>> is >> > >>> > > strictly prohibited. >> > >>> > > >> > >>> > >> > >>> > This message is confidential and subject to terms at: >> > >>> > https://www.jpmorgan.com/emaildisclaimer including on >> confidential, >> > >>> > privileged or legal entity information, malicious content and >> > >>> monitoring of >> > >>> > electronic messages. If you are not the intended recipient, please >> > >>> delete >> > >>> > this message and notify the sender immediately. Any unauthorized >> use >> > is >> > >>> > strictly prohibited. >> > >>> > >> > >>> >> > >>> This message is confidential and subject to terms at: >> > >>> https://www.jpmorgan.com/emaildisclaimer including on confidential, >> > >>> privileged or legal entity information, malicious content and >> > monitoring of >> > >>> electronic messages. If you are not the intended recipient, please >> > delete >> > >>> this message and notify the sender immediately. Any unauthorized >> use is >> > >>> strictly prohibited. >> > >>> >> > >> >> > >> >
