Hi Fabio, Re: your point (4): I do not think Polaris has any knowledge of client configuration at all.
Polaris (as an IRC impl.) can send config overwrites to clients, but we need to review how they work exactly on the client side with respect to KMS (I do not know off the top of my head). Cheers, Dmitri. On Mon, Oct 20, 2025 at 5:34 AM Rizzo Cascio, Fabio <[email protected]> wrote: > Hi guys, > > I have done a little bit of research in the past few days; I needed to > understand better how the s3 key properties are passed around in the config > (Polaris and iceberg). > > What I have done is: > > > 1. Updated DefaultFileIOFactory to pass down to the credentials > storage integrations the props that come in (loadFileIO method). In most > cases we have the s3 keys props. Not sure if this is the best way, any > suggestions is welcome. > 2. In AwsCredentialsStorageIntegration at this point we might have 2 > keys one from the kmsKeyArn prop that I have added in storageConfigInfo > and the one coming in from the iceberg config > 3. My logic at this point will use the iceberg property as preference > if it exists otherwise revert on the kmsKeyArn if that doesn’t exist don’t > add any policies and if both exists and both keys to the policies. > 4. > > [image: flowchart-v2-Iceberg_Property_Preference_2025_10_20 (1).png] > 4. One thing that I find out is that while writing to a table using the > spark client and passing the s3 key props ( --conf > spark.sql.catalog.polaris.s3.sse.key=arn:aws:kms:us-west-1:****:key/***\ > --conf spark.sql.catalog.polaris.s3.sse.type=kms\) these values don't > appear in the properties that reach AwsCredentialsStorageIntegration. Any > help to understand where I can capture those properties correctly would be > greatly appreciated > > Thanks > > Fabio > > *From: *Ashok Krishna <[email protected]> > *Date: *Thursday, 16 October 2025 at 18:19 > *To: *[email protected] <[email protected]> > *Subject: *Re: [EXTERNAL]Re: KMS Key addition for s3 > > Hi JB / MK / Yufei / Dmitri > > Previously there was an alignment to support KMS support at multiple levels > > 1. Default with no key configuration > 2. Catalog level with key configuration > 3. Table level key configuration > > It seems the PR is leaning towards handling #2. > > I initially wanted to address #1 as part of this PR > <https://github.com/apache/polaris/pull/1424/files>, but unfortunately got > delayed. We didn't want to create friction with users for data lakes in > case of hard rotation(disable and create new) of keys. I wanted to > understand the current stance on KMS.Could you let us know your thoughts on > #1? > > On Tue, Oct 14, 2025 at 11:22 PM Michael Collado <[email protected]> > wrote: > > > Sorry, my ask re: 2 was to include this support in the PR - right now in > > > > > https://github.com/apache/polaris/pull/2802/files#diff-d305f7a426a7690c576722c114257792b3fcee726624655d15893b71499827f8R181 > > , the KMS key is specified by > > reading storageConfigurationInfo.getKmsKeyArn() in the storage > integration. > > If someone creates a table with a different key, as in your example, that > > key will be ignored in the policy generation. > > > > Mike > > > > On Tue, Oct 14, 2025 at 7:02 AM Rizzo Cascio, Fabio > > <[email protected]> wrote: > > > > > Hi Mike, > > > > > > 2. Yes, that is correct. The override should be already doable in > > Polaris > > > during a table creation, something like this: > > > > > > { > > > "name": "my_table", > > > "location": "s3://your-bucket/path/to/table", > > > "schema": { > > > "type": "struct", > > > "fields": [ > > > { "id": 1, "name": "id", "type": "long", "required": true }, > > > { "id": 2, "name": "data", "type": "string", "required": false } > > > ] > > > }, > > > "properties": { > > > "s3.encryption.key": > > > "arn:aws:kms:us-west-2:123456789012:key/your-key-id" > > > } > > > } > > > > > > The options are: > > > 1. use the kms key when you create the catalog so that all data in the > > > bucket is encrypted with the same key > > > 2. Don't use the kms key when you create the catalog and use different > > > keys when you create tables > > > > > > We use number one cause number 2 gets expensive when you have a lot of > > > data. > > > > > > > > > 1. > > > > > > > > > From: Michael Collado <[email protected]> > > > Date: Monday, 13 October 2025 at 18:07 > > > To: [email protected] <[email protected]> > > > Subject: [EXTERNAL]Re: KMS Key addition for s3 > > > > > > Very exciting change. A couple of questions/suggestions: > > > > > > 1. Ideally, the IAM policy statement should have the encryption context > > set > > > so that the key can only be used to en/decrypt files under the specific > > > table path. > > > 2. The structure suggests that all tables under the catalog will use > the > > > same KMS key for encryption. Should we support overriding the key at > the > > > table level? The S3FileIOProperties class in Iceberg defines the > > s3.sse.key > > > property at the table-level so each table can specify its own KMS key. > > Can > > > we allow for that override, if present? > > > > > > Mike > > > > > > On Mon, Oct 13, 2025 at 7:27 AM Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > > > Hi Fabio > > > > > > > > Thanks for the PR ! We will take a look on it. > > > > > > > > Regards > > > > JB > > > > > > > > On Mon, Oct 13, 2025 at 9:50 AM Rizzo Cascio, Fabio > > > > <[email protected]> wrote: > > > > > > > > > > Hi guys, > > > > > > > > > > I have created a new PR to be able to use a kms key for the S3 > > bucket, > > > > it is mandatory for me to use any S3 storage and hopefully a good > > > addition > > > > for other people that want to use it. > > > > > > > > > > PR link: https://github.com/apache/polaris/pull/2802 > > > > > > > > > > Thanks > > > > > > > > > > Fabio > > > > > > > > > > This message is confidential and subject to terms at: > > > > https://www.jpmorgan.com/emaildisclaimer including on confidential, > > > > privileged or legal entity information, malicious content and > > monitoring > > > of > > > > electronic messages. If you are not the intended recipient, please > > delete > > > > this message and notify the sender immediately. Any unauthorized use > is > > > > strictly prohibited. > > > > > > > > > > This message is confidential and subject to terms at: > > > https://www.jpmorgan.com/emaildisclaimer including on confidential, > > > privileged or legal entity information, malicious content and > monitoring > > of > > > electronic messages. If you are not the intended recipient, please > delete > > > this message and notify the sender immediately. Any unauthorized use is > > > strictly prohibited. > > > > > > This message is confidential and subject to terms at: > https://www.jpmorgan.com/emaildisclaimer including on confidential, > privileged or legal entity information, malicious content and monitoring of > electronic messages. If you are not the intended recipient, please delete > this message and notify the sender immediately. Any unauthorized use is > strictly prohibited. >
