I can’t attach any files, so ASCII diagram it is.
Check Properties
|
v
Does iceberg s3 key exist?
/ \
Yes No
/ \
Does kmsKeyArn exist? Does kmsKeyArn exist?
/ \ / \
Yes No Yes No
| | | |
Add both Use iceberg Revert Don't add
iceberg s3 key only to any
s3 key - add to kmsKeyArn policies
and policies - add to
kmsKeyArn policies
to policies
From: Rizzo Cascio, Fabio <[email protected]>
Date: Monday, 20 October 2025 at 12:05
To: [email protected] <[email protected]>
Subject: Re: [EXTERNAL]Re: KMS Key addition for s3
Re-attached the chart as image.
From: Rizzo Cascio, Fabio <[email protected]>
Date: Monday, 20 October 2025 at 10:33
To: [email protected] <[email protected]>
Subject: Re: [EXTERNAL]Re: KMS Key addition for s3
Hi guys,
I have done a little bit of research in the past few days; I needed to
understand better how the s3 key properties are passed around in the config
(Polaris and iceberg).
What I have done is:
1.
Updated DefaultFileIOFactory to pass down to the credentials storage
integrations the props that come in (loadFileIO method). In most cases we have
the s3 keys props. Not sure if this is the best way, any suggestions is welcome.
2.
In AwsCredentialsStorageIntegration at this point we might have 2 keys one from
the kmsKeyArn prop that I have added in storageConfigInfo and the one coming
in from the iceberg config
3.
My logic at this point will use the iceberg property as preference if it exists
otherwise revert on the kmsKeyArn if that doesn’t exist don’t add any policies
and if both exists and both keys to the policies.
4.
[flowchart-v2-Iceberg_Property_Preference_2025_10_20 (1).png]
4. One thing that I find out is that while writing to a table using the spark
client and passing the s3 key props ( --conf
spark.sql.catalog.polaris.s3.sse.key=arn:aws:kms:us-west-1:****:key/***\
--conf spark.sql.catalog.polaris.s3.sse.type=kms\) these values don't
appear in the properties that reach AwsCredentialsStorageIntegration. Any help
to understand where I can capture those properties correctly would be greatly
appreciated
Thanks
Fabio
From: Ashok Krishna <[email protected]>
Date: Thursday, 16 October 2025 at 18:19
To: [email protected] <[email protected]>
Subject: Re: [EXTERNAL]Re: KMS Key addition for s3
Hi JB / MK / Yufei / Dmitri
Previously there was an alignment to support KMS support at multiple levels
1. Default with no key configuration
2. Catalog level with key configuration
3. Table level key configuration
It seems the PR is leaning towards handling #2.
I initially wanted to address #1 as part of this PR
<https://github.com/apache/polaris/pull/1424/files>, but unfortunately got
delayed. We didn't want to create friction with users for data lakes in
case of hard rotation(disable and create new) of keys. I wanted to
understand the current stance on KMS.Could you let us know your thoughts on
#1?
On Tue, Oct 14, 2025 at 11:22 PM Michael Collado <[email protected]>
wrote:
> Sorry, my ask re: 2 was to include this support in the PR - right now in
>
> https://github.com/apache/polaris/pull/2802/files#diff-d305f7a426a7690c576722c114257792b3fcee726624655d15893b71499827f8R181
> , the KMS key is specified by
> reading storageConfigurationInfo.getKmsKeyArn() in the storage integration.
> If someone creates a table with a different key, as in your example, that
> key will be ignored in the policy generation.
>
> Mike
>
> On Tue, Oct 14, 2025 at 7:02 AM Rizzo Cascio, Fabio
> <[email protected]> wrote:
>
> > Hi Mike,
> >
> > 2. Yes, that is correct. The override should be already doable in
> Polaris
> > during a table creation, something like this:
> >
> > {
> > "name": "my_table",
> > "location": "s3://your-bucket/path/to/table",
> > "schema": {
> > "type": "struct",
> > "fields": [
> > { "id": 1, "name": "id", "type": "long", "required": true },
> > { "id": 2, "name": "data", "type": "string", "required": false }
> > ]
> > },
> > "properties": {
> > "s3.encryption.key":
> > "arn:aws:kms:us-west-2:123456789012:key/your-key-id"
> > }
> > }
> >
> > The options are:
> > 1. use the kms key when you create the catalog so that all data in the
> > bucket is encrypted with the same key
> > 2. Don't use the kms key when you create the catalog and use different
> > keys when you create tables
> >
> > We use number one cause number 2 gets expensive when you have a lot of
> > data.
> >
> >
> > 1.
> >
> >
> > From: Michael Collado <[email protected]>
> > Date: Monday, 13 October 2025 at 18:07
> > To: [email protected] <[email protected]>
> > Subject: [EXTERNAL]Re: KMS Key addition for s3
> >
> > Very exciting change. A couple of questions/suggestions:
> >
> > 1. Ideally, the IAM policy statement should have the encryption context
> set
> > so that the key can only be used to en/decrypt files under the specific
> > table path.
> > 2. The structure suggests that all tables under the catalog will use the
> > same KMS key for encryption. Should we support overriding the key at the
> > table level? The S3FileIOProperties class in Iceberg defines the
> s3.sse.key
> > property at the table-level so each table can specify its own KMS key.
> Can
> > we allow for that override, if present?
> >
> > Mike
> >
> > On Mon, Oct 13, 2025 at 7:27 AM Jean-Baptiste Onofré <[email protected]>
> > wrote:
> >
> > > Hi Fabio
> > >
> > > Thanks for the PR ! We will take a look on it.
> > >
> > > Regards
> > > JB
> > >
> > > On Mon, Oct 13, 2025 at 9:50 AM Rizzo Cascio, Fabio
> > > <[email protected]> wrote:
> > > >
> > > > Hi guys,
> > > >
> > > > I have created a new PR to be able to use a kms key for the S3
> bucket,
> > > it is mandatory for me to use any S3 storage and hopefully a good
> > addition
> > > for other people that want to use it.
> > > >
> > > > PR link: https://github.com/apache/polaris/pull/2802
> > > >
> > > > Thanks
> > > >
> > > > Fabio
> > > >
> > > > This message is confidential and subject to terms at:
> > > https://www.jpmorgan.com/emaildisclaimer including on confidential,
> > > privileged or legal entity information, malicious content and
> monitoring
> > of
> > > electronic messages. If you are not the intended recipient, please
> delete
> > > this message and notify the sender immediately. Any unauthorized use is
> > > strictly prohibited.
> > >
> >
> > This message is confidential and subject to terms at:
> > https://www.jpmorgan.com/emaildisclaimer including on confidential,
> > privileged or legal entity information, malicious content and monitoring
> of
> > electronic messages. If you are not the intended recipient, please delete
> > this message and notify the sender immediately. Any unauthorized use is
> > strictly prohibited.
> >
>
This message is confidential and subject to terms at:
https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged
or legal entity information, malicious content and monitoring of electronic
messages. If you are not the intended recipient, please delete this message and
notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at:
https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged
or legal entity information, malicious content and monitoring of electronic
messages. If you are not the intended recipient, please delete this message and
notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at:
https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged
or legal entity information, malicious content and monitoring of electronic
messages. If you are not the intended recipient, please delete this message and
notify the sender immediately. Any unauthorized use is strictly prohibited.
