Re: [DISCUSS] REST Catalog: Credential management for KMS/Vault and Table level Encryption
Hi all, Thanks everyone for the discussion and comments on the KMS credential vending spec proposal doc. I opened a draft PR to move the current proposal into the REST OpenAPI spec: https://github.com/apache/iceberg/pull/17155 Please take a look and continue review on the PR. Cheers, Adam On Mon, 8 Jun 2026 at 12:22, Adam Szita wrote: > Hi folks, > > I wrote up a spec proposal for the KMS credential vending part of this > discussion: > > https://docs.google.com/document/d/1VSewbVmjukU5eTiZruCJVmUZcRZvcAsd4JOeC6Lt3fo > > The intent is to separate this from the current key-metadata / > encryption-metadata work, so that the existing PR can hopefully move > forward independently while we discuss the REST credential-vending shape > here. > At a high level, the proposal mirrors storage credential vending: the REST > catalog can return short-lived, scoped key-management-credentials, and > clients match those credentials to KMS key IDs referenced by Iceberg > encryption metadata. > Any feedback welcome, especially on whether this is the right REST-level > direction and if the proposed response shape is enough for current > table-level encryption while also leaving room for future column-level > encryption. > > Cheers, > Adam > > On Thu, 7 May 2026 at 13:56, Ádám Szita wrote: > >> Hi all, >> >> I think there are two related but separate topics here: >> >> 1. supporting Iceberg table encryption with a REST catalog client, and >> 2. defining a REST mechanism for vending/refreshing KMS or Vault >> credentials. >> >> For the current PR, I don’t think we necessarily need to solve #2. >> Yes, REST catalog deployments may use credential vending for storage >> access, >> but that is still a catalog feature/configuration choice rather than a >> requirement >> for REST catalog usage. So, similarly, I don’t think supporting table >> encryption >> with a REST catalog should require the catalog to also implement KMS >> credential vending. >> >> I do think it's a good idea to have the KMS creds provided by the catalog >> side, >> especially for catalogs that want to enforce least privilege at table >> scope. >> But I see that as a follow-up design/API discussion. I also think it >> should be >> general enough to avoid coupling certain cloud storage types with their >> respective KMS offerings. (e.g. S3 storage doesn't necessarily imply AWS >> KMS >> usage). >> >> We have KeyManagementClient interface that users can implement >> for their own KMS needs. Large enterprise deployments may already use an >> internal KMS, I don’t think we should require them to implement both >> this interface and KMS credential vending before they can use table >> encryption with a REST catalog. >> >> Let me know what you all think. >> >> Cheers, >> Adam >> >> On 2026/05/06 04:59:23 Prashant Singh wrote: >> > Thanks everyone for chiming in on this, >> > >> > *Regarding the PR : * >> > The approach we are taking here where we pick encryption creds from the >> > client side (catalog level) and storage creds from server vended creds, >> is >> > fairly concerning to me. >> > if the REST server is giving one access because someone has *SELECT* >> > grant, *SERVER* should mint them everything that required for the >> client to >> > successfully read the table, sending storage creds but not KMS creds >> > feels incorrect to me, how is the server gonna know the client has creds >> > for *decryption* for this already ? This is entirely different from what >> > other catalogs expect i.e entirely a client side configuration for >> example >> > Hive Catalog / Glue catalog etc >> > >> > Regarding backward compatibility, if we do this, server vending KMS >> > credentials eventually, these clients will simply not work since they >> > always expect the credentials to come from the client side (Note : I am >> > strictly talking of table level) >> > >> > *Regarding the Spec Change : * >> > It makes sense to me as well, to decouple KMS & storage creds ! >> > >> > All in all I believe we should conclude this discussion and not rush >> into >> > things (we can make incremental progress for sure), if we have open >> > questions and an awaiting spec change. >> > >> > Best, >> > Prashant Singh >> > >> > On Tue, May 5, 2026 at 12:11 AM Gidon Gershinsky >> wrote: >> > >> > > Hi all, >> > > >> > > The built-in encryption is designed to protect tables in any storage >> > > backend, including untrusted storage. So I also think the storage >> > > credentials and KMS credentials are best treated separately, as they >> don't >> > > have the same purpose/scope. >> > > >> > > Regarding the current PR (13225) - it doesn't handle credentials >> directly; >> > > instead, it focuses on building the scaffolding for table encryption >> in the >> > > REST catalog client - similarly to the Hive catalog client (and other >> > > catalogs in the future). >> > > >> > > Cheers, Gidon >> > > >> > > >> > > On Tue, May 5, 2026 at 3:40 AM Chris Lu wrote: >> > > >> > >> Hi Prashant, >>
Re: [DISCUSS] REST Catalog: Credential management for KMS/Vault and Table level Encryption
Hi folks, I wrote up a spec proposal for the KMS credential vending part of this discussion: https://docs.google.com/document/d/1VSewbVmjukU5eTiZruCJVmUZcRZvcAsd4JOeC6Lt3fo The intent is to separate this from the current key-metadata / encryption-metadata work, so that the existing PR can hopefully move forward independently while we discuss the REST credential-vending shape here. At a high level, the proposal mirrors storage credential vending: the REST catalog can return short-lived, scoped key-management-credentials, and clients match those credentials to KMS key IDs referenced by Iceberg encryption metadata. Any feedback welcome, especially on whether this is the right REST-level direction and if the proposed response shape is enough for current table-level encryption while also leaving room for future column-level encryption. Cheers, Adam On Thu, 7 May 2026 at 13:56, Ádám Szita wrote: > Hi all, > > I think there are two related but separate topics here: > > 1. supporting Iceberg table encryption with a REST catalog client, and > 2. defining a REST mechanism for vending/refreshing KMS or Vault > credentials. > > For the current PR, I don’t think we necessarily need to solve #2. > Yes, REST catalog deployments may use credential vending for storage > access, > but that is still a catalog feature/configuration choice rather than a > requirement > for REST catalog usage. So, similarly, I don’t think supporting table > encryption > with a REST catalog should require the catalog to also implement KMS > credential vending. > > I do think it's a good idea to have the KMS creds provided by the catalog > side, > especially for catalogs that want to enforce least privilege at table > scope. > But I see that as a follow-up design/API discussion. I also think it > should be > general enough to avoid coupling certain cloud storage types with their > respective KMS offerings. (e.g. S3 storage doesn't necessarily imply AWS > KMS > usage). > > We have KeyManagementClient interface that users can implement > for their own KMS needs. Large enterprise deployments may already use an > internal KMS, I don’t think we should require them to implement both > this interface and KMS credential vending before they can use table > encryption with a REST catalog. > > Let me know what you all think. > > Cheers, > Adam > > On 2026/05/06 04:59:23 Prashant Singh wrote: > > Thanks everyone for chiming in on this, > > > > *Regarding the PR : * > > The approach we are taking here where we pick encryption creds from the > > client side (catalog level) and storage creds from server vended creds, > is > > fairly concerning to me. > > if the REST server is giving one access because someone has *SELECT* > > grant, *SERVER* should mint them everything that required for the client > to > > successfully read the table, sending storage creds but not KMS creds > > feels incorrect to me, how is the server gonna know the client has creds > > for *decryption* for this already ? This is entirely different from what > > other catalogs expect i.e entirely a client side configuration for > example > > Hive Catalog / Glue catalog etc > > > > Regarding backward compatibility, if we do this, server vending KMS > > credentials eventually, these clients will simply not work since they > > always expect the credentials to come from the client side (Note : I am > > strictly talking of table level) > > > > *Regarding the Spec Change : * > > It makes sense to me as well, to decouple KMS & storage creds ! > > > > All in all I believe we should conclude this discussion and not rush into > > things (we can make incremental progress for sure), if we have open > > questions and an awaiting spec change. > > > > Best, > > Prashant Singh > > > > On Tue, May 5, 2026 at 12:11 AM Gidon Gershinsky > wrote: > > > > > Hi all, > > > > > > The built-in encryption is designed to protect tables in any storage > > > backend, including untrusted storage. So I also think the storage > > > credentials and KMS credentials are best treated separately, as they > don't > > > have the same purpose/scope. > > > > > > Regarding the current PR (13225) - it doesn't handle credentials > directly; > > > instead, it focuses on building the scaffolding for table encryption > in the > > > REST catalog client - similarly to the Hive catalog client (and other > > > catalogs in the future). > > > > > > Cheers, Gidon > > > > > > > > > On Tue, May 5, 2026 at 3:40 AM Chris Lu wrote: > > > > > >> Hi Prashant, > > >> > > >> My own preference would be to keep storage credentials and > encryption/KMS > > >> credentials separate in the REST contract. > > >> > > >> Storage credentials and KMS/Vault credentials have different scopes, > > >> lifetimes, providers, and failure modes. > > >> > > >> > > >>- Storage credentials authorize access to object locations. > > >>- > > >> > > >>KMS/Vault credentials authorize key wrap/unwrap operations. > > >> > > >> Object-store SSE-KMS is a storage-
Re: [DISCUSS] REST Catalog: Credential management for KMS/Vault and Table level Encryption
Hi all, I think there are two related but separate topics here: 1. supporting Iceberg table encryption with a REST catalog client, and 2. defining a REST mechanism for vending/refreshing KMS or Vault credentials. For the current PR, I don’t think we necessarily need to solve #2. Yes, REST catalog deployments may use credential vending for storage access, but that is still a catalog feature/configuration choice rather than a requirement for REST catalog usage. So, similarly, I don’t think supporting table encryption with a REST catalog should require the catalog to also implement KMS credential vending. I do think it's a good idea to have the KMS creds provided by the catalog side, especially for catalogs that want to enforce least privilege at table scope. But I see that as a follow-up design/API discussion. I also think it should be general enough to avoid coupling certain cloud storage types with their respective KMS offerings. (e.g. S3 storage doesn't necessarily imply AWS KMS usage). We have KeyManagementClient interface that users can implement for their own KMS needs. Large enterprise deployments may already use an internal KMS, I don’t think we should require them to implement both this interface and KMS credential vending before they can use table encryption with a REST catalog. Let me know what you all think. Cheers, Adam On 2026/05/06 04:59:23 Prashant Singh wrote: > Thanks everyone for chiming in on this, > > *Regarding the PR : * > The approach we are taking here where we pick encryption creds from the > client side (catalog level) and storage creds from server vended creds, is > fairly concerning to me. > if the REST server is giving one access because someone has *SELECT* > grant, *SERVER* should mint them everything that required for the client to > successfully read the table, sending storage creds but not KMS creds > feels incorrect to me, how is the server gonna know the client has creds > for *decryption* for this already ? This is entirely different from what > other catalogs expect i.e entirely a client side configuration for example > Hive Catalog / Glue catalog etc > > Regarding backward compatibility, if we do this, server vending KMS > credentials eventually, these clients will simply not work since they > always expect the credentials to come from the client side (Note : I am > strictly talking of table level) > > *Regarding the Spec Change : * > It makes sense to me as well, to decouple KMS & storage creds ! > > All in all I believe we should conclude this discussion and not rush into > things (we can make incremental progress for sure), if we have open > questions and an awaiting spec change. > > Best, > Prashant Singh > > On Tue, May 5, 2026 at 12:11 AM Gidon Gershinsky wrote: > > > Hi all, > > > > The built-in encryption is designed to protect tables in any storage > > backend, including untrusted storage. So I also think the storage > > credentials and KMS credentials are best treated separately, as they don't > > have the same purpose/scope. > > > > Regarding the current PR (13225) - it doesn't handle credentials directly; > > instead, it focuses on building the scaffolding for table encryption in the > > REST catalog client - similarly to the Hive catalog client (and other > > catalogs in the future). > > > > Cheers, Gidon > > > > > > On Tue, May 5, 2026 at 3:40 AM Chris Lu wrote: > > > >> Hi Prashant, > >> > >> My own preference would be to keep storage credentials and encryption/KMS > >> credentials separate in the REST contract. > >> > >> Storage credentials and KMS/Vault credentials have different scopes, > >> lifetimes, providers, and failure modes. > >> > >> > >>- Storage credentials authorize access to object locations. > >>- > >> > >>KMS/Vault credentials authorize key wrap/unwrap operations. > >> > >> Object-store SSE-KMS is a storage-layer feature, but Iceberg table > >> encryption is not necessarily tied to the object store. Vault or > >> external KMS support would not naturally fit inside a storage credential > >> object. So mixing KMS credentials into the storage credentials object > >> may work for some S3/SSE cases, but it makes the REST model harder to > >> extend. > >> > >> A cleaner model may be to have separate sections for storage credentials > >> and encryption credentials. Another option is a generic typed credential > >> model, where each credential has a type such as storage or encryption, and > >> a provider such as S3, Vault, or a cloud KMS. > >> > >> The typed model may be easier to extend later if we expect more > >> credential types. > >> > >> For backward compatibility, I think the current PR can still move > >> forward, but the client should not assume that storage credentials are the > >> permanent place for KMS credentials. If we need an interim implementation, > >> maybe it should be documented as catalog-level KMS configuration support, > >> not table-level encryption credential vending. > >> > >> I also
Re: [DISCUSS] REST Catalog: Credential management for KMS/Vault and Table level Encryption
Thanks everyone for chiming in on this, *Regarding the PR : * The approach we are taking here where we pick encryption creds from the client side (catalog level) and storage creds from server vended creds, is fairly concerning to me. if the REST server is giving one access because someone has *SELECT* grant, *SERVER* should mint them everything that required for the client to successfully read the table, sending storage creds but not KMS creds feels incorrect to me, how is the server gonna know the client has creds for *decryption* for this already ? This is entirely different from what other catalogs expect i.e entirely a client side configuration for example Hive Catalog / Glue catalog etc Regarding backward compatibility, if we do this, server vending KMS credentials eventually, these clients will simply not work since they always expect the credentials to come from the client side (Note : I am strictly talking of table level) *Regarding the Spec Change : * It makes sense to me as well, to decouple KMS & storage creds ! All in all I believe we should conclude this discussion and not rush into things (we can make incremental progress for sure), if we have open questions and an awaiting spec change. Best, Prashant Singh On Tue, May 5, 2026 at 12:11 AM Gidon Gershinsky wrote: > Hi all, > > The built-in encryption is designed to protect tables in any storage > backend, including untrusted storage. So I also think the storage > credentials and KMS credentials are best treated separately, as they don't > have the same purpose/scope. > > Regarding the current PR (13225) - it doesn't handle credentials directly; > instead, it focuses on building the scaffolding for table encryption in the > REST catalog client - similarly to the Hive catalog client (and other > catalogs in the future). > > Cheers, Gidon > > > On Tue, May 5, 2026 at 3:40 AM Chris Lu wrote: > >> Hi Prashant, >> >> My own preference would be to keep storage credentials and encryption/KMS >> credentials separate in the REST contract. >> >> Storage credentials and KMS/Vault credentials have different scopes, >> lifetimes, providers, and failure modes. >> >> >>- Storage credentials authorize access to object locations. >>- >> >>KMS/Vault credentials authorize key wrap/unwrap operations. >> >> Object-store SSE-KMS is a storage-layer feature, but Iceberg table >> encryption is not necessarily tied to the object store. Vault or >> external KMS support would not naturally fit inside a storage credential >> object. So mixing KMS credentials into the storage credentials object >> may work for some S3/SSE cases, but it makes the REST model harder to >> extend. >> >> A cleaner model may be to have separate sections for storage credentials >> and encryption credentials. Another option is a generic typed credential >> model, where each credential has a type such as storage or encryption, and >> a provider such as S3, Vault, or a cloud KMS. >> >> The typed model may be easier to extend later if we expect more >> credential types. >> >> For backward compatibility, I think the current PR can still move >> forward, but the client should not assume that storage credentials are the >> permanent place for KMS credentials. If we need an interim implementation, >> maybe it should be documented as catalog-level KMS configuration support, >> not table-level encryption credential vending. >> >> I also think the catalog should have a clear capability or failure >> signal. If a table requires Iceberg encryption and the catalog supports >> vended credentials, then the catalog should either return the required >> encryption credentials or fail early during table load or credential >> refresh with a clear error. Failing later during file IO or key unwrap >> would be much harder to debug. >> >> So my suggested direction would be: >> >> >>- Keep object-store SSE credentials/config separate from Iceberg >>table encryption credentials. >>- >> >>Add a dedicated encryption credential model, or a generic typed >>credential model. >>- >> >>Add a capability or requirement signal so clients can fail early when >>required credentials are missing. >> >> Avoid baking in the assumption that KMS credentials always live under >> storage credentials. This should leave room for catalogs that vend >> table-scoped KMS/Vault credentials in the future, instead of only >> supporting catalog-level KMS client configuration. >> >> Thanks, >> Chris >> > > On Tue, May 5, 2026 at 3:22 AM Sreesh Maheshwar > wrote: > >> Hey Prashant, >> >> Thank you for kicking off this discussion! Some thoughts: >> >> > The current PR advocates for including KMS credentials within the >> storage credentials object. >> >> This isn't true - the KMS client is currently configured with just the >> catalog config client-side (though that includes the server's `getConfig` >> response) [1]; there is no per-table vending of KMS credentials as such. >> >> > Separation of KM
Re: [DISCUSS] REST Catalog: Credential management for KMS/Vault and Table level Encryption
Hi all, The built-in encryption is designed to protect tables in any storage backend, including untrusted storage. So I also think the storage credentials and KMS credentials are best treated separately, as they don't have the same purpose/scope. Regarding the current PR (13225) - it doesn't handle credentials directly; instead, it focuses on building the scaffolding for table encryption in the REST catalog client - similarly to the Hive catalog client (and other catalogs in the future). Cheers, Gidon On Tue, May 5, 2026 at 3:40 AM Chris Lu wrote: > Hi Prashant, > > My own preference would be to keep storage credentials and encryption/KMS > credentials separate in the REST contract. > > Storage credentials and KMS/Vault credentials have different scopes, > lifetimes, providers, and failure modes. > > >- Storage credentials authorize access to object locations. >- > >KMS/Vault credentials authorize key wrap/unwrap operations. > > Object-store SSE-KMS is a storage-layer feature, but Iceberg table > encryption is not necessarily tied to the object store. Vault or external > KMS support would not naturally fit inside a storage credential object. So > mixing KMS credentials into the storage credentials object may work for > some S3/SSE cases, but it makes the REST model harder to extend. > > A cleaner model may be to have separate sections for storage credentials > and encryption credentials. Another option is a generic typed credential > model, where each credential has a type such as storage or encryption, and > a provider such as S3, Vault, or a cloud KMS. > > The typed model may be easier to extend later if we expect more credential > types. > > For backward compatibility, I think the current PR can still move forward, > but the client should not assume that storage credentials are the permanent > place for KMS credentials. If we need an interim implementation, maybe it > should be documented as catalog-level KMS configuration support, not > table-level encryption credential vending. > > I also think the catalog should have a clear capability or failure signal. > If a table requires Iceberg encryption and the catalog supports vended > credentials, then the catalog should either return the required encryption > credentials or fail early during table load or credential refresh with a > clear error. Failing later during file IO or key unwrap would be much > harder to debug. > > So my suggested direction would be: > > >- Keep object-store SSE credentials/config separate from Iceberg table >encryption credentials. >- > >Add a dedicated encryption credential model, or a generic typed >credential model. >- > >Add a capability or requirement signal so clients can fail early when >required credentials are missing. > > Avoid baking in the assumption that KMS credentials always live under > storage credentials. This should leave room for catalogs that vend > table-scoped KMS/Vault credentials in the future, instead of only > supporting catalog-level KMS client configuration. > > Thanks, > Chris > On Tue, May 5, 2026 at 3:22 AM Sreesh Maheshwar wrote: > Hey Prashant, > > Thank you for kicking off this discussion! Some thoughts: > > > The current PR advocates for including KMS credentials within the > storage credentials object. > > This isn't true - the KMS client is currently configured with just the > catalog config client-side (though that includes the server's `getConfig` > response) [1]; there is no per-table vending of KMS credentials as such. > > > Separation of KMS/Vault Creds from Storage Creds > > I agree, if/when it supports this, the AWS KMS client could choose to > introduce `kms.access-key-id` and co. instead of reusing `s3.access-key-id` > and co. (similar to how we have `rest.access-key-id` and co. [2]) to > differentiate from storage credentials. And similarly for vending: if we'd > like to support the vending of KMS credentials [3], we can discuss its > format to enable separation. > > (Specifically on the current PR [4], I don't think that the concerns > described affect it due to my first point above. I think > catalog-initialised KMS clients, as implemented, are useful to support now > and in the future.) > > Thanks, > Sreesh Maheshwar > > [1] > https://github.com/apache/iceberg/pull/13225/files#diff-86450612dbe323d6d06cbc3846aa1913f042eaedadc0ca027c36bfbe08d3a46cR284 > [2] > https://github.com/apache/iceberg/blob/2d54125734ddc9b9fb87db147ff255918108fa2c/aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java#L193 > [3] https://github.com/apache/iceberg/issues/16194 > [4] https://github.com/apache/iceberg/pull/13225 > > On Monday, May 4th, 2026 at 9:03 AM, Prashant Singh < > [email protected]> wrote: > > Hi everyone, > > I’d like to start a discussion regarding how we handle credentials for > encryption (like KMS or Vault (recently being discussed)) in the REST > catalog. > > As we know, unlike other catalogs, the REST catalog mints
Re: [DISCUSS] REST Catalog: Credential management for KMS/Vault and Table level Encryption
Hi Prashant, My own preference would be to keep storage credentials and encryption/KMS credentials separate in the REST contract. Storage credentials and KMS/Vault credentials have different scopes, lifetimes, providers, and failure modes. - Storage credentials authorize access to object locations. - KMS/Vault credentials authorize key wrap/unwrap operations. Object-store SSE-KMS is a storage-layer feature, but Iceberg table encryption is not necessarily tied to the object store. Vault or external KMS support would not naturally fit inside a storage credential object. So mixing KMS credentials into the storage credentials object may work for some S3/SSE cases, but it makes the REST model harder to extend. A cleaner model may be to have separate sections for storage credentials and encryption credentials. Another option is a generic typed credential model, where each credential has a type such as storage or encryption, and a provider such as S3, Vault, or a cloud KMS. The typed model may be easier to extend later if we expect more credential types. For backward compatibility, I think the current PR can still move forward, but the client should not assume that storage credentials are the permanent place for KMS credentials. If we need an interim implementation, maybe it should be documented as catalog-level KMS configuration support, not table-level encryption credential vending. I also think the catalog should have a clear capability or failure signal. If a table requires Iceberg encryption and the catalog supports vended credentials, then the catalog should either return the required encryption credentials or fail early during table load or credential refresh with a clear error. Failing later during file IO or key unwrap would be much harder to debug. So my suggested direction would be: - Keep object-store SSE credentials/config separate from Iceberg table encryption credentials. - Add a dedicated encryption credential model, or a generic typed credential model. - Add a capability or requirement signal so clients can fail early when required credentials are missing. Avoid baking in the assumption that KMS credentials always live under storage credentials. This should leave room for catalogs that vend table-scoped KMS/Vault credentials in the future, instead of only supporting catalog-level KMS client configuration. Thanks, Chris On Monday, May 4th, 2026 at 9:03 AM, Prashant Singh wrote: > Hi everyone, > > I’d like to start a discussion regarding how we handle credentials for > encryption (like KMS or Vault (recently being discussed)) in the REST catalog. > > As we know, unlike other catalogs, the REST catalog mints credentials at the > table level for the client to use in subsequent operations, and we already > have the dedicated /credentials endpoint in place to handle refreshing these. > > While reviewing the recent encryption PR, a few architectural concerns came > up that I believe we need to conclude on before we mark the REST catalog as > "ready" for supporting encryption: > > - > > Separation of KMS/Vault Creds from Storage Creds: How should we handle > external key managers like Vault? The current PR [1] advocates for including > KMS credentials within the storage credentials object. If we end up > supporting Vault, this can't be mixed with storage cred. (Side note: catalogs > have historically mixed KMS creds with this object for things like SSE, but > that is entirely an object-store-level concept). We need a clear path forward > for how REST will return per-table credentials specifically for Vault/KMS > stores. > > - > > Catalog Awareness & Client-Side Assertions: If the catalog returns > credentials, it overrides the client-side credentials. This means a naive > catalog that is unaware of encryption and just treats metadata as-is will > have no clue it needs to vend these specific KMS credentials and if it > forgerts to do that there is no way for the client to know this (for object > store cases) except to fail during runtime ? Should the catalog fail such > requests as part of their contract of supporting v3 (we don't need this in > spec), should the catalog send some signal, hey i send you creds for > encryption too, and if the client doesn't find it fails early ? > > - > > Backward Compatibility Risks: If we release the client now with the > expectation that "storage credentials" will always contain KMS credentials. > If we later introduce a dedicated field for encryption credentials in the > loadTable response, we will be forced to maintain backward compatibility to > support both ways of returning credentials. > > To be clear, I do not want to block the progress on the current PR. I really > appreciate all the hard work that has gone into it! However, I think it is > crucial that we align on these design points for the REST catalog's > encryption architecture before finalizing it. > > I would appreciate any thoughts or
Re: [DISCUSS] REST Catalog: Credential management for KMS/Vault and Table level Encryption
Hey Prashant, Thank you for kicking off this discussion! Some thoughts: > The current PR advocates for including KMS credentials within the storage credentials object. This isn't true - the KMS client is currently configured with just the catalog config client-side (though that includes the server's `getConfig` response) [1]; there is no per-table vending of KMS credentials as such. > Separation of KMS/Vault Creds from Storage Creds I agree, if/when it supports this, the AWS KMS client could choose to introduce `kms.access-key-id` and co. instead of reusing `s3.access-key-id` and co. (similar to how we have `rest.access-key-id` and co. [2]) to differentiate from storage credentials. And similarly for vending: if we'd like to support the vending of KMS credentials [3], we can discuss its format to enable separation. (Specifically on the current PR [4], I don't think that the concerns described affect it due to my first point above. I think catalog-initialised KMS clients, as implemented, are useful to support now and in the future.) Thanks, Sreesh Maheshwar [1] https://github.com/apache/iceberg/pull/13225/files#diff-86450612dbe323d6d06cbc3846aa1913f042eaedadc0ca027c36bfbe08d3a46cR284 [2] https://github.com/apache/iceberg/blob/2d54125734ddc9b9fb87db147ff255918108fa2c/aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java#L193 [3] https://github.com/apache/iceberg/issues/16194 [4] https://github.com/apache/iceberg/pull/13225 >
