ggershinsky commented on pull request #3470: URL: https://github.com/apache/iceberg/pull/3470#issuecomment-965087037
> I am personally more inclined right now with having a separated KmsClient interface, mostly because this is the service integration point with different KMS services on the market, they don't need to know any logic related to actual Iceberg encryption, this separated interface seems cleaner to me. I would expect only very advanced users to provide an entire custom `EncryptionManager` implementation. I also think this is a simpler and cleaner approach for organizations and their users. One developer/ team would implement the KmsClient class for their organization, integrating with the local KMS and IAM services (the KmsClient interface should be able to support any). Once the implemented class is deployed, any authorized user in the org can write encrypted tables, passing only one parameter - a string with the table key name (later, we'll add column keys). The readers don't need to pass anything. Both writers and readers use auth credentials (KMS-specific). Last but not least, this approach is based on the standard "envelope encryption" practice. > during the discussion I remember Ryan was for having these key generation logic directly in the manager itself as a `BaseKmsEncryptionManager`. I think adding that part of the logic here can facilitate the discussion. One area where we might need to expand the basic approach, described above, is custom encrypted file streaming. By default, Iceberg will use its built-in encryption for the tables (PME/ORC for data; GcmStream for metadata, and potentially for Avro data; we can also add a built-in AwsS3ClientStream in Iceberg). But if an organization prefers to use its custom encryptors instead of those built in Iceberg, we can load them. I'm sending a new commit to this PR, that adds an interface currently called BaseKmsEncryptionManager. The name and content are a subject for discussion. > I think it's beneficial to also include code about how this class will be used in `EncryptionManager`. We have an internal implementation of EncryptionManager, a class called EnvelopeEncryptionManager that uses a KmsClient plug-in, and leverages the "envelope encryption" practice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
