ggershinsky commented on pull request #2638: URL: https://github.com/apache/iceberg/pull/2638#issuecomment-849378241
Sure, TBD, but we need to sync up first on this point > there are enough information provided in key_metadata. which (in my view) seems to be a major gap here - `key_metadata` is a field in manifest files, which keeps the wrapped encryption keys for each data file. The keys (DEKs) are either generated randomly by Iceberg, or fetched from a KMS. The key wrapping is always done via KMS calls. Key_metadata doesn't store the keys themselves, since this would make them exposed in manifest files. Therefore, passing `key_metadata` to the data file writers doesn't help, because these writers need the data encryption keys. Calling KMS to unwrap them doesn't make sense in this particular point... Therefore, we need a separate mechanism to deliver the DEK(s) and other parameters to the writers. The classes in this PR list the required parameters. We have a similar issue on the reader side, but lets get to it after syncing on the writer path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
