[
https://issues.apache.org/jira/browse/HADOOP-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102240#comment-16102240
]
Xiao Chen commented on HADOOP-14688:
------------------------------------
Thanks again [~daryn] for the review series! :)
True the edeks are stored in a xattr. During re-encryption though, the EDEK
object is constructed, and sent to KMS, where a new EDEK is returned.
What's tricky here is, contacting KMS requires to be done outside of the lock.
Therefore, the EDEK object has to exist for that time being. And since we're
trying to re-encrypt many EDEKs per batch, there're many on-the-fly EDEK
objects. Relative code in {{ReencryptionHandler$EDEKReencryptCallable#call}} of
HDFS-10899.
To make things worse, since KMS is proven to be the bottleneck of this, we'd
like to multi thread the 'contact KMS' part, which means more on-the-fly
EDEKs.... (see multi-threading part of HDFS-10899's
[doc|https://issues.apache.org/jira/secure/attachment/12874358/Re-encrypt%20edek%20design%20doc%20V2.pdf])
> Intern strings in KeyVersion and EncryptedKeyVersion
> ----------------------------------------------------
>
> Key: HADOOP-14688
> URL: https://issues.apache.org/jira/browse/HADOOP-14688
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
> Reporter: Xiao Chen
> Assignee: Xiao Chen
> Attachments: HADOOP-14688.01.patch
>
>
> This is inspired by [[email protected]]'s work on HDFS-11383.
> The key names and key version names are usually the same for a bunch of
> {{KeyVersion}} and {{EncryptedKeyVersion}}. We should not create duplicate
> objects for them.
> This is more important to HDFS-10899, where we try to re-encrypt all files'
> EDEKs in a given EZ. Those EDEKs all has the same key name, and mostly using
> no more than a couple of key version names.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
