[
https://issues.apache.org/jira/browse/HADOOP-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045560#comment-14045560
]
Alejandro Abdelnur commented on HADOOP-10720:
---------------------------------------------
further details on how we could optimizing the extension operations in the KMS
(server/client) to reduce/eliminate calls from the NN on file create():
The {{KMSClientProvider}} would implement the {{CryptoExtensions}} interface,
thus the {{KeyProviderExtensions}} would delegate to it for generating and
decrypting EDEKs.
The KMS client will have a queue with IV&EDEK per key. If the client queue is
new/empty a call to the KMS asking for N IV&EDEKs for the key will be send, the
KMS response will have at least one IV&EDEK. the returned IV&EDEKs are put int
he queue and the first entry from the queue is returned to the KMS client
caller. The KMS client will have a daemon thread that keeps requesting IV&EDEKs
to the KMS for a key using a low and a high marks to start and stop requesting
more IV&EDEKs to the KMS. This means that most of the time the KMS client will
have IV&EDEKs locally cached for the NN to use.
On the KMS side a the same queue and replenisher daemon thread will have
another cache of IV&EDEK. On KMS client requests, typically, IV&EDEKS will not
be generated synchronously but simply drained from the queue. if the queue is
new/empty a small number of IV&EDEKs will be generated synchronously and
returned to the caller. if the queue is not empty and the client request N but
the queue has less than N, the available elements will be returned and the
daemon thread will start producing more up to the high mark.
This dual cache will significantly reduce NN->KMS calls and reduce/eliminate
the calls that happen synchronously with NN create() call.
Suggestion: look at the JDK ThreadPoolExecutor uses a similar logic using
wait/notify thus not sleeping/looping-around.
> KMS: Implement generateEncryptedKey and decryptEncryptedKey in the REST API
> ---------------------------------------------------------------------------
>
> Key: HADOOP-10720
> URL: https://issues.apache.org/jira/browse/HADOOP-10720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
> Affects Versions: 3.0.0
> Reporter: Alejandro Abdelnur
> Assignee: Arun Suresh
> Attachments: COMBO.patch, COMBO.patch, COMBO.patch, COMBO.patch,
> COMBO.patch, HADOOP-10720.patch, HADOOP-10720.patch, HADOOP-10720.patch,
> HADOOP-10720.patch, HADOOP-10720.patch
>
>
> KMS client/server should implement support for generating encrypted keys and
> decrypting them via the REST API being introduced by HADOOP-10719.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
