[
https://issues.apache.org/jira/browse/HADOOP-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084145#comment-16084145
]
Rushabh S Shah commented on HADOOP-14563:
-----------------------------------------
{quote} When NN is generating the next edek, it has 1/3 chance to fetch one
from local cache, and 2/3 chance to make a call to KMS.
Is this acceptable?
{quote}
Before this fix, even if warmup failed on all the 3 providers, it didn't throw
any Exception and the probability of making a synchronous call( on
{{genrateEdek}}) to KMS was 100%.
Atleast after the fix, if all the 3 provider's {{warmUpEncryptedKeys}} fail, it
will fail to create encryption zone.
As a part of HDFS-12124 or maybe a separate jira, I am thinking of handling the
{{generatingEdek}} case.
If one provider return null, it will try all the providers before giving up and
throwing RetryStartFile exception to dfs clients.
> LoadBalancingKMSClientProvider#warmUpEncryptedKeys swallows IOException
> -----------------------------------------------------------------------
>
> Key: HADOOP-14563
> URL: https://issues.apache.org/jira/browse/HADOOP-14563
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.8.1
> Reporter: Rushabh S Shah
> Assignee: Rushabh S Shah
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: HADOOP-14563-1.patch, HADOOP-14563-2.patch,
> HADOOP-14563.patch
>
>
> TestAclsEndToEnd is failing consistently in HADOOP-14521.
> The reason behind it is LoadBalancingKMSClientProvider#warmUpEncryptedKeys
> swallows IOException while KMSClientProvider#warmUpEncryptedKeys throws all
> the way back to createEncryptionZone and creation of EZ fails.
> Following are the relevant piece of code snippets.
> {code:title=KMSClientProvider.java|borderStyle=solid}
> @Override
> public void warmUpEncryptedKeys(String... keyNames)
> throws IOException {
> try {
> encKeyVersionQueue.initializeQueuesForKeys(keyNames);
> } catch (ExecutionException e) {
> throw new IOException(e);
> }
> }
> {code}
> {code:title=LoadBalancingKMSClientProvider.java|borderStyle=solid}
> // This request is sent to all providers in the load-balancing group
> @Override
> public void warmUpEncryptedKeys(String... keyNames) throws IOException {
> for (KMSClientProvider provider : providers) {
> try {
> provider.warmUpEncryptedKeys(keyNames);
> } catch (IOException ioe) {
> LOG.error(
> "Error warming up keys for provider with url"
> + "[" + provider.getKMSUrl() + "]", ioe);
> }
> }
> }
> {code}
> In HADOOP-14521, I intend to always instantiate
> LoadBalancingKMSClientProvider even if there is only one provider so that the
> retries can applied at only one place.
> We need to decide whether we want to fail in both the case or continue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
