[
https://issues.apache.org/jira/browse/HADOOP-16199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795315#comment-16795315
]
Xiaoyu Yao edited comment on HADOOP-16199 at 3/18/19 7:08 PM:
--------------------------------------------------------------
As can be seen below, a kms-dt with service field *Service:
kms://[[email protected]|mailto:[email protected]];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms]*
can't be selected for LoadBalancingKMSClientProvider because it does not match
its *canonical service: 172.25.36.130:9292*. Subsequent matching with
individual KMSClientProvider also failed in this case.
The comments on reason of hard code canonical service of
LoadBalancingKMSClientProvider to the ip:port of the first KMSClientProvider
instance can be improved.
The proposed fix it to allow
LoadBalancingKMSClientProvider#selectDelegationToken to match not only the
canonical service but also the delegation token service, which is similar to
what we have done in KMSClientProvider#selectDelegationToken.
Below is detailed failure log for reference:
{code:java}
2019-03-13 18:51:33,056 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getServerDefaults took 5ms
2019-03-13 18:51:33,086 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created
for KMS url: [http://c316-node3.raghav.com:9292/kms/v1/] delegation token
service:
[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical
service: 172.25.36.130:9292.
2019-03-13 18:51:33,087 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created
for KMS url: [http://c316-node4.raghav.com:9292/kms/v1/] delegation token
service:
[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical
service: 172.25.38.4:9292.
2019-03-13 18:51:33,089 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: Created
LoadBalancingKMSClientProvider for KMS url:
kms://[[email protected]|mailto:[email protected]];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms]
with 2 providers. delegation token service:
kms://[[email protected]|mailto:[email protected]];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms],
canonical service: 172.25.36.130:9292
...
2019-03-13 18:51:33,112 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: Current UGI: hr1
(auth:SIMPLE)
2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: Localizer,
Service: , Ident:
(org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.security.LocalizerTokenIdentifier@54604a95)
*2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: kms-dt,
Service:
kms://[[email protected]|mailto:[email protected]];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms],
Ident: (kms-dt owner=hr1, renewer=yarn, realUser=oozie,
issueDate=1552503090542, maxDate=1553107890542, sequenceNumber=27,
masterKeyId=30)*
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind:
HDFS_DELEGATION_TOKEN, Service: 172.25.35.133:8020, Ident: (token for hr1:
HDFS_DELEGATION_TOKEN owner=hr1, renewer=yarn,
realUser=oozie/c316-node1.[[email protected]|mailto:[email protected]],
issueDate=1552503090263, maxDate=1553107890263, sequenceNumber=443,
masterKeyId=93)
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.token.Token: Cannot find class for token kind
HIVE_DELEGATION_TOKEN
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind:
HIVE_DELEGATION_TOKEN, Service: hiveserver2ClientToken, Ident: 00 03 68 72 31
04 68 69 76 65 05 6f 6f 7a 69 65 8a 01 69 78 65 2b a8 8a 01 69 9c 71 af a8 03
8f 84
2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind:
RM_DELEGATION_TOKEN, Service: 172.25.35.133:8050, Ident: (RM_DELEGATION_TOKEN
owner=hr1, renewer=yarn,
realUser=oozie/c316-node1.[[email protected]|mailto:[email protected]],
issueDate=1552503090238, maxDate=1553107890238, sequenceNumber=21,
masterKeyId=139)
2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind:
MR_DELEGATION_TOKEN, Service: 172.25.35.133:10020, Ident: (MR_DELEGATION_TOKEN
owner=hr1, renewer=yarn,
realUser=oozie/c316-node1.[[email protected]|mailto:[email protected]],
issueDate=1552503090488, maxDate=1553107890488, sequenceNumber=5,
masterKeyId=107)
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: Login UGI: hr1 (auth:SIMPLE)
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: Searching for KMS
delegation token in user hr1 (auth:SIMPLE)'s credentials
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=172.25.36.130:9292 token=null
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=172.25.36.130:9292 token=null
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=172.25.38.4:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=172.25.38.4:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=172.25.36.130:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=172.25.36.130:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when
Kerberos is enabled but the actual user does not have either KMS Delegation
Token or Kerberos Credentials
2019-03-13 18:51:33,146 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hr1
(auth:SIMPLE)
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:506)
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL:
Connecting to url
[http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]with
token as null
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL:
Token not set, looking for delegation token. Creds:[], size:0
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator:
No delegation token found for
url=[http://c316-node4|http://c316-node4/].[raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt|http://raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt],
token=, authenticating with class
org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticator$1
2019-03-13 18:51:33,178 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.authentication.client.KerberosAuthenticator:
Performing our own SPNEGO sequence.
2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.authentication.client.KerberosAuthenticator: No
subject in context, logging in
2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Using
subject: Subject:
Principal: UnixPrincipal: hr1
Principal: UnixNumericUserPrincipal: 1016
Principal: UnixNumericGroupPrincipal [Primary Group]: 1016
2019-03-13 18:51:33,182 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException
as:hr1 (auth:SIMPLE)
cause:org.apache.hadoop.security.authentication.client.AuthenticationException:
Error while authenticating with endpoint:
[http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]
2019-03-13 18:51:33,182 WARN [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: KMS provider
at [[http://c316-node4.raghav.com:9292/kms/v1/]] threw an IOException:
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException: Error
while authenticating with endpoint:
[http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]
{code}
was (Author: xyao):
As can be seen below, a kms-dt with service field *Service:
kms://[[email protected]|mailto:[email protected]];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms]*
can't be selected for LoadBalancingKMSClientProvider because it does not match
its *canonical service: 172.25.36.130:9292*. Subsequent matching with
individual KMSClientProvider also failed in this case. The proposed fix it to
allow LoadBalancingKMSClientProvider#selectDelegationToken to match not only
the canonical service but also the delegation token service.
Also, the comments on reason of hard code canonical service of
LoadBalancingKMSClientProvider to the ip:port of the first KMSClientProvider
instance can be improved.
Below is detailed failure log for reference:
{code:java}
2019-03-13 18:51:33,056 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getServerDefaults took 5ms
2019-03-13 18:51:33,086 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created
for KMS url: [http://c316-node3.raghav.com:9292/kms/v1/] delegation token
service:
[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical
service: 172.25.36.130:9292.
2019-03-13 18:51:33,087 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created
for KMS url: [http://c316-node4.raghav.com:9292/kms/v1/] delegation token
service:
[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical
service: 172.25.38.4:9292.
2019-03-13 18:51:33,089 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: Created
LoadBalancingKMSClientProvider for KMS url:
kms://[[email protected]|mailto:[email protected]];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms]
with 2 providers. delegation token service:
kms://[[email protected]|mailto:[email protected]];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms],
canonical service: 172.25.36.130:9292
...
2019-03-13 18:51:33,112 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: Current UGI: hr1
(auth:SIMPLE)
2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: Localizer,
Service: , Ident:
(org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.security.LocalizerTokenIdentifier@54604a95)
*2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: kms-dt,
Service:
kms://[[email protected]|mailto:[email protected]];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms],
Ident: (kms-dt owner=hr1, renewer=yarn, realUser=oozie,
issueDate=1552503090542, maxDate=1553107890542, sequenceNumber=27,
masterKeyId=30)*
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind:
HDFS_DELEGATION_TOKEN, Service: 172.25.35.133:8020, Ident: (token for hr1:
HDFS_DELEGATION_TOKEN owner=hr1, renewer=yarn,
realUser=oozie/c316-node1.[[email protected]|mailto:[email protected]],
issueDate=1552503090263, maxDate=1553107890263, sequenceNumber=443,
masterKeyId=93)
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.token.Token: Cannot find class for token kind
HIVE_DELEGATION_TOKEN
2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind:
HIVE_DELEGATION_TOKEN, Service: hiveserver2ClientToken, Ident: 00 03 68 72 31
04 68 69 76 65 05 6f 6f 7a 69 65 8a 01 69 78 65 2b a8 8a 01 69 9c 71 af a8 03
8f 84
2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind:
RM_DELEGATION_TOKEN, Service: 172.25.35.133:8050, Ident: (RM_DELEGATION_TOKEN
owner=hr1, renewer=yarn,
realUser=oozie/c316-node1.[[email protected]|mailto:[email protected]],
issueDate=1552503090238, maxDate=1553107890238, sequenceNumber=21,
masterKeyId=139)
2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind:
MR_DELEGATION_TOKEN, Service: 172.25.35.133:10020, Ident: (MR_DELEGATION_TOKEN
owner=hr1, renewer=yarn,
realUser=oozie/c316-node1.[[email protected]|mailto:[email protected]],
issueDate=1552503090488, maxDate=1553107890488, sequenceNumber=5,
masterKeyId=107)
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: Login UGI: hr1 (auth:SIMPLE)
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: Searching for KMS
delegation token in user hr1 (auth:SIMPLE)'s credentials
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=172.25.36.130:9292 token=null
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=172.25.36.130:9292 token=null
2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=172.25.38.4:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=172.25.38.4:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
alias=172.25.36.130:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by
service=172.25.36.130:9292 token=null
2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when
Kerberos is enabled but the actual user does not have either KMS Delegation
Token or Kerberos Credentials
2019-03-13 18:51:33,146 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hr1
(auth:SIMPLE)
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:506)
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL:
Connecting to url
[http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]with
token as null
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL:
Token not set, looking for delegation token. Creds:[], size:0
2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator:
No delegation token found for
url=[http://c316-node4|http://c316-node4/].[raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt|http://raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt],
token=, authenticating with class
org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticator$1
2019-03-13 18:51:33,178 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.authentication.client.KerberosAuthenticator:
Performing our own SPNEGO sequence.
2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.authentication.client.KerberosAuthenticator: No
subject in context, logging in
2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Using
subject: Subject:
Principal: UnixPrincipal: hr1
Principal: UnixNumericUserPrincipal: 1016
Principal: UnixNumericGroupPrincipal [Primary Group]: 1016
2019-03-13 18:51:33,182 DEBUG [ContainerLocalizer Downloader]
org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException
as:hr1 (auth:SIMPLE)
cause:org.apache.hadoop.security.authentication.client.AuthenticationException:
Error while authenticating with endpoint:
[http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]
2019-03-13 18:51:33,182 WARN [ContainerLocalizer Downloader]
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: KMS provider
at [[http://c316-node4.raghav.com:9292/kms/v1/]] threw an IOException:
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException: Error
while authenticating with endpoint:
[http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]
{code}
> KMSLoadBlanceClientProvider does not select token correctly
> -----------------------------------------------------------
>
> Key: HADOOP-16199
> URL: https://issues.apache.org/jira/browse/HADOOP-16199
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.0.2
> Reporter: Xiaoyu Yao
> Assignee: Xiaoyu Yao
> Priority: Major
> Labels: kms
>
> After HADOOP-14445 and HADOOP-15997, there are still cases where
> KMSLoadBlanceClientProvider does not select token correctly.
> Here is the use case:
> The new configuration key
> hadoop.security.kms.client.token.use.uri.format=true is set cross all the
> cluster, including both Submitter and Yarn RM(renewer), which is not covered
> in the test matrix in this [HADOOP-14445
> comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16505761&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16505761].
> I will post the debug log and the proposed fix shortly, cc: [~xiaochen] and
> [~jojochuang].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
