Mohammadhani Fouladgar created HDDS-17762:
---------------------------------------------

             Summary: DataNode fails to register with a dynamically-added SCM 
on a Kerberos-enabled cluste
                 Key: HDDS-17762
                 URL: https://issues.apache.org/jira/browse/HDDS-17762
             Project: Apache Ozone
          Issue Type: New Feature
          Components: Ozone Datanode, SCM HA
         Environment: Kerberos ({{ozone.security.enabled=true}}, 
{{hadoop.security.authentication=kerberos}}). Does not reproduce with security 
disabled.
            Reporter: Mohammadhani Fouladgar


HDDS-13890 lets a running DataNode pick up a newly added SCM by reconfiguring 
{{ozone.scm.nodes.<serviceId>}} (via {{ozone admin reconfig 
--service=DATANODE}}) instead of requiring a restart. On a secure cluster this 
does not work: after the reconfig, the DataNode never registers with the newly 
added SCM(s), which therefore stay in SCM safemode ("No Datanodes registered 
with SCM yet"). Connections to the pre-existing SCM(s) are unaffected. 
Restarting the DataNode resolves it (it then registers with all SCMs)

Steps to reproduce
1. Secure (Kerberized) Ozone cluster with SCM HA, 1 SCM initially, DataNodes 
registered and out of safemode.
2. Add new SCM(s) and bootstrap them into the Ratis ring (they appear in 
{{ozone admin scm roles}}).
3. Deliver the updated {{ozone.scm.nodes.<svc>}} + 
{{ozone.scm.address.<svc>.<nodeId>}} to DataNodes and run {{ozone admin 
reconfig --service=DATANODE <dn> start}}. The reconfig reports success. 
4. Observe the new SCM(s) stay in safemode; DataNode log shows repeated auth 
failures to the new SCM(s).

Actual behavior — DataNode log, repeating against the new SCM only:
 WARN  ...EndpointStateMachineTaskThread-<newscm>:9861... ipc.Client:
     Exception encountered while connecting to the server <newscm>/<ip>:9861
   javax.security.sasl.SaslException: GSS initiate failed
     [Caused by GSSException: No valid credentials provided
      (Mechanism level: Failed to find any Kerberos tgt)]
       at ...SaslRpcClient.saslConnect(...)
       at 
...ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.getVersion(...)
       at ...container.common.states.endpoint.VersionEndpointTask.call(...)

Connections to the original SCM succeed; only the dynamically-added endpoint 
fails.

Expected behavior — the DataNode authenticates to the newly added SCM using its 
service (keytab) identity and registers, exactly as it does for SCMs present at 
startup; the new SCM leaves safemode without a DataNode restart.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to