Mohammadhani Fouladgar created HDDS-17762:
---------------------------------------------
Summary: DataNode fails to register with a dynamically-added SCM
on a Kerberos-enabled cluste
Key: HDDS-17762
URL: https://issues.apache.org/jira/browse/HDDS-17762
Project: Apache Ozone
Issue Type: New Feature
Components: Ozone Datanode, SCM HA
Environment: Kerberos ({{ozone.security.enabled=true}},
{{hadoop.security.authentication=kerberos}}). Does not reproduce with security
disabled.
Reporter: Mohammadhani Fouladgar
HDDS-13890 lets a running DataNode pick up a newly added SCM by reconfiguring
{{ozone.scm.nodes.<serviceId>}} (via {{ozone admin reconfig
--service=DATANODE}}) instead of requiring a restart. On a secure cluster this
does not work: after the reconfig, the DataNode never registers with the newly
added SCM(s), which therefore stay in SCM safemode ("No Datanodes registered
with SCM yet"). Connections to the pre-existing SCM(s) are unaffected.
Restarting the DataNode resolves it (it then registers with all SCMs)
Steps to reproduce
1. Secure (Kerberized) Ozone cluster with SCM HA, 1 SCM initially, DataNodes
registered and out of safemode.
2. Add new SCM(s) and bootstrap them into the Ratis ring (they appear in
{{ozone admin scm roles}}).
3. Deliver the updated {{ozone.scm.nodes.<svc>}} +
{{ozone.scm.address.<svc>.<nodeId>}} to DataNodes and run {{ozone admin
reconfig --service=DATANODE <dn> start}}. The reconfig reports success.
4. Observe the new SCM(s) stay in safemode; DataNode log shows repeated auth
failures to the new SCM(s).
Actual behavior — DataNode log, repeating against the new SCM only:
WARN ...EndpointStateMachineTaskThread-<newscm>:9861... ipc.Client:
Exception encountered while connecting to the server <newscm>/<ip>:9861
javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
at ...SaslRpcClient.saslConnect(...)
at
...ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.getVersion(...)
at ...container.common.states.endpoint.VersionEndpointTask.call(...)
Connections to the original SCM succeed; only the dynamically-added endpoint
fails.
Expected behavior — the DataNode authenticates to the newly added SCM using its
service (keytab) identity and registers, exactly as it does for SCMs present at
startup; the new SCM leaves safemode without a DataNode restart.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]