[
https://issues.apache.org/jira/browse/HDDS-16300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammadhani Fouladgar updated HDDS-16300:
------------------------------------------
Description:
HDDS-13890 made DataNodes refresh their SCM connections without a restart, but
it only touched the DataNode heartbeat/registration path
({{{}SCMConnectionManager{}}} + {{{}StateContext{}}} endpoints). The OM reaches
SCM through a different mechanism: the {{SCMFailoverProxyProvider}} family,
which loads the SCM list once at startup via
{{SCMNodeInfo.buildNodeInfo(conf)}} ({{SCMFailoverProxyProviderBase}}) and
never reloads. The OM does not register {{ozone.scm.nodes}} as reconfigurable,
and the proxy providers have no reload hook.
Consequence: when a new SCM is added, the OM does not learn about it. While the
new SCM stays a follower this is harmless -- the OM still reaches the leader
via the existing configured SCMs. But once leadership moves to the new SCM, the
OM cannot route to a leader that is not in its static config and requests fail
(the same failover-to-unconfigured-leader outage as the leadership-transfer
scenario). So the OM must eventually learn the new SCM -- and the only
alternative today is an OM restart, which violates the restart-free requirement.
*Proposed change:*
* Register {{ozone.scm.nodes.<svc>}} (and the address keys) as reconfigurable
on the OM, implemented in the {{SCMFailoverProxyProviderBase}} reload path.
* On reconfiguration, reload the SCM list from config and rebuild the SCM
failover proxy set (add SCM proxies) -- the client-side counterpart to
HDDS-13890.
*Acceptance criteria:*
* With the OM running, updating the SCM node config and invoking
reconfiguration makes it aware of a newly added SCM without a restart.
* After reconfiguration, the OM can reach the new SCM when it becomes leader.
* Covers the SCM proxy providers the OM actually uses: block-location and
container-location.
was:
HDDS-13890 made DataNodes refresh their SCM connections without a restart, but
it only touched the DataNode heartbeat/registration path
({{{}SCMConnectionManager{}}} + {{{}StateContext{}}} endpoints). The OM reaches
SCM through a different mechanism: the {{SCMFailoverProxyProvider}} family,
which loads the SCM list once at startup via
{{SCMNodeInfo.buildNodeInfo(conf)}} ({{SCMFailoverProxyProviderBase}}) and
never reloads. The OM does not register {{ozone.scm.nodes}} as reconfigurable,
and the proxy providers have no reload hook.
Consequence: when a new SCM is added, the OM does not learn about it. While the
new SCM stays a follower this is harmless -- the OM still reaches the leader
via the existing configured SCMs. But once leadership moves to the new SCM, the
OM cannot route to a leader that is not in its static config and requests fail
(the same failover-to-unconfigured-leader outage as the leadership-transfer
scenario). So the OM must eventually learn the new SCM -- and the only
alternative today is an OM restart, which violates the restart-free requirement.
Proposed change:
* Register {{ozone.scm.nodes.<svc>}} (and the address keys) as reconfigurable
on the OM, implemented in the {{SCMFailoverProxyProviderBase}} reload path.
* On reconfiguration, reload the SCM list from config and rebuild the SCM
failover proxy set (add SCM proxies) -- the client-side counterpart to
HDDS-13890.
Acceptance criteria:
* With the OM running, updating the SCM node config and invoking
reconfiguration makes it aware of a newly added SCM without a restart.
* After reconfiguration, the OM can reach the new SCM when it becomes leader.
* Covers the SCM proxy providers the OM actually uses: block-location and
container-location.
> Allow the Ozone Manager to dynamically reconfigure its SCM node list
> (ozone.scm.nodes / ozone.scm.address) without a restart
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16300
> URL: https://issues.apache.org/jira/browse/HDDS-16300
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Mohammadhani Fouladgar
> Assignee: Mohammadhani Fouladgar
> Priority: Major
>
> HDDS-13890 made DataNodes refresh their SCM connections without a restart,
> but it only touched the DataNode heartbeat/registration path
> ({{{}SCMConnectionManager{}}} + {{{}StateContext{}}} endpoints). The OM
> reaches SCM through a different mechanism: the {{SCMFailoverProxyProvider}}
> family, which loads the SCM list once at startup via
> {{SCMNodeInfo.buildNodeInfo(conf)}} ({{SCMFailoverProxyProviderBase}}) and
> never reloads. The OM does not register {{ozone.scm.nodes}} as
> reconfigurable, and the proxy providers have no reload hook.
> Consequence: when a new SCM is added, the OM does not learn about it. While
> the new SCM stays a follower this is harmless -- the OM still reaches the
> leader via the existing configured SCMs. But once leadership moves to the new
> SCM, the OM cannot route to a leader that is not in its static config and
> requests fail (the same failover-to-unconfigured-leader outage as the
> leadership-transfer scenario). So the OM must eventually learn the new SCM --
> and the only alternative today is an OM restart, which violates the
> restart-free requirement.
> *Proposed change:*
> * Register {{ozone.scm.nodes.<svc>}} (and the address keys) as reconfigurable
> on the OM, implemented in the {{SCMFailoverProxyProviderBase}} reload path.
> * On reconfiguration, reload the SCM list from config and rebuild the SCM
> failover proxy set (add SCM proxies) -- the client-side counterpart to
> HDDS-13890.
> *Acceptance criteria:*
> * With the OM running, updating the SCM node config and invoking
> reconfiguration makes it aware of a newly added SCM without a restart.
> * After reconfiguration, the OM can reach the new SCM when it becomes leader.
> * Covers the SCM proxy providers the OM actually uses: block-location and
> container-location.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
