[
https://issues.apache.org/jira/browse/HDDS-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17117:
------------------------------
Description:
Start the OM HTTP server early in the OzoneManager constructor, right after the
OM storage check, instead of in start(). Start it only once (including for
inactive OMs in MiniOzoneHAClusterImpl, which get it in restart() if it is not
running), and stop it if the constructor fails. Gate /serviceList,
/snapshotList, /dbCheckpoint and /v2/dbCheckpoint until OM is ready. Without
the gate, /serviceList reports the OM as LEADER while the Ratis server is not
yet created.
Track phases: Initializing, Waiting for SCM, Opening OM DB, Loading snapshot
chain, Loading prefixes, Initializing snapshot manager, Initializing Ratis,
Starting services, Replaying Raft log.
h3. Implementation notes
* As for SCM (HDDS-17116), the metrics system is still initialized in start().
/jmx (including the StartupProgress bean) works during startup; metrics2
sources on /prom fill in once start() runs.
* The constructor body after the HTTP server start is wrapped in try/catch to
stop the HTTP server on failure, so most of the constructor diff is
re-indentation ({{git diff -w}} shows the real change).
* Raft log replay: done / total are the entries applied vs. the local commit
index at the end of start(), relative to the applied index at that point.
Progress is reported from OzoneManagerStateMachine when the applied index
advances, including when a leader checkpoint is installed. A failure to set up
this tracking only logs a warning and ends the phase; it does not fail start().
* The StartupProgress MXBean is unregistered in stop() and registered again in
restart().
h3. Known limitations
* The replay target is the local commit index, not the leader's, so a lagging
follower reports startup complete before it has caught up with the leader.
Entries applied while the rest of start() runs are not counted.
* The snapshot chain is loaded again by metadataManager.start() in start();
that second load is shown under "Starting services". Some phases also cover
work they don't name: "Loading snapshot chain" includes the multi-tenant
manager, S3 secret store and delegation token manager setup, "Loading prefixes"
includes the key manager and authorizer setup, and the Ratis log segment load
is part of "Starting services".
* The shared startup page reads the first StartupProgress bean on /jmx, so in a
single-JVM MiniOzoneCluster the OM page may show another service's progress.
Separate processes are not affected.
h3. Dependencies
*Depends on HDDS-17116 (and so on HDDS-17115 and HDDS-17107).*
HDDS-17117.001.diff applies on top of HDDS-17116.001.diff, which applies on top
of HDDS-17115.001.diff and the HDDS-17107 change
(https://github.com/apache/ozone/pull/11611).
Testing: new TestOzoneManagerStartupProgress (503 from /serviceList before
start(), phase transitions, completion after start, 200 from /serviceList);
TestOzoneManagerStateMachine, TestKeyManagerUnit, TestOzoneManagerHttpServer,
TestOmSnapshotManager; checkstyle.
was:
Start the OM HTTP server right after Kerberos login in the OzoneManager
constructor, instead of in start(). Initialize metrics before it, start it only
once (including for inactive OMs in MiniOzoneHAClusterImpl), and stop it if the
constructor fails. Gate /serviceList, /snapshotList, /dbCheckpoint and
/v2/dbCheckpoint until OM is ready. Without the gate, /serviceList reports the
OM as LEADER while the Ratis server is not yet created.
Track phases: Initializing, Waiting for SCM, Opening OM DB, Loading snapshot
chain, Loading prefixes, Initializing snapshot manager, Starting Ratis,
Starting services, Replaying Raft log.
> Start OM HTTP server early and show OM startup progress
> -------------------------------------------------------
>
> Key: HDDS-17117
> URL: https://issues.apache.org/jira/browse/HDDS-17117
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Manager
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-17117.001.diff
>
>
> Start the OM HTTP server early in the OzoneManager constructor, right after
> the OM storage check, instead of in start(). Start it only once (including
> for inactive OMs in MiniOzoneHAClusterImpl, which get it in restart() if it
> is not running), and stop it if the constructor fails. Gate /serviceList,
> /snapshotList, /dbCheckpoint and /v2/dbCheckpoint until OM is ready. Without
> the gate, /serviceList reports the OM as LEADER while the Ratis server is not
> yet created.
> Track phases: Initializing, Waiting for SCM, Opening OM DB, Loading snapshot
> chain, Loading prefixes, Initializing snapshot manager, Initializing Ratis,
> Starting services, Replaying Raft log.
> h3. Implementation notes
> * As for SCM (HDDS-17116), the metrics system is still initialized in
> start(). /jmx (including the StartupProgress bean) works during startup;
> metrics2 sources on /prom fill in once start() runs.
> * The constructor body after the HTTP server start is wrapped in try/catch to
> stop the HTTP server on failure, so most of the constructor diff is
> re-indentation ({{git diff -w}} shows the real change).
> * Raft log replay: done / total are the entries applied vs. the local commit
> index at the end of start(), relative to the applied index at that point.
> Progress is reported from OzoneManagerStateMachine when the applied index
> advances, including when a leader checkpoint is installed. A failure to set
> up this tracking only logs a warning and ends the phase; it does not fail
> start().
> * The StartupProgress MXBean is unregistered in stop() and registered again
> in restart().
> h3. Known limitations
> * The replay target is the local commit index, not the leader's, so a lagging
> follower reports startup complete before it has caught up with the leader.
> Entries applied while the rest of start() runs are not counted.
> * The snapshot chain is loaded again by metadataManager.start() in start();
> that second load is shown under "Starting services". Some phases also cover
> work they don't name: "Loading snapshot chain" includes the multi-tenant
> manager, S3 secret store and delegation token manager setup, "Loading
> prefixes" includes the key manager and authorizer setup, and the Ratis log
> segment load is part of "Starting services".
> * The shared startup page reads the first StartupProgress bean on /jmx, so in
> a single-JVM MiniOzoneCluster the OM page may show another service's
> progress. Separate processes are not affected.
> h3. Dependencies
> *Depends on HDDS-17116 (and so on HDDS-17115 and HDDS-17107).*
> HDDS-17117.001.diff applies on top of HDDS-17116.001.diff, which applies on
> top of HDDS-17115.001.diff and the HDDS-17107 change
> (https://github.com/apache/ozone/pull/11611).
> Testing: new TestOzoneManagerStartupProgress (503 from /serviceList before
> start(), phase transitions, completion after start, 200 from /serviceList);
> TestOzoneManagerStateMachine, TestKeyManagerUnit, TestOzoneManagerHttpServer,
> TestOmSnapshotManager; checkstyle.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
