[
https://issues.apache.org/jira/browse/HDDS-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17116:
------------------------------
Description:
Start the SCM HTTP server right after Kerberos login in the
StorageContainerManager constructor, instead of in start(). Start it only once,
and stop it if the constructor fails. Gate /dbCheckpoint until SCM is ready.
Track phases: Initializing, Opening SCM DB, Loading pipelines, Loading
containers, Initializing managers, Starting services, Replaying Raft log (last
applied index vs. the leader's commit index), Waiting for safe mode exit (one
step per safe mode rule).
Guard the SCM UI scripts against missing beans during startup, and change the
rolling-upgrade driver.sh to wait for SCM readiness instead of port 9876.
h3. Implementation notes
* The metrics system is still initialized in start(), not before the early HTTP
server: starting it in the constructor makes duplicate metrics sources fail in
unit tests that construct SCMs without starting them. /jmx (including the
StartupProgress bean) works during startup; metrics2 sources on /prom fill in
once start() runs.
* The constructor body after the HTTP server start is wrapped in try/catch to
stop the HTTP server on failure, so most of the constructor diff is
re-indentation ({{git diff -w}} shows the real change).
* Container loading is a counted step (estimated key count of the container
table, corrected at the end). Pipeline loading only shows elapsed time.
* Raft log replay is reported from SCMStateMachine's follower catch-up check:
done / total are the applied entries vs. the leader's commit index captured at
start, relative to the applied index when the phase began.
* Safe mode steps end when their rule is validated. The phase ends on safe mode
exit, including forced exit and disabled safe mode. The rule counts (e.g.
registered datanodes) are not shown as progress yet.
* rolling-upgrade driver.sh now waits on the SCM datanode port 9861, which an
HA SCM opens only once it has caught up with the leader. The HTTP port 9876 now
opens early in startup.
h3. Dependencies
*Depends on HDDS-17115 (and so on HDDS-17107).* HDDS-17116.001.diff applies on
top of HDDS-17115.001.diff, which applies on top of the HDDS-17107 change
(https://github.com/apache/ozone/pull/11611).
Testing: new TestStorageContainerManager#startupProgress (503 from
/dbCheckpoint before start(), phase transitions, completion on safe mode exit);
hdds-server-scm unit tests; checkstyle.
was:
Start the SCM HTTP server right after Kerberos login in the
StorageContainerManager constructor, instead of in start(). Initialize metrics
before it, start it only once, and stop it if the constructor fails. Gate
/dbCheckpoint until SCM is ready.
Track phases: Initializing, Opening SCM DB, Loading pipelines, Loading
containers, Initializing managers, Starting services, Replaying Raft log (last
applied index vs. the leader's commit index), Waiting for safe mode exit (one
step per safe mode rule).
Guard the SCM UI scripts against missing beans during startup, and change the
rolling-upgrade driver.sh to wait for SCM readiness instead of port 9876.
> Start SCM HTTP server early and show SCM startup progress
> ---------------------------------------------------------
>
> Key: HDDS-17116
> URL: https://issues.apache.org/jira/browse/HDDS-17116
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-17116.001.diff
>
>
> Start the SCM HTTP server right after Kerberos login in the
> StorageContainerManager constructor, instead of in start(). Start it only
> once, and stop it if the constructor fails. Gate /dbCheckpoint until SCM is
> ready.
> Track phases: Initializing, Opening SCM DB, Loading pipelines, Loading
> containers, Initializing managers, Starting services, Replaying Raft log
> (last applied index vs. the leader's commit index), Waiting for safe mode
> exit (one step per safe mode rule).
> Guard the SCM UI scripts against missing beans during startup, and change the
> rolling-upgrade driver.sh to wait for SCM readiness instead of port 9876.
> h3. Implementation notes
> * The metrics system is still initialized in start(), not before the early
> HTTP server: starting it in the constructor makes duplicate metrics sources
> fail in unit tests that construct SCMs without starting them. /jmx (including
> the StartupProgress bean) works during startup; metrics2 sources on /prom
> fill in once start() runs.
> * The constructor body after the HTTP server start is wrapped in try/catch to
> stop the HTTP server on failure, so most of the constructor diff is
> re-indentation ({{git diff -w}} shows the real change).
> * Container loading is a counted step (estimated key count of the container
> table, corrected at the end). Pipeline loading only shows elapsed time.
> * Raft log replay is reported from SCMStateMachine's follower catch-up check:
> done / total are the applied entries vs. the leader's commit index captured
> at start, relative to the applied index when the phase began.
> * Safe mode steps end when their rule is validated. The phase ends on safe
> mode exit, including forced exit and disabled safe mode. The rule counts
> (e.g. registered datanodes) are not shown as progress yet.
> * rolling-upgrade driver.sh now waits on the SCM datanode port 9861, which an
> HA SCM opens only once it has caught up with the leader. The HTTP port 9876
> now opens early in startup.
> h3. Dependencies
> *Depends on HDDS-17115 (and so on HDDS-17107).* HDDS-17116.001.diff applies
> on top of HDDS-17115.001.diff, which applies on top of the HDDS-17107 change
> (https://github.com/apache/ozone/pull/11611).
> Testing: new TestStorageContainerManager#startupProgress (503 from
> /dbCheckpoint before start(), phase transitions, completion on safe mode
> exit); hdds-server-scm unit tests; checkstyle.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
