[
https://issues.apache.org/jira/browse/HDDS-17119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17119:
------------------------------
Description:
Replace the TCP liveness probes on the OM and SCM RPC ports with HTTP probes on
/health/live, so that a slow start (DB load, Raft log replay) is not killed. If
readiness probes are added, the headless services need
publishNotReadyAddresses: true. Otherwise pods that are not ready drop out of
DNS, datanodes cannot reach SCM, and SCM can never leave safe mode.
h3. Implementation notes
Changes are in hadoop-ozone/dist/src/main/k8s/definitions/ozone; the examples
are regenerated with flekszible 2.3.0. The unrelated config-configmap.yaml
drift that regeneration produces on master is left out.
* OM and SCM liveness probes: httpGet /health/live on 9874 / 9876 instead of
tcpSocket 9862 / 9861. In SCM HA, 9861 only opens once the SCM has caught up
with the leader, so the old probe could kill an SCM during replay.
* Datanode: new liveness probe, httpGet /health/live on 9882.
* OM and Recon WAITFOR: scm-0.scm:9860 instead of scm-0.scm:9876. With the
early HTTP server (HDDS-17116), 9876 opens long before SCM can serve RPC. 9860
used to open just before 9876 in SCM start(), so the wait stays the same.
* k8s tests (examples/testlib.sh): wait for OM /health/ready instead of the
"HTTP server of ozoneManager listening" log line, which is now logged early in
the OM constructor (HDDS-17117).
* No readiness probes for OM, SCM and datanodes. Besides the DNS problem above,
the statefulsets use OrderedReady pod management, so in ozone-ha scm-1 would
not be created until scm-0 has left safe mode, which needs datanodes. A
readiness probe would also remove a starting pod from the *-public services and
hide the startup progress page. Adding them would need
publishNotReadyAddresses: true on the headless services (and
podManagementPolicy: Parallel for SCM HA); S3 Gateway keeps its existing
readiness probe.
h3. Known limitations
* WAITFOR and om --init run in the main container, so the liveness probe
already runs while OM waits for SCM. If SCM takes longer than about 50s
(initialDelaySeconds 30 + 3 failed probes) to open 9860, OM is restarted. This
was the same before. The datanode HTTP server starts after the volume DBs are
loaded, so a very slow datanode start could be killed the same way. A
startupProbe on /health/live would cover both.
* An HTTP server start failure, which these services treat as non-fatal, or
ozone.http.policy=HTTPS_ONLY now fails the liveness probe.
h3. Dependencies
*Depends on HDDS-17118 (and so on HDDS-17117, HDDS-17116, HDDS-17115 and
HDDS-17107).* HDDS-17119.001.diff applies on top of HDDS-17118.001.diff,
HDDS-17117.001.diff, HDDS-17116.001.diff, HDDS-17115.001.diff and the
HDDS-17107 change (https://github.com/apache/ozone/pull/11611), in that order.
Testing: regenerated examples match flekszible output for all example dirs;
bash -n on testlib.sh. Not run on a Kubernetes cluster.
was:Replace the TCP liveness probes on the OM and SCM RPC ports with HTTP
probes on /health/live, so that a slow start (DB load, Raft log replay) is not
killed. If readiness probes are added, the headless services need
publishNotReadyAddresses: true. Otherwise pods that are not ready drop out of
DNS, datanodes cannot reach SCM, and SCM can never leave safe mode.
> Use health endpoints for OM, SCM and Datanode probes in Kubernetes examples
> ---------------------------------------------------------------------------
>
> Key: HDDS-17119
> URL: https://issues.apache.org/jira/browse/HDDS-17119
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: kubernetes
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-17119.001.diff
>
>
> Replace the TCP liveness probes on the OM and SCM RPC ports with HTTP probes
> on /health/live, so that a slow start (DB load, Raft log replay) is not
> killed. If readiness probes are added, the headless services need
> publishNotReadyAddresses: true. Otherwise pods that are not ready drop out of
> DNS, datanodes cannot reach SCM, and SCM can never leave safe mode.
> h3. Implementation notes
> Changes are in hadoop-ozone/dist/src/main/k8s/definitions/ozone; the examples
> are regenerated with flekszible 2.3.0. The unrelated config-configmap.yaml
> drift that regeneration produces on master is left out.
> * OM and SCM liveness probes: httpGet /health/live on 9874 / 9876 instead of
> tcpSocket 9862 / 9861. In SCM HA, 9861 only opens once the SCM has caught up
> with the leader, so the old probe could kill an SCM during replay.
> * Datanode: new liveness probe, httpGet /health/live on 9882.
> * OM and Recon WAITFOR: scm-0.scm:9860 instead of scm-0.scm:9876. With the
> early HTTP server (HDDS-17116), 9876 opens long before SCM can serve RPC.
> 9860 used to open just before 9876 in SCM start(), so the wait stays the same.
> * k8s tests (examples/testlib.sh): wait for OM /health/ready instead of the
> "HTTP server of ozoneManager listening" log line, which is now logged early
> in the OM constructor (HDDS-17117).
> * No readiness probes for OM, SCM and datanodes. Besides the DNS problem
> above, the statefulsets use OrderedReady pod management, so in ozone-ha scm-1
> would not be created until scm-0 has left safe mode, which needs datanodes. A
> readiness probe would also remove a starting pod from the *-public services
> and hide the startup progress page. Adding them would need
> publishNotReadyAddresses: true on the headless services (and
> podManagementPolicy: Parallel for SCM HA); S3 Gateway keeps its existing
> readiness probe.
> h3. Known limitations
> * WAITFOR and om --init run in the main container, so the liveness probe
> already runs while OM waits for SCM. If SCM takes longer than about 50s
> (initialDelaySeconds 30 + 3 failed probes) to open 9860, OM is restarted.
> This was the same before. The datanode HTTP server starts after the volume
> DBs are loaded, so a very slow datanode start could be killed the same way. A
> startupProbe on /health/live would cover both.
> * An HTTP server start failure, which these services treat as non-fatal, or
> ozone.http.policy=HTTPS_ONLY now fails the liveness probe.
> h3. Dependencies
> *Depends on HDDS-17118 (and so on HDDS-17117, HDDS-17116, HDDS-17115 and
> HDDS-17107).* HDDS-17119.001.diff applies on top of HDDS-17118.001.diff,
> HDDS-17117.001.diff, HDDS-17116.001.diff, HDDS-17115.001.diff and the
> HDDS-17107 change (https://github.com/apache/ozone/pull/11611), in that order.
> Testing: regenerated examples match flekszible output for all example dirs;
> bash -n on testlib.sh. Not run on a Kubernetes cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
