[
https://issues.apache.org/jira/browse/HDDS-17119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Huang reassigned HDDS-17119:
---------------------------------
Assignee: (was: Andy Huang)
> Use health endpoints for OM, SCM and Datanode probes in Kubernetes examples
> ---------------------------------------------------------------------------
>
> Key: HDDS-17119
> URL: https://issues.apache.org/jira/browse/HDDS-17119
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: kubernetes
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-17119.001.diff
>
>
> Replace the TCP liveness probes on the OM and SCM RPC ports with HTTP probes
> on /health/live, so that a slow start (DB load, Raft log replay) is not
> killed. If readiness probes are added, the headless services need
> publishNotReadyAddresses: true. Otherwise pods that are not ready drop out of
> DNS, datanodes cannot reach SCM, and SCM can never leave safe mode.
> h3. Implementation notes
> Changes are in hadoop-ozone/dist/src/main/k8s/definitions/ozone; the examples
> are regenerated with flekszible 2.3.0. The unrelated config-configmap.yaml
> drift that regeneration produces on master is left out.
> * OM and SCM liveness probes: httpGet /health/live on 9874 / 9876 instead of
> tcpSocket 9862 / 9861. In SCM HA, 9861 only opens once the SCM has caught up
> with the leader, so the old probe could kill an SCM during replay.
> * Datanode: new liveness probe, httpGet /health/live on 9882.
> * OM and Recon WAITFOR: scm-0.scm:9860 instead of scm-0.scm:9876. With the
> early HTTP server (HDDS-17116), 9876 opens long before SCM can serve RPC.
> 9860 used to open just before 9876 in SCM start(), so the wait stays the same.
> * k8s tests (examples/testlib.sh): wait for OM /health/ready instead of the
> "HTTP server of ozoneManager listening" log line, which is now logged early
> in the OM constructor (HDDS-17117).
> * No readiness probes for OM, SCM and datanodes. Besides the DNS problem
> above, the statefulsets use OrderedReady pod management, so in ozone-ha scm-1
> would not be created until scm-0 has left safe mode, which needs datanodes. A
> readiness probe would also remove a starting pod from the *-public services
> and hide the startup progress page. Adding them would need
> publishNotReadyAddresses: true on the headless services (and
> podManagementPolicy: Parallel for SCM HA); S3 Gateway keeps its existing
> readiness probe.
> h3. Known limitations
> * WAITFOR and om --init run in the main container, so the liveness probe
> already runs while OM waits for SCM. If SCM takes longer than about 50s
> (initialDelaySeconds 30 + 3 failed probes) to open 9860, OM is restarted.
> This was the same before. The datanode HTTP server starts after the volume
> DBs are loaded, so a very slow datanode start could be killed the same way. A
> startupProbe on /health/live would cover both.
> * An HTTP server start failure, which these services treat as non-fatal, or
> ozone.http.policy=HTTPS_ONLY now fails the liveness probe.
> h3. Dependencies
> *Depends on HDDS-17118 (and so on HDDS-17117, HDDS-17116, HDDS-17115 and
> HDDS-17107).* HDDS-17119.001.diff applies on top of HDDS-17118.001.diff,
> HDDS-17117.001.diff, HDDS-17116.001.diff, HDDS-17115.001.diff and the
> HDDS-17107 change (https://github.com/apache/ozone/pull/11611), in that order.
> Testing: regenerated examples match flekszible output for all example dirs;
> bash -n on testlib.sh. Not run on a Kubernetes cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
