shuan1026 opened a new pull request, #43: URL: https://github.com/apache/ozone-helm-charts/pull/43
## What changes were proposed in this pull request? This change adds a `readinessProbe` for HA OM/SCM deployments so pods are removed from Service endpoints until they have a stable Ratis role (`LEADER` or `FOLLOWER`). The chart already has (or will have via [[HDDS-14349](https://issues.apache.org/jira/browse/HDDS-14349)]) an HTTP `livenessProbe` and a roles-based `startupProbe`. **Startup** answers “still initializing do not treat liveness failures as fatal.” **Readiness** answers “should this pod receive client traffic via Service endpoints now?” A roles exec `readinessProbe` matches the [[reviewer sample](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2631983830)] and is stricter than UI `httpGet`: a process can respond on the UI port before OM/SCM has joined the HA ring in a follower/leader role. ### In scope * OM HA (`replicas > 1`): `readinessProbe` via `ozone admin om roles`, requiring `LEADER|FOLLOWER` on the line that matches `$(hostname)` * SCM HA (`replicas > 1`): `readinessProbe` via `ozone admin scm roles`, with `$(hostname -f)` to match SCM roles output format * Probe timings from kind testing: `timeoutSeconds: 20` (sample `5` is too low while peers/DNS/Ratis are still forming) * **SCM HA headless DNS (if not already present):** `publishNotReadyAddresses: true` on `scm-service-headless` when `scm.replicas > 1` — same peer-DNS requirement as OM during bootstrap/readiness ### Why add readiness (Things to confirm) [[HDDS-1668](https://issues.apache.org/jira/browse/HDDS-1668)] warned against readiness on the OM HA example: Ready-gated endpoints can fight peer DNS needed during startup (`OzoneManager.loadOMHAConfigs`). That concern is may still valid if readiness is added **without** companions. This PR follows the reviewer sample and addresses former by: 1. Relying on **14349** for startup + OM `Parallel` + OM headless `publishNotReadyAddresses` so peers are discoverable while pods are not Ready 2. Using readiness only to gate **Service endpoints** (traffic), not to block headless peer DNS once `publishNotReadyAddresses` is set **Trade-off:** `ozone admin * roles` as a **recurring** readiness check has ongoing cost (fork JVM/CLI + RPC). Acceptable for HA correctness in this chart; a lighter `httpGet` readiness remains a possible follow-up if maintainers prefer. ### Probe shape (illustrative; OM) ```yaml readinessProbe: exec: command: - sh - -c - | ozone admin om roles -id {{ .Values.clusterId }} 2>/dev/null \ | grep $(hostname) | grep -qE 'LEADER|FOLLOWER' && exit 0 || exit 1 initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 20 failureThreshold: 3 ``` SCM HA uses `ozone admin scm roles` with `$(hostname -f)` and the same `LEADER|FOLLOWER` grep. Non-HA (`replicas == 1`) does not add roles exec readiness. Startup ([[HDDS-14349](https://issues.apache.org/jira/browse/HDDS-14349)]) only checks membership in the roles list; readiness additionally requires `LEADER|FOLLOWER`. ## What is the link to the Apache JIRA * This issue (readiness): https://issues.apache.org/jira/browse/HDDS-15874 * Startup (prerequisite): https://issues.apache.org/jira/browse/HDDS-14349 * Parent: https://issues.apache.org/jira/browse/HDDS-14382 Review discussion: https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2631983830 ## How was this patch tested? ### 1. kind — default install (OM HA) On a fresh kind cluster: ```bash helm upgrade --install ozone charts/ozone --wait --timeout 15m ``` Results: * Release deployed; all pods Ready with no restarts * During cold start, OM may show 1–2 Startup `timed out after 20s` events while peers/roles are not ready; within `failureThreshold` the probe succeeds and Ready is reached **without** repeated liveness Killing * Green CI run and manually verified leader transfer and basic ozone commands ### 2. kind — SCM HA (`scm.replicas=3`, fresh install) On a **new** release/cluster (do not scale an existing `scm.replicas=1` in place): ```bash helm upgrade --install ozone charts/ozone \ --set scm.replicas=3 \ --wait --timeout 20m ``` Same pass criteria: Ready, no restart storm. **Out of scope for this issue/PR:** `kubectl scale` or `helm upgrade --set scm.replicas=3` migrating an existing **scm 1→3** deployment (known STS revision / `addSCM` issues; track separately). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
