Wei-Chiu Chuang created HDDS-16638:
--------------------------------------
Summary: Fix kubernetes CI ozone-ha SCM HA startup (DNS / safe
mode wait)
Key: HDDS-16638
URL: https://issues.apache.org/jira/browse/HDDS-16638
Project: Apache Ozone
Issue Type: Bug
Reporter: Wei-Chiu Chuang
h2. Problem
The {{kubernetes}} CI job fails on the {{ozone-ha}} example (see GitHub
Actions). {{wait_for_startup}} never sees {{SCM exiting safe mode.}} in
{{scm-0}} logs, so Robot tests never run and {{output.xml}} is missing.
h2. Root cause
SCM HA pods start with Ratis configured for {{scm-0}}, {{scm-1}}, and {{scm-2}}
while the StatefulSet uses default {{OrderedReady}} scheduling. {{scm-0}}
starts before peer pods (and their headless-service DNS records) exist, which
produces {{UnknownHostException}} for {{scm-1.scm...}} / {{scm-2.scm...}} and
prevents the SCM HA quorum from forming.
A secondary issue: {{all_pods_are_running}} counted every pod in the namespace,
so leftover {{prometheus}}/{{jaeger}} pods from {{ozone-dev}} could block
startup checks even when Ozone pods were healthy.
h2. Fix
* Set {{podManagementPolicy: Parallel}} on the SCM StatefulSet and add a
{{wait-for-scm-peers}} init container that waits for {{scm-0.scm}},
{{scm-1.scm}}, and {{scm-2.scm}} DNS before {{init}}/{{bootstrap}}.
* Scope {{all_pods_are_running}} to {{app=ozone}} and wait for pod deletion
after {{reset_k8s_env}}.
* Treat any SCM pod logging safe-mode exit as cluster ready (HA leader may not
be {{scm-0}}).
h2. Testing
* {{bash -n}} on {{testlib.sh}}
* {{./hadoop-ozone/dev-support/checks/bats.sh}}
* {{kubernetes}} CI job on PR
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
