[
https://issues.apache.org/jira/browse/HDDS-16638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16638:
----------------------------------
Labels: ci kubernetes pull-request-available scm-ha (was: ci kubernetes
scm-ha)
> Fix kubernetes CI ozone-ha SCM HA startup (DNS / safe mode wait)
> ----------------------------------------------------------------
>
> Key: HDDS-16638
> URL: https://issues.apache.org/jira/browse/HDDS-16638
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Labels: ci, kubernetes, pull-request-available, scm-ha
>
> h2. Problem
> The {{kubernetes}} CI job fails on the {{ozone-ha}} example (see GitHub
> Actions). {{wait_for_startup}} never sees {{SCM exiting safe mode.}} in
> {{scm-0}} logs, so Robot tests never run and {{output.xml}} is missing.
> h2. Root cause
> SCM HA pods start with Ratis configured for {{scm-0}}, {{scm-1}}, and
> {{scm-2}} while the StatefulSet uses default {{OrderedReady}} scheduling.
> {{scm-0}} starts before peer pods (and their headless-service DNS records)
> exist, which produces {{UnknownHostException}} for {{scm-1.scm...}} /
> {{scm-2.scm...}} and prevents the SCM HA quorum from forming.
> A secondary issue: {{all_pods_are_running}} counted every pod in the
> namespace, so leftover {{prometheus}}/{{jaeger}} pods from {{ozone-dev}}
> could block startup checks even when Ozone pods were healthy.
> h2. Fix
> * Set {{podManagementPolicy: Parallel}} on the SCM StatefulSet and add a
> {{wait-for-scm-peers}} init container that waits for {{scm-0.scm}},
> {{scm-1.scm}}, and {{scm-2.scm}} DNS before {{init}}/{{bootstrap}}.
> * Scope {{all_pods_are_running}} to {{app=ozone}} and wait for pod deletion
> after {{reset_k8s_env}}.
> * Treat any SCM pod logging safe-mode exit as cluster ready (HA leader may
> not be {{scm-0}}).
> h2. Testing
> * {{bash -n}} on {{testlib.sh}}
> * {{./hadoop-ozone/dev-support/checks/bats.sh}}
> * {{kubernetes}} CI job on PR
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
