[
https://issues.apache.org/jira/browse/HDDS-16172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Meng-Shuan Tsai reassigned HDDS-16172:
--------------------------------------
Assignee: Meng-Shuan Tsai
> Scaling up OM replicas leaves the new OM pod stuck in Init CrashLoopBackOff
> ---------------------------------------------------------------------------
>
> Key: HDDS-16172
> URL: https://issues.apache.org/jira/browse/HDDS-16172
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Meng-Shuan Tsai
> Assignee: Meng-Shuan Tsai
> Priority: Major
>
> h3. Problem
> Scaling up OM replicas (e.g. {{om.replicas: 2 -> 3}} with
> {{{}om.persistence.enabled=true{}}}) leaves the new OM pod stuck in
> {{Init:0/1}} CrashLoopBackOff forever, and {{helm upgrade --wait}} times out.
> The {{om-bootstrap}} init container script
> (charts/ozone/templates/om/om-bootstrap-configmap.yaml) runs:
> {code:java}
> ozone admin om --set "ozone.om.nodes.$OZONE_CLUSTER_ID=$overwriteCmd"
> --bootstrap
> {code}
> but {{ozone admin om}} has no {{{}-{}}}set or {{-bootstrap}} option (neither
> in 2.0.0, the chart's appVersion when the script was introduced in
> HDDS-11618, nor in the current 2.2.0). The command fails with exit code 2,
> the script retries 3 times and gives up, and the init container crash-loops:
> {code:java}
> Unknown options: '--set',
> 'ozone.om.nodes.cluster1=ozone-om-0,ozone-om-1,ozone-om-2', '--bootstrap'
> Usage: ozone admin om [-hV] [--verbose] [COMMAND]
> Bootstrap failed after 3 attempts with exit code 2
> {code}
> Note that simply fixing the command to {{ozone om --bootstrap}} is not
> enough: that command keeps running as the OM daemon on success, so it can
> never complete inside an init container, and the config check it performs
> ({{{}checkConfigBeforeBootstrap{}}}) requires the existing OMs to be
> restarted with the new node list first - a rolling restart which is itself
> blocked by the new pod being stuck in Init. Details and fix proposals will
> follow in the comments/PR.
> h3. Steps to reproduce (kind)
> # Install with {{{}om.replicas=3{}}}, persistence enabled for om/scm/datanode
> # {{helm upgrade}} to {{om.replicas=2}} (scale-down works)
> # {{helm upgrade}} back to {{om.replicas=3}}
> # {{ozone-om-2}} is stuck in {{Init:0/1}} CrashLoopBackOff with the log above
> This path has never worked since it was introduced in HDDS-11618; chart CI
> ({{{}ct lint{}}} / {{{}ct install{}}}) never exercises a scale-up against a
> live cluster, so it went unnoticed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
