RosiKyu commented on PR #14033: URL: https://github.com/apache/cloudstack/pull/14033#issuecomment-5725532567
Upgrade testing is done. Built a second environment for it: populated 4.23.0.0, Ubuntu 24.04, **2 management servers**, three hypervisors with 2 hosts each (VMware ESXi 8.0.3, XenServer 8.3, KVM). Populated before upgrading with 3 isolated networks each with a running VM on its own hypervisor, a standard VPC, a redundant VPC (so a redundant VR pair), 6 routers Running, 5 VMs and a snapshot. Upgraded to `24.0.0`. Everything on the upgrade path passed. ### Upgrade results | Check | Result | |---|---| | Upgrade refused while a second MS is still running | PASS, refused cleanly and left the database untouched at 4.23.0.0 | | `Upgrade42300to2400` on a populated 4.23.0.0 database | PASS, single step, `version` gains exactly one row `24.0.0` | | Second MS started against the already upgraded database | PASS, `DB version = 24.0.0 Code Version = 24.0.0 ... no upgrade needed` | | `minreq.sysvmtemplate.version` after upgrade | PASS, rewritten `4.10.0` to `4.22.0`, three parts | | System VM template registration during the upgrade step | PASS, resolved 4.22.0 templates for all six hypervisor types | | Data survival | PASS, 6 routers, 5 VMs, 5 networks, 2 VPCs and the snapshot all intact and Running | | Routers across VMware, XenServer and KVM | PASS, all Running after the upgrade | CloudStack refuses to upgrade the database while another management server is still up, so I tested that first: mgmt2 left running on 4.23, mgmt1 started on 24.0.0. It refused and the database stayed at 4.23.0.0. ``` root@ref-trl-12374-v-Mu24-rositsa-kyuchukova-mgmt1:~# grep -a "clustered environment" /var/log/cloudstack/management/management-server.err 2026-09-17 18:13:59,823 ERROR [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) Database upgrade is required but the management server is running in a clustered environment. Please perform the database upgrade when the management server is not running in a clustered environment. ``` One note: that message only goes to `management-server.err` and `.out`. ### The two issues **Issue 1, export-templates.sh**: fixed by 569417ce and verified. I patched the file by hand on the management server and both KVM hosts. `bash -n` passes and the filename derivation is right for both schemes, including `4.22.0.0` giving `systemvmtemplate-4.22.0` which matches the template actually on disk. Good from my side. **Issue 2, minreq.sysvmtemplate.version**: still open. It reproduces the same way on the upgraded environment, so it is not environment specific. However, the overall error message in case of a badly formatted input is quite descriptive ### Separate problem, not caused by this PR Flagging here because it will stop anyone upgrading to 24.0.0 on Ubuntu. After the upgrade the management server starts and listens on 8080 but serves HTTP 503 permanently. Reproduced on both management servers. ``` root@ref-trl-12374-v-Mu24-rositsa-kyuchukova-mgmt1:~# grep -a "Failed to load module" /var/log/cloudstack/management/management-server.err | tail -1 2026-09-17 18:35:13,833 ERROR [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Failed to load module [root-ca] due to: org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is java.lang.NoSuchFieldError: xmss_SHAKE128_512ph ``` The management server ends up with two copies of the same BouncyCastle class on its classpath, one complete and one old and incomplete, and Java loads whichever it finds first. The order is not defined, so which one wins is luck. On both Ubuntu management servers it picked the incomplete one and the server never finished starting. My el9 environment has both copies too but picks the good one, so this is not Ubuntu specific. Forcing the good copy first on the classpath makes it start normally in about 75 seconds. Raising this as a separate issue with the details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
