bhouse-nexthop opened a new issue, #14626: URL: https://github.com/apache/cloudstack/issues/14626
### problem If the system VM template registration step of a database upgrade fails, the upgrade can't be completed by fixing the cause and restarting the management server. The restart skips the template step, and the server comes up on the new version with the old system VM template still configured. The only way back we found was to restore the database backup and run the upgrade again. The cause is the order of operations in `DatabaseUpgradeChecker.upgrade()`: ```java executeProcedureScripts(); final DbUpgrade[] upgrades = executeUpgrades(dbVersion, currentVersion); executeViewScripts(); updateSystemVmTemplates(upgrades); ``` - `executeUpgrades()` runs each step of the path. `executeUpgrade()` commits the schema and data changes and inserts the `version` row, and `executeUpgradeCleanup()` then sets that row to `Complete` and commits. - Only after every step is `Complete` does `updateSystemVmTemplates()` run. It calls `SystemVmTemplateRegistration.updateSystemVmTemplates()`, which looks for the template under the name from `metadata.ini` (for example `systemvm-kvm-4.22.0-x86_64`). If no template has that name, it registers one from the packaged template file. If that fails, it throws, and the management server stops. - On the next start, `VersionDaoImpl.getCurrentVersion()` returns the newest `Complete` row, which is now the code version. `doUpgrades()` logs "DB version and code version matches so no upgrade needed." and returns. `updateSystemVmTemplates()` is only reached from `upgrade()`, so it doesn't run again. The result is a management server on the new version with `router.template.*` and `minreq.sysvmtemplate.version` still at the old values, and nothing that marks the upgrade as incomplete. Nothing in the startup path checks for it, and nothing retries it. In our case, the template step failed because the new template had been registered ahead of the upgrade under a name other than the one the upgrade expects (`systemvm-kvm-4.22.0` instead of `systemvm-kvm-4.22.0-x86_64`). That was an operator error. But any failure in that step leaves the database in the same state, for example a missing or corrupt template file, or a secondary storage that can't be mounted. ### versions Seen on an upgrade from 4.20.3 to 4.22.1 on KVM with NFS secondary storage. The same ordering is on the 4.22 branch and on main. It has been there since template registration became part of the database upgrade in 4.16 (#4329). ### The steps to reproduce the bug 1. Run a 4.20 setup with at least one KVM cluster. 2. Make the template registration step fail. One way is to register the 4.22 system VM template ahead of the upgrade under a different name, such as `systemvm-kvm-4.22.0`, and move the packaged KVM template file out of `/usr/share/cloudstack-management/templates/systemvm/`. 3. Install the 4.22 packages and start the management server. The schema upgrade runs, then the template step fails with "Failed to register SystemVM Template. Upgrade failed" and the management server stops. 4. Check the `version` table. The rows for every step up to 4.22.x are `Complete`. 5. Fix the cause, for example by renaming the template to `systemvm-kvm-4.22.0-x86_64`, and start the management server again. 6. It logs "DB version and code version matches so no upgrade needed." and starts. The template step doesn't run. `router.template.kvm` and `minreq.sysvmtemplate.version` still hold the 4.20 values, and the template is not marked as a system template. ### What to do about it? Two ways to fix it come to mind. Both change upgrade or startup behaviour, so I'd rather agree on the approach here first, and then test it on a real upgrade. 1. **Retry on startup.** When the database version matches the code version, check whether the template named in `metadata.ini` is registered for each hypervisor and architecture in use, and run the registration step if one is missing. This is the smallest change, but it runs on every start. It has to tell a failed upgrade apart from a setup that is missing the template for another reason, for example a cluster of a new architecture added after the upgrade. It also has to decide whether a failure blocks startup. 2. **Mark the last step `Complete` only after the template step succeeds.** If the template step fails, leave the last `version` row at `Cleanup`. On the next start, when the newest row is the code version at `Cleanup`, run only the template step and then mark it `Complete`. Its SQL must not run again, because the scripts aren't idempotent. This keeps the retry to failed upgrades only. But `getCurrentVersion()` would return the previous version while the schema is already at the new one, so every reader of that value needs checking, and so does downgrading to the old packages. Until this is fixed, the only recovery we found is to restore the database backup, fix the cause, and run the upgrade again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
