github-actions[bot] commented on issue #9523: URL: https://github.com/apache/cloudstack/issues/9523#issuecomment-5571866620
## ๐ฏ Triage report Hosts intermittently fail to enter maintenance mode: some VMs migrate off successfully while others remain, eventually driving the host into `ErrorInMaintenance`. The reporter observed no corresponding activity in the agent log on the vacating host during the stalled migration attempts, suggesting the management server may not be issuing (or is losing track of) migration commands for the stuck VMs. ### ๐ Assessment | Dimension | Value | Reasoning | |---|---|---| | **Type** | type:bug | Reproducible operational defect: expected clean transition to maintenance, actual is ErrorInMaintenance state. | | **Component** | component:kvm | Environment is a 3-node KVM cluster (management-server + KVM agents), and the reported symptom is centered on host maintenance/live-migration behavior. | | **Severity** | Severity:Major | Blocks a core operational workflow (host maintenance) and can leave a cluster degraded, but a documented workaround (manual migration) exists. | | **Labels** | type:bug, component:kvm, Severity:Major | See above. | | **Coding agent** | Not suitable | Root cause is unconfirmed โ requires log analysis/investigation of management-server โ agent interaction during migration scheduling; not a well-defined, self-contained fix. | ### ๐ Similar issues - https://github.com/apache/cloudstack/issues/11286 (related) โ Also involves maintenance-mode migrations interacting badly with agent state (`PrepareForMigrationCommand`/`AgentUnavailableException`), leading to VM stop; different failure mode but same problem domain (maintenance mode + live migration race conditions). - https://github.com/apache/cloudstack/issues/3720 (related) โ Older, closed issue about VMs being hard-stopped during maintenance-mode migration when migration fails; same general area (maintenance mode reliability on KVM) though the specific symptom (stop vs. stuck/ErrorInMaintenance) differs. <details><summary>๐ก Notes and suggestions</summary> - The issue has had some back-and-forth (manual migration was confirmed to work, and rebalancing the cluster appeared to "fix" it temporarily), but the underlying root cause was never identified. It was marked "unplanned" in Nov 2024 and picked up by the stale bot in Dec 2025 with no resolution. - To move this forward, it would help to have: - Management-server logs spanning the full maintenance attempt (not just a screenshot) โ ideally text, covering both the failed migration attempt and a subsequent manual migration for comparison. - Confirmation of the specific CloudStack version/build (4.19.1.0 was reported, but this is now over a year old โ worth checking if reproducible on a current release, e.g., 4.19.x/4.20.x). - Whether `expunge_delay`/`migrate wait` or host allocation exceptions were logged for the specific VMs that got stuck, and whether they share HA or reserved-capacity settings that differ from the VMs that migrated successfully. - Given the multi-year gap and lack of a confirmed root cause, a maintainer should decide whether to keep this open for further investigation, ask for fresh logs on a current release, or close as stale/needs-reproducing. </details> > Generated by [Daily Issue Triage](https://github.com/apache/cloudstack/actions/runs/34127932605) ยท sonnet50 95K ยท [โท](https://github.com/search?q=repo%3Aapache%2Fcloudstack+%22gh-aw-workflow-call-id%3A+apache%2Fcloudstack%2Fdaily-issue-triage%22&type=issues) > <details> <summary>Add this agentic workflows to your repo</summary> To install this agentic workflow, run ``` gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9 ``` </details> <!-- gh-aw-agentic-workflow: Daily Issue Triage, engine: copilot, version: 1.0.52, model: claude-sonnet-5, id: 34127932605, workflow_id: daily-issue-triage, run: https://github.com/apache/cloudstack/actions/runs/34127932605 --> <!-- gh-aw-workflow-call-id: apache/cloudstack/daily-issue-triage --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
