github-actions[bot] commented on issue #14206: URL: https://github.com/apache/cloudstack/issues/14206#issuecomment-5780830016
## ๐ฏ Triage report The reporter documents a race condition on KVM hosts where an out-of-band `PingCommand` (triggered by an unrelated instance crashing/shutting down on the same host) causes CloudStack to mistakenly mark a *different, still-starting* instance as `PowerReportMissing`. This releases the instance's IP/NICs without ever sending a `StopCommand`, so the domain keeps running unmanaged on the hypervisor while its IP is reassigned to another instance and its volume becomes stranded in a pending-delete state. The report includes a real-world timeline, relevant code excerpts (`VirtualMachinePowerStateSyncImpl`, `VirtualMachineManagerImpl`, `LibvirtDomainListener`), and a clear root-cause breakdown across five contributing code paths. ### ๐ Assessment | Dimension | Value | Reasoning | |---|---|---| | **Type** | type:bug | Concrete defect with reproducible sequence and root-cause code analysis | | **Component** | component:kvm | Confirmed KVM/libvirt-specific interaction (out-of-band ping from `LibvirtDomainListener`) | | **Severity** | Severity:Major | Can cause IP address conflicts between live instances and stranded/undeletable primary storage; requires a specific timing window so not universally triggered, but has real operational impact | | **Labels** | type:bug, component:kvm, Severity:Major | Based on documented technical root cause | | **Coding agent** | Needs more info | Root cause is clearly identified, but the fix touches five separate code paths (power-state sync, missing-report handling, expunge/advanceStop early-return, power-on-report-for-destroyed handling, unknown-instance logging) and requires a design decision on how much of the behavior to change (e.g., is `force` semantics change safe for other callers?) before an automated agent could safely implement it | ### ๐ Similar issues No close duplicates were found. Related KVM/HA issues surfaced by search (e.g., #12921, #13605, #14178, #13376) concern out-of-band-management (OOBM)/host-HA fencing scenarios, not this VM power-state race condition, so they are not flagged as related. <details><summary>๐ก Notes and suggestions</summary> - Maintainers should confirm intended semantics of `force`/`outOfBand` in `PingCommand`: it appears designed only to expedite a *reported* stop, not to bypass the graceful period for a *missing* report. Applying `force` to `processMissingVmReport()` conflates "instance reported stopped" with "instance absent from a report," which is the core defect per the reporter's analysis. - Consider whether `handlePowerOffReportWithNoPendingJobsOnVM()` should send a `StopCommand` even in the `PowerReportMissing` branch before releasing resources, to avoid orphaned running domains. - `advanceStop()`'s early return for `Stopped`/`Destroyed`/`Expunging`/`Error` states means `vm.destroy.forcestop` never gets a chance to contact the host โ worth checking if that early-return should be conditioned on whether the host still might have an active domain. - Given RBD/Ceph is called out as the only reason the extra disk isn't deleted from under a running guest, this may deserve cross-checking against other primary storage backends where the same protection might not exist. - Recommend a maintainer with engine-orchestration expertise validate the proposed fix approach before any implementation work begins, given the five interacting code paths and behavioral trade-offs involved. </details> > Generated by [Daily Issue Triage](https://github.com/apache/cloudstack/actions/runs/35734152618) ยท sonnet50 93.5K ยท [โท](https://github.com/search?q=repo%3Aapache%2Fcloudstack+%22gh-aw-workflow-call-id%3A+apache%2Fcloudstack%2Fdaily-issue-triage%22&type=issues) > <details> <summary>Add this agentic workflows to your repo</summary> To install this agentic workflow, run ``` gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9 ``` </details> <!-- gh-aw-agentic-workflow: Daily Issue Triage, engine: copilot, version: 1.0.52, model: claude-sonnet-5, id: 35734152618, workflow_id: daily-issue-triage, run: https://github.com/apache/cloudstack/actions/runs/35734152618 --> <!-- gh-aw-workflow-call-id: apache/cloudstack/daily-issue-triage --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
