github-actions[bot] commented on issue #14185: URL: https://github.com/apache/cloudstack/issues/14185#issuecomment-5715253097
## ๐ฏ Triage report Well-documented report of an infinite AutoScale scale-up loop: when a scale-up VM fails to **start** (landing in `State.Stopped` rather than `State.Error`), neither the errored-instance guard from #11244 nor the active-member counter account for `Stopped` VMs, so the group scales up forever. The reporter also identifies a related defect where `doScaleUp`'s cleanup only catches `ServerApiException`, missing the `CloudRuntimeException` actually thrown by `VirtualMachineManagerImpl.start()`, so failed VMs are never destroyed. Real-world impact was VM/IP exhaustion (2,296 VMs from a max_members=2 group). ### ๐ Assessment | Dimension | Value | Reasoning | |---|---|---| | **Type** | type:bug | Clear functional defect with reproducible logic flaw and code-level evidence | | **Component** | component:management-server | AutoScale scaling logic lives in `AutoScaleManagerImpl`/`AutoScaleVmGroupVmMapDaoImpl` (management server core); no dedicated `component:autoscale` label exists | | **Severity** | Severity:Major | Caused resource exhaustion (guest IP/subnet, thousands of leaked VMs) impacting unrelated deployments in the reporter's environment, though it requires a specific start-failure trigger to manifest | | **Labels** | type:bug, Severity:Major, component:management-server | See above | | **Coding agent** | Suitable | Root cause is precisely identified with exact file/method names and code snippets, and three concrete, scoped fix options are proposed (broaden guard to include `Stopped`, catch `CloudRuntimeException` in `doScaleUp`, or count `Stopped` in `countAvailableVmsByGroup`) | ### ๐ Similar issues - PR #11244 (referenced in issue) โ merged fix for the original infinite-scale-up bug (#9318), but only guards against `State.Error`, not `State.Stopped`; this issue identifies the gap left by that fix. - PR #9574 (referenced in issue) โ an earlier, unmerged one-line proposal that reportedly would have addressed a related variant. No separate open GitHub issue duplicating this exact report was found via search. <details><summary>๐ก Notes and suggestions</summary> - Recommend maintainers/assignee evaluate the reporter's three suggested fixes together rather than in isolation: option 1 (include `Stopped` in `getErroredInstanceCount()`) stops the infinite loop quickly, but combining it with option 2 (broadening the `doScaleUp` catch to `CloudRuntimeException`) prevents the VM leak from accumulating in the first place, which is the root cause of the exhaustion. - Worth double-checking whether other terminal/failure VM states (e.g. `Destroyed`, `Expunging`) should also be considered in `countAvailableVmsByGroup()` for consistency. - A regression/unit test simulating a start failure that raises `CloudRuntimeException` (not `ServerApiException`) from `advanceStart` would help confirm the fix and guard against future regressions. </details> > Generated by [Daily Issue Triage](https://github.com/apache/cloudstack/actions/runs/35227636453) ยท sonnet50 102.2K ยท [โท](https://github.com/search?q=repo%3Aapache%2Fcloudstack+%22gh-aw-workflow-call-id%3A+apache%2Fcloudstack%2Fdaily-issue-triage%22&type=issues) > <details> <summary>Add this agentic workflows to your repo</summary> To install this agentic workflow, run ``` gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9 ``` </details> <!-- gh-aw-agentic-workflow: Daily Issue Triage, engine: copilot, version: 1.0.52, model: claude-sonnet-5, id: 35227636453, workflow_id: daily-issue-triage, run: https://github.com/apache/cloudstack/actions/runs/35227636453 --> <!-- gh-aw-workflow-call-id: apache/cloudstack/daily-issue-triage --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
