khumps opened a new issue, #14235:
URL: https://github.com/apache/cloudstack/issues/14235
### problem
In a multi-arch zone (aarch64 + x86_64 clusters), if a system VM (SSVM/CPVM)
cannot be
deployed on the architecture chosen by `system.vm.preferred.architecture`,
CloudStack retries
allocation using the other architecture's SystemVM template — logged as
*"Retrying with another
template"*. A ROOT volume has already been created for the first template at
that point, and
after the retry the VM has **two ROOT volumes, both at `device_id = 0`**,
one per architecture.
Nothing in the logs shows the first volume being released.
Every subsequent start attempt produces libvirt domain XML with two disks on
target `vda`,
which libvirt rejects:
```
XML error: target 'vda' duplicated for disk sources '<VOL_X86_UUID>' and
'<VOL_AARCH64_UUID>'
```
The system VM therefore cannot start on **any** host of **either**
architecture, and enters a
permanent create → fail → destroy → recreate loop.
### Impact
Losing all hosts of the preferred architecture in a mixed-arch zone leaves
the zone unable to
run system VMs at all — no SSVM (no template/ISO/snapshot operations) and no
CPVM (no console
access) — even though healthy hosts of the other architecture and a ready
SystemVM template
for that architecture are available. The zone does not self-heal.
### Determinism
Reproduced on every recreation attempt (VM ids 235, 236, 241, 242, 243).
Control cases confirm
the inverse: VMs allocated without hitting the fallback path (239, 240, 244)
each received
exactly one ROOT volume and started normally.
### Setup
Multi-arch zone, both architectures in the same zone:
```
ZONE1 (id 1)
├── POD_AARCH64 (id 1)
│ └── CLUSTER_AARCH64 (id 1) arch=aarch64 → AARCH64_HOST01 (host id 1)
└── POD_X86 (id 3)
├── CLUSTER_X86_A (id 5) arch=x86_64 → X86_HOST01 (host id 9)
└── CLUSTER_X86_B (id 6) arch=x86_64 → X86_HOST02 (host id 10),
X86_HOST03 (host id 11)
```
Host state during reproduction — the only aarch64 host is not schedulable,
all x86_64 hosts
are healthy:
```
AARCH64_HOST01 aarch64 Up / ErrorInMaintenance
X86_HOST01 x86_64 Up / Enabled
X86_HOST02 x86_64 Up / Enabled
X86_HOST03 x86_64 Up / Enabled
```
Both SystemVM templates are registered and `Ready`, so a valid x86_64
template is available:
```sql
SELECT id, unique_name, name, type, hypervisor_type, arch, state FROM
vm_template WHERE id IN (3,202);
+-----+-------------+-------------------------+--------+-----------------+---------+--------+
| id | unique_name | name | type | hypervisor_type |
arch | state |
+-----+-------------+-------------------------+--------+-----------------+---------+--------+
| 3 | routing-3 | SystemVM Template (KVM) | SYSTEM | KVM |
x86_64 | Active |
| 202 | routing-202 | SystemVM Template (KVM) | SYSTEM | KVM |
aarch64 | Active |
+-----+-------------+-------------------------+--------+-----------------+---------+--------+
```
Both `isready=true`, both QCOW2, both `Debian GNU/Linux 12 (64-bit)`, both
on the shared pool.
Relevant configuration at reproduction time
(`system.vm.preferred.architecture` is
**zone-scoped**; the value below is the effective zone value):
```
system.vm.preferred.architecture = aarch64 <- the trigger (zone-scoped)
migrate.vm.across.clusters = false
recreate.systemvm.enabled = false
systemvm.root.disk.size = -1
system.vm.default.hypervisor = NULL
```
### Evidence — two ROOT volumes on one VM, both `device_id = 0`
```sql
SELECT v.id vol, v.name, v.instance_id, v.volume_type, v.device_id,
v.template_id,
t.unique_name, t.arch, v.pool_id, v.state, v.created
FROM volumes v LEFT JOIN vm_template t ON t.id = v.template_id
WHERE v.instance_id IN (240,241) AND v.removed IS NULL ORDER BY
v.instance_id, v.id;
+-----+----------+-------------+-------------+-----------+-------------+-------------+---------+---------+-------+---------------------+
| vol | name | instance_id | volume_type | device_id | template_id |
unique_name | arch | pool_id | state | created |
+-----+----------+-------------+-------------+-----------+-------------+-------------+---------+---------+-------+---------------------+
| 262 | ROOT-240 | 240 | ROOT | 0 | 202 |
routing-202 | aarch64 | 1 | Ready | 2026-09-23 18:16:20 |
| 263 | ROOT-241 | 241 | ROOT | 0 | 202 |
routing-202 | aarch64 | 1 | Ready | 2026-09-23 18:18:44 |
| 264 | ROOT-241 | 241 | ROOT | 0 | 3 |
routing-3 | x86_64 | 1 | Ready | 2026-09-23 18:18:45 |
+-----+----------+-------------+-------------+-----------+-------------+-------------+---------+---------+-------+---------------------+
```
- VM **240** (CPVM, allocated while the aarch64 host was still available) —
1 ROOT volume.
- VM **241** (SSVM, allocated after the aarch64 host became unschedulable) —
2 ROOT volumes,
created one second apart, both at `device_id = 0`.
```sql
SELECT vm.id, vm.instance_name, vm.type, vm.state, vm.vm_template_id,
vm.host_id, vm.created
FROM vm_instance vm WHERE vm.id IN (240,241);
+-----+---------------+--------------------+----------+----------------+---------+---------------------+
| id | instance_name | type | state | vm_template_id |
host_id | created |
+-----+---------------+--------------------+----------+----------------+---------+---------------------+
| 240 | v-240-VM | ConsoleProxy | Running | 202 |
1 | 2026-09-23 18:16:19 |
| 241 | s-241-VM | SecondaryStorageVm | Starting | 3 |
11 | 2026-09-23 18:18:44 |
+-----+---------------+--------------------+----------+----------------+---------+---------------------+
```
`vm_instance.vm_template_id = 3` — the VM record was correctly re-templated
to x86_64; only the
aarch64 ROOT volume (263) from the abandoned first attempt was left behind.
The two volumes are backed by different-architecture templates:
```
$ qemu-img info <POOL_MOUNT>/<VOL_AARCH64_UUID>
virtual size: 4.88 GiB backing file: <POOL_MOUNT>/<TMPL_AARCH64_UUID>
# aarch64 template (id 202)
$ qemu-img info <POOL_MOUNT>/<VOL_X86_UUID>
virtual size: 4.88 GiB backing file: <POOL_MOUNT>/<TMPL_X86_UUID>
# x86_64 template (id 3)
```
### Evidence — the start failure
Management server, identical on every x86_64 host:
```
Unable to start VM on Host
{"id":10,"name":"X86_HOST02","type":"Routing","uuid":"<HOST02_UUID>"} due to
XML error: target 'vda' duplicated for disk sources
'<POOL_MOUNT>/<VOL_X86_UUID>' and '<POOL_MOUNT>/<VOL_AARCH64_UUID>'
Unable to start VM on Host
{"id":11,"name":"X86_HOST03","type":"Routing","uuid":"<HOST03_UUID>"} due to
XML error: target 'vda' duplicated for disk sources
'<POOL_MOUNT>/<VOL_X86_UUID>' and '<POOL_MOUNT>/<VOL_AARCH64_UUID>'
```
Agent side (`/var/log/cloudstack/agent/agent.log`):
```
2026-09-23 18:25:52,529 WARN [resource.wrapper.LibvirtStartCommandWrapper]
(AgentRequest-Handler-5:[]) (logid:62824aef) LibvirtException
org.libvirt.LibvirtException: XML error: target 'vda' duplicated for disk
sources '<POOL_MOUNT>/<VOL_X86_UUID>' and '<POOL_MOUNT>/<VOL_AARCH64_UUID>'
2026-09-23 18:26:22,295 WARN [resource.wrapper.LibvirtStartCommandWrapper]
(AgentRequest-Handler-3:[]) (logid:62824aef) LibvirtException ... (same)
2026-09-23 18:26:38,965 WARN [resource.wrapper.LibvirtStartCommandWrapper]
(AgentRequest-Handler-4:[]) (logid:62824aef) LibvirtException ... (same)
```
### Full management-server trace of the failing allocation
Single allocation pass for `s-241-VM` — `logid:343a509f`, thread
`secstorage-1`. Note the two
`Allocating Instance from Template` lines and the two `Adding disk object
[ROOT-241]` lines
within ~450 ms.
```
2026-09-23 18:18:44,716 INFO [c.c.v.ClusteredVirtualMachineManagerImpl]
(secstorage-1:[ctx-804dea2b]) (logid:343a509f) Allocating Instance from
Template: Template {"format":"QCOW2","id":202,"name":"SystemVM Template
(KVM)","uniqueName":"routing-202","uuid":"<TMPL_AARCH64_UUID>"} with hostname:
s-241-VM and 3 networks
2026-09-23 18:18:44,976 INFO [o.a.c.e.o.VolumeOrchestrator]
(secstorage-1:[ctx-804dea2b, ctx-c112c21a]) (logid:343a509f) Adding disk object
[ROOT-241] to VM [VM instance
{"id":241,"instanceName":"s-241-VM","state":"Stopped","type":"SecondaryStorageVm","uuid":"<SSVM_UUID>"}]
^-- creates volume 263 (template 202, aarch64)
2026-09-23 18:18:45,037 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(secstorage-1:[ctx-804dea2b]) (logid:343a509f) Excluding 2 clusters as they are
x86_64 arch, conflicting with the requested arch aarch64
2026-09-23 18:18:45,045 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl]
(secstorage-1:[ctx-804dea2b]) (logid:343a509f) Unable to allocate secondary
storage VM instance {"id":241,...} with Template
{"format":"QCOW2","id":202,...,"uniqueName":"routing-202",...} due to [Unable
to create a deployment for VM instance {"id":241,...}]. Retrying with another
template com.cloud.exception.InsufficientServerCapacityException: Unable to
create a deployment for VM instance {"id":241,...}Scope=interface
com.cloud.dc.DataCenter; id=1
^-- "Retrying with another template": no line shows volume 263 being
released
2026-09-23 18:18:45,074 INFO [c.c.v.ClusteredVirtualMachineManagerImpl]
(secstorage-1:[ctx-804dea2b]) (logid:343a509f) Allocating Instance from
Template: Template {"format":"QCOW2","id":3,"name":"SystemVM Template
(KVM)","uniqueName":"routing-3","uuid":"<TMPL_X86_UUID>"} with hostname:
s-241-VM and 3 networks
2026-09-23 18:18:45,164 INFO [o.a.c.e.o.VolumeOrchestrator]
(secstorage-1:[ctx-804dea2b, ctx-70e344c6]) (logid:343a509f) Adding disk object
[ROOT-241] to VM [VM instance {"id":241,...}]
^-- creates volume 264 (template 3, x86_64) — VM now has TWO ROOT
volumes
2026-09-23 18:18:45,239 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(secstorage-1:[ctx-804dea2b]) (logid:343a509f) Excluding 1 clusters as they are
aarch64 arch, conflicting with the requested arch x86_64
2026-09-23 18:18:45,292 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(secstorage-1:[ctx-804dea2b]) (logid:343a509f) Checking suitable pools for
volume [Volume
{"id":263,"instanceId":241,"name":"ROOT-241","uuid":"<VOL_AARCH64_UUID>","volumeType":"ROOT"},
ROOT] of VM [VM instance {"id":241,...}].
2026-09-23 18:18:45,349 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(secstorage-1:[ctx-804dea2b]) (logid:343a509f) Checking suitable pools for
volume [Volume
{"id":264,"instanceId":241,"name":"ROOT-241","uuid":"<VOL_X86_UUID>","volumeType":"ROOT"},
ROOT] of VM [VM instance {"id":241,...}].
^-- the planner now carries BOTH ROOT volumes for the same VM
```
Every subsequent start attempt plans both volumes and is rejected by the
agent:
```
2026-09-23 18:19:26,965 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-91:[ctx-8acced15, job-1069/job-2490, ctx-db3c7a5c])
(logid:62824aef) Excluding 1 clusters as they are aarch64 arch, conflicting
with the requested arch x86_64
2026-09-23 18:19:27,025 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-91:[...]) (logid:62824aef) Checking suitable pools for
volume [Volume {"id":263,...,"volumeType":"ROOT"}, ROOT] of VM [VM instance
{"id":241,...,"state":"Starting"}].
2026-09-23 18:19:27,030 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-91:[...]) (logid:62824aef) Checking suitable pools for
volume [Volume {"id":264,...,"volumeType":"ROOT"}, ROOT] of VM [VM instance
{"id":241,...,"state":"Starting"}].
2026-09-23 18:19:33,557 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-91:[...]) (logid:62824aef) Excluding 1 clusters as they are
aarch64 arch, conflicting with the requested arch x86_64
2026-09-23 18:19:33,596 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-91:[...]) (logid:62824aef) Checking suitable pools for
volume [Volume {"id":263,...}] ...
2026-09-23 18:19:33,601 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-91:[...]) (logid:62824aef) Checking suitable pools for
volume [Volume {"id":264,...}] ...
```
### Observations
No source was inspected; the following is only what the logs and database
show.
1. `PremiumSecondaryStorageManagerImpl` reports the preferred-arch attempt
failing with
`InsufficientServerCapacityException` and logs *"Retrying with another
template"*. This
occurs **after** `VolumeOrchestrator` has already logged `Adding disk
object [ROOT-241]`
for the first template.
2. Immediately after, `ClusteredVirtualMachineManagerImpl` logs a second
`Allocating Instance
from Template` — same VM instance id, different template id — and
`VolumeOrchestrator` logs
`Adding disk object [ROOT-241]` a second time.
3. Two ROOT volume rows are persisted for that VM, one per template/arch,
both with
`device_id = 0`. No log line indicates the first volume being released,
detached or
expunged between the two attempts.
4. From then on `DeploymentPlanningManagerImpl` evaluates **both** volumes
on every start
attempt, and the agent rejects the resulting domain XML with `target
'vda' duplicated`.
5. The condition does not clear by itself. Destroying the VM expunges both
volumes, but the
next recreation reproduces it identically as long as the fallback path is
reached.
The observable defect: **the arch-fallback retry results in a second ROOT
volume at the same
`device_id` on the same VM instance, and the volume from the abandoned
attempt is never
removed.** Whether the right fix is to clean up before retrying, reuse the
existing volume, or
prevent a duplicate `device_id` is for maintainers to judge.
Possibly related: #9981 (multi-arch zone — wrong-arch hosts offered for live
migration),
another case of incomplete arch handling. Multi-arch zones were introduced
in #9619 (4.20).
### Workaround (verified)
Set the preferred architecture to one that actually has capacity, so the
first allocation
attempt succeeds and the fallback retry is never reached.
⚠️ `system.vm.preferred.architecture` is **zone-scoped**. Changing only the
global value has no
effect while a zone-level override exists. Verified: the global was set to
`x86_64` at
18:36:02, and a system VM created at 18:37:56 — 114 s later — still
allocated the aarch64
template first, because `data_center_details` still held `aarch64`.
```sql
-- check for an overriding zone-scoped value first
SELECT dc_id, name, value FROM data_center_details
WHERE name = 'system.vm.preferred.architecture';
```
```
update configuration name=system.vm.preferred.architecture value=x86_64
zoneid=<ZONE_UUID>
```
Then expunge the stuck system VM so it is recreated. After doing so, the
next recreation
(`s-244-VM`) allocated exactly **one** ROOT volume from template 3 and
reached `Running` on an
x86_64 host, with the aarch64 host still unavailable:
```
v-240-VM Running tmpl=202 roots=1 (CPVM, on AARCH64_HOST01)
s-244-VM Running tmpl=3 roots=1 (SSVM, on X86_HOST03)
```
*(Summarised from `vm_instance` / `volumes`; `roots` = count of non-removed
ROOT volumes.)*
This isolates the defect to the fallback retry path — when the first
allocation attempt
succeeds, a system VM runs correctly on the non-preferred architecture with
a single ROOT
volume. It is not a general problem with x86_64 system VMs.
The reverse direction is untested: losing all x86_64 hosts while preferring
`x86_64` would be
expected to trigger the identical failure toward aarch64.
### versions
**Apache CloudStack:** 4.22.1.0 (3 management servers, all `Up`; agent
4.22.1.0)
**Hypervisor:** KVM
- libvirt 8.0.0
- QEMU 6.2.0
- Host OS Ubuntu 22.04, kernel 5.15.0-191-generic
**Storage:** NFS primary storage (`NetworkFilesystem`), single shared pool,
reachable from all
hosts of both architectures. Secondary storage also NFS.
**Network:** Advanced zone networking, single shared bridge per host (no
per-traffic-type
separation).
### The steps to reproduce the bug
```
1. Build a multi-arch zone containing at least one aarch64 cluster and one
or more x86_64
clusters, sharing primary storage.
2. Register both the aarch64 and the x86_64 KVM SystemVM templates, and
confirm both are
Ready.
3. Set system.vm.preferred.architecture = aarch64 at ZONE scope.
4. Allow the SSVM and CPVM to start normally. They land on the aarch64 host,
each with a
single ROOT volume from the aarch64 template. This is correct behaviour.
5. Make every aarch64 host unschedulable — putting the only aarch64 host
into maintenance is
sufficient.
6. Destroy the SSVM and/or CPVM so that CloudStack must recreate them.
7. Observe the recreated system VM:
- the management log shows "Retrying with another template", with a
second
"Allocating Instance from Template" and a second "Adding disk object
[ROOT-<id>]"
- the VM now has TWO ROOT volumes, both device_id = 0, one per
architecture:
SELECT id, instance_id, volume_type, device_id, template_id
FROM volumes WHERE instance_id = <vm id> AND removed IS NULL;
- every start attempt fails on every host with:
XML error: target 'vda' duplicated for disk sources '...' and '...'
- the VM loops Starting -> Stopped indefinitely and never runs
Expected: the system VM is recreated from the x86_64 template with exactly
one ROOT volume and
starts on an available x86_64 host.
```
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
