bradh352 commented on issue #11710: URL: https://github.com/apache/cloudstack/issues/11710#issuecomment-5624542014
Root caused, with a reliable reproduction on 4.22.1.0 (KVM, two management servers). Creating 8 tiers + 8 VMs in parallel through the API corrupted the routers on the first attempt. **Sequence:** 1. Each concurrently-deploying VM implements its tier inside its own `VmWorkStart` work job. 2. Tier implement calls `VpcVirtualRouterElement.implement()` → `addVpcRouterToGuestNetwork(router, network)` → `VirtualMachineManagerImpl.addVmToNetwork(router, ...)`. 3. `addVmToNetwork()` picks queue-vs-direct with `jobContext.isJobDispatchedBy(VM_WORK_JOB_DISPATCHER)`. That only asks "am I inside *any* VM work job", not "a work job *for this router*". Inside a user VM's work job it is true, so the router NIC is added via `orchestrateAddVmToNetwork()` directly — the router's own job queue is never used. (The placeholder work job created there is expunged in a `finally` and serializes nothing.) 4. So N concurrent tier implements = N concurrent unserialized `getFreeDeviceId()` calls on the same router, which is read-then-pick with no lock → the same device id is handed to multiple NICs. 5. The colliding `PlugNicCommand`s partially fail, cleanup/retries churn, and the two routers end up with different device→network maps, leftover `"ethN": []` slots in `guest_network.json`, and keepalived `virtual_ipaddress` blocks that disagree in content → VRRP breaks → FAULT / dual PRIMARY. **Evidence — `nics` rows for one router from the repro (all created live):** | nic id | network | device_id | created | |---|---|---|---| | 3499 | 300 | **2** | 19:33:40 | | 3502 | 301 | **2** | 19:33:40 | | 3520 | 302 | **3** | 19:33:46 | | 3523 | 303 | **3** | 19:33:46 | | 3565 | 301 | **4** | 19:34:12 (retry of 3502) | | 3568 | 302 | **4** | 19:34:12 (retry of 3520, collided again) | - `vm_work_job` for both routers contains only `VmWorkStart` — zero `VmWorkAddVmToNetwork` rows. Every tier attach took the direct path. - Management server logs show two `Work-Job-Executor` threads (different user VMs' deploy jobs) allocating a NIC on the same router 14 ms apart, both receiving device 2; three threads 6 s later, two receiving device 3; a retry executing on the *other* management server, two receiving device 4. The race is thread-level — a single management server is enough to hit it. - End state matched my original report: empty `[]` device entries and `virtual_ipaddress` blocks that differ between the two routers. **Why the job queue doesn't protect this:** it does serialize `addVmToNetwork` when it is called from the API path — but not when it is called from inside *another VM's* work job, which is exactly how tier implement reaches it. PR with a fix to follow: take the direct path only when the current work job belongs to the same VM the NIC is being added to; otherwise dispatch through the router's own job queue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
