bhouse-nexthop commented on PR #14139:
URL: https://github.com/apache/cloudstack/pull/14139#issuecomment-6733519808
### Test report: parallel tier bring-up on a redundant VPC
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs. One management server.
- Three KVM hosts on Ubuntu 24.04, nested virtualisation; Ceph RBD primary
storage.
- System VM template 4.22.0.
- A VPC on the default *Redundant VPC offering*. Its two routers ran on
different hosts.
**Method.** I ran three rounds. Each round:
- created 8 tiers one after another (allocated, not yet implemented);
- deployed one cirros VM into each tier, all 8 deploys at once;
- waited 60 s, then checked the routers' NICs in the DB, `ip -br a` and
`keepalived.conf` on both routers, and `redundantstate`;
- destroyed the VMs and tiers.
After the three rounds I kept two tiers with a VM each and ran `restartVPC
cleanup=true`.
**1. Duplicate device ids: none**
This query returned no rows after every round, before the restart, and after
it:
```sql
SELECT n.instance_id, n.device_id, COUNT(*) FROM nics n JOIN vm_instance v
ON v.id=n.instance_id
WHERE n.removed IS NULL AND v.type='DomainRouter' AND v.vpc_id=<rvpc> GROUP
BY 1,2 HAVING COUNT(*)>1;
```
No two guest NICs shared an `ethN` on either router.
**2. Recovery with `restartVPC cleanup=true`**
- The new routers came up one PRIMARY and one BACKUP.
- Their `virtual_ipaddress` sets were identical.
- The two `keepalived.conf` files differ only in `router_id` (each router's
own name). That line is expected to differ, so "byte-identical" in the
description is true apart from that line.
So the corruption this PR targets did not appear: two networks on one
`ethN`, which persists until a restart.
**3. A separate failure mode, not fixed by this PR**
Not every parallel deploy succeeded:
| Round | Deploys Running | `redundantstate` | VIP sets |
|---|---|---|---|
| 1 | 5 / 8 | PRIMARY + PRIMARY | differ by one tier gateway |
| 2 | 4 / 8 | PRIMARY + PRIMARY | differ by one tier gateway |
| 3 | 3 / 8 | PRIMARY + BACKUP | identical |
All 12 failures have the same cause:
```
Failed to re-program the network as a part of network Network {"name":
"r1t5", ...} implement due to aggregated commands execution failure!
```
The agent of the router's host logged the answer to the
`AggregationControlCommand` `Finish`:
```
java.lang.NullPointerException: Cannot invoke "java.util.Queue.clear()"
because "queue" is null
```
`VirtualRoutingResource` keeps one aggregation queue per router name, in a
plain `HashMap`. So when several networks are implemented on the same router at
once, their aggregations share and overwrite one queue:
- A second `Start` replaces the first job's queue, so the commands the first
job had already queued are dropped.
- The first `Finish` applies whatever is in the queue and removes it.
- The second `Finish` finds no queue and throws the NPE above. Its deploy
fails.
So a deploy can report success while one router never got its tier's config.
That explains the VIP sets in rounds 1 and 2. There, the missing gateway
belonged to a tier whose deploy was Running. I think this is the "partial
application" mentioned under *How Has This Been Tested*. It comes from the
agent, not from storage. It needs its own fix: serialise aggregated applies per
router, or key the aggregation by job rather than by router name.
**Result:** pass for this PR's change.
- Zero duplicate device ids across 24 parallel attaches and the restart.
- The restart gives one PRIMARY, one BACKUP and matching configs.
- The deploy failures and the dual PRIMARY in rounds 1 and 2 come from the
separate agent-side aggregation race above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
