bhouse-nexthop commented on PR #14640:
URL: https://github.com/apache/cloudstack/pull/14640#issuecomment-6752445075
### Test report: parallel tier deploys on a redundant VPC
**Summary:** I repeated the parallel tier bring-up that found #14639, for 40
deploys in 5 rounds. Every deploy succeeded. The concurrent aggregation the bug
is about happened 27 times on one router. Each time, the second Start joined
the queue and the late Finish reported the earlier apply's result. There was no
NPE. After every round both routers had all 8 tiers, identical keepalived VIP
sets, and exactly one PRIMARY. The stock run failed half its deploys.
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs. The management server and all three agents were upgraded in place to
this build. The change is in the agent (`VirtualRoutingResource`, in `core`).
- The PR targets 4.22; it wasn't run on a 4.22 build. The code it changes is
the same on both branches: 4.22's `VirtualRoutingResource` still does `assert
(!_vrAggregateCommandsSet.containsKey(routerName))` on Start, then replaces the
queue, in a plain `HashMap`.
- Three KVM hosts, Ceph RBD primary storage, one management server. System
VM template 4.22.0.
- A new VPC on the default *Redundant VPC offering*. Its two routers ran on
different hosts, router 1 on host 3 and router 2 on host 1.
- `vpc.max.networks` was already 12 and was not changed.
**Method**
- Rounds 1-3 have the same shape as the stock run. Each round creates 8
tiers one after another (allocated, not yet implemented), then deploys one
cirros VM into each tier, all 8 deploys at once. The tiers are implemented by
those deploys, so 8 jobs program both routers at the same time.
- Rounds 4-5 run 8 workers at once. Each worker creates its tier and then
deploys a VM into it, so the `createNetwork` calls are also concurrent.
- After each round I waited 60 s and checked:
- every deploy's result;
- the management server log for `aggregated commands execution failure`;
- the agent logs on all three hosts for `NullPointerException`,
`Queue.clear` and any exception in `VirtualRoutingResource`, and for the PR's
two new warnings;
- `listRouters` `redundantstate`, and `checkrouter.sh` on each router;
- on both routers: the keepalived `virtual_ipaddress` set; for each tier,
the gateway VIP in that set, an interface holding the router's own address in
the tier (taken from `nics`), and that interface's `ACL_INBOUND_ethN` chain in
`iptables-save`.
- Then the round's VMs and tiers were destroyed.
**Results**
| Round | Deploys Running | `aggregated commands execution failure` | Agent
NPE / VRR exceptions | Joined Starts / Finishes with no queue | Routers | VIP
sets | Every tier on both routers |
|---|---|---|---|---|---|---|---|
| 1 | 8 / 8 | 0 | 0 / 0 | 3 / 3 | PRIMARY + BACKUP | identical (8) | yes |
| 2 | 8 / 8 | 0 | 0 / 0 | 6 / 6 | BACKUP + PRIMARY | identical (8) | yes |
| 3 | 8 / 8 | 0 | 0 / 0 | 6 / 6 | PRIMARY + BACKUP | identical (8) | yes |
| 4 (concurrent create) | 8 / 8 | 0 | 0 / 0 | 6 / 6 | PRIMARY + BACKUP |
identical (8) | yes |
| 5 (concurrent create) | 8 / 8 | 0 | 0 / 0 | 6 / 6 | PRIMARY + BACKUP |
identical (8) | yes |
| **Total** | **40 / 40** | **0** | **0** | **27 / 27** | one PRIMARY every
round | | |
**Before this PR, with the same shape (the earlier stock run, 3 rounds of
8):**
- 12 of 24 deploys failed with `aggregated commands execution failure`.
- The agents logged `Cannot invoke "java.util.Queue.clear()" because "queue"
is null` 34 times.
- In two of the three rounds the routers' VIP sets differed by one tier
gateway, and both routers were PRIMARY.
**The race was hit, and handled**
- All 27 overlaps were on router 2. Router 1 never had two aggregations at
once in these runs. Agent log on router 2's host, round 2:
```
20:50:07,611 WARN VirtualRoutingResource Command aggregation is already
in progress for router <router 2>, joining it
20:50:28,515 WARN VirtualRoutingResource Command aggregation is already
in progress for router <router 2>, joining it
20:50:28,515 WARN VirtualRoutingResource Command aggregation is already
in progress for router <router 2>, joining it
...
WARN VirtualRoutingResource No command aggregation in progress for
router <router 2>, the last aggregated apply succeeded
```
- Each join was matched by one Finish that found no queue, and every one of
those reported `the last aggregated apply succeeded`. Before this PR, each of
those Finishes would have been the `Queue.clear()` NPE and a failed deploy.
- The commands of the joined jobs were applied. In every round router 2 had
all 8 tiers: the gateway VIP, its own address on an interface, and that
interface's ACL chain. Round 5, router 1, as an example (router 2 is the same):
```
10.30.51.1/24 ... 10.30.58.1/24 8 gateway VIPs, the
same set on both routers
ACL_INBOUND_eth2 ... ACL_INBOUND_eth9 one chain per tier
interface
```
- No agent logged any exception in these rounds.
**Note on round 2:** the routers swapped roles: router 2 was PRIMARY and
router 1 BACKUP. There was still exactly one PRIMARY, and both routers had the
same configuration. From round 3 on, router 1 was PRIMARY again.
**Result:** pass. Concurrent aggregations on one router no longer drop
commands or fail jobs. 40 of 40 parallel deploys succeeded. Before the PR, 12
of 24 failed with the same shape. Both routers end each round with every tier
configured, the same VIPs and one PRIMARY.
**Not covered**
- A failed aggregated apply while another job has joined it. Then both
Finishes should report the failure; all applies in this run succeeded. That
case is covered by the unit tests.
- A router-start batch that stops between Start and Finish, leaving a queue
behind. The PR lists this as a limit.
- Only one of the two routers saw overlapping aggregations. The other
router's tiers were applied without overlap, so they show the normal path only.
- Non-redundant VPCs and isolated networks: they use the same code, but I
didn't run them in parallel here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
