bhouse-nexthop opened a new issue, #14639:
URL: https://github.com/apache/cloudstack/issues/14639
### problem
Command aggregation on a virtual router isn't safe when two jobs aggregate
commands for the same router at the same time. Parallel tier deploys in one VPC
do this. Commands get dropped, and jobs fail with a NullPointerException.
`VirtualRoutingResource`
(`core/.../agent/resource/virtualnetwork/VirtualRoutingResource.java`) keeps
one aggregation queue per router name in a `HashMap`:
- `AggregationControlCommand` Start puts a new queue for the router. A
second Start while one exists replaces it, so the commands the first job has
queued so far are never applied. Their answers were already successful, because
a queued command is answered with success at once.
- Every other `NetworkElementCommand` for that router goes into whichever
queue is current.
- Finish applies the current queue and removes it. The first job's Finish
applies the second job's queue, and the second job's Finish then finds no
queue. The loop and then `queue.clear()` in the `finally` block throw
`java.lang.NullPointerException: Cannot invoke "java.util.Queue.clear()"
because "queue" is null`.
The result is that one job fails, and another reports success although some
of its configuration never reached the router.
The per-router lock map (`_vrLockMap`) and the aggregation map are also
plain `HashMap`s changed from parallel agent request threads. The lock is
created with a check-then-put, so two threads can get different locks for the
same router.
This may be one cause of #11710: with redundant routers, the two routers of
a VPC can end up with different configurations.
### versions
4.22 (the code is the same on main and 4.20).
### The steps to reproduce the bug
1. Create a VPC with redundant routers on KVM.
2. Create several tiers and deploy VMs into them in parallel, for example 24
deploys across the tiers at once.
3. Some deploys fail. The management server logs `Failed to re-program the
network as a part of network ... implement due to aggregated commands execution
failure!`, and the router's host agent answers an `AggregationControlCommand`
Finish with `java.lang.NullPointerException: Cannot invoke
"java.util.Queue.clear()" because "queue" is null`.
4. In our lab run, 12 of 24 parallel tier deploys failed. In two of three
rounds, the two routers ended up with different keepalived VIP sets, and both
were PRIMARY.
### What to do about it?
- On the agent:
- do not replace an existing queue on Start;
- do not fail with an NPE on a Finish whose queue was already applied;
- make the maps thread-safe.
- A complete fix would serialise aggregated applies per router on the
management server, or tag each aggregation with an id. That is a larger change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
