bhouse-nexthop commented on PR #14636:
URL: https://github.com/apache/cloudstack/pull/14636#issuecomment-6752445886
### Test report: load balancer change when the router can't replace
haproxy.cfg
**Summary:** with `/etc/haproxy/haproxy.cfg` immutable on the router,
`updateLoadBalancerRule` now fails, the rule keeps its old value, and HAProxy
keeps running on the old config. After `chattr -i`, the same update succeeds
and is applied.
**Environment**
- A CloudStack 24.0 (main) build with this PR, alongside some of my other
open PRs. Three KVM hosts, one management server, system VM template 4.22.0.
- The PR targets 4.22; it wasn't run on a 4.22 build. 4.22's
`CsLoadBalancer.process()` installs the file with `CsHelper.copy()`, as main
did before this change.
- A new isolated network with source NAT, its virtual router, and one VM.
The router was created after the upgrade, so it runs this build's scripts:
`CsLoadBalancer.py` on the router contains the new `shutil.copy2` call.
- A load balancer rule on a second public IP: port 80, `roundrobin`, with
the VM assigned.
**Steps and results**
| Step | Result |
|---|---|
| A: `chattr +i /etc/haproxy/haproxy.cfg` on the router, then
`updateLoadBalancerRule algorithm=leastconn` | **fails**, 530 |
| Rule after A | `algorithm: roundrobin`, Active: rolled back |
| Router after A | `haproxy.cfg` still has `balance roundrobin`; HAProxy
still running with the same PID |
| B: `chattr -i`, then the same update | succeeds |
| Rule and router after B | `algorithm: leastconn`; `haproxy.cfg` has
`balance leastconn`; HAProxy reloaded (new PID) |
**Evidence for A**
- The router's `/var/log/cloud.log`:
```
21:38:47,515 INFO Wrote edited file /etc/haproxy/haproxy.cfg.new
21:38:47,515 INFO Executing2: haproxy -c -f /etc/haproxy/haproxy.cfg.new
21:38:47,543 ERROR Could not copy /etc/haproxy/haproxy.cfg.new to
/etc/haproxy/haproxy.cfg: [Errno 1] Operation not permitted:
'/etc/haproxy/haproxy.cfg'
```
There is no reload after the error.
- Management server:
```
21:38:47,751 WARN LoadBalancingRulesManagerImpl Unable to apply the load
balancer config because resource is unavailable.
com.cloud.exception.ResourceUnavailableException: ...
```
The API answers `Failed to update load balancer rule: ...` with error code
530.
- Before this PR, the same step returned success, `listLoadBalancerRules`
showed `leastconn`, and the router kept `roundrobin`.
**Result:** pass. The management server now sees the failed install, rolls
the rule back, and the router keeps serving the old, consistent config. Once
the file is writable again, the change goes through.
**Side note, not from this PR:** the failed `updateLoadBalancerRule` is
logged at ERROR as `Unexpected exception while executing ...
UpdateLoadBalancerRuleCmd` (the `CloudRuntimeException` reaches the
dispatcher). That is how this command reports any failed apply.
**Not covered**
- A full filesystem or a read-only mount, the other causes the PR mentions:
same code path, but not reproduced.
- A VPC router: the same `CsLoadBalancer.py` runs there.
- Adding a new rule, rather than changing one, while the file is immutable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
