bhouse-nexthop commented on PR #14131:
URL: https://github.com/apache/cloudstack/pull/14131#issuecomment-6733519502
### Test report: per-rule keepalive and timeouts, and the HAProxy fixes, on
running virtual routers
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs. This PR targets main, so no backport caveat applies.
- Three KVM hosts on Ubuntu 24.04, nested virtualisation; Ceph RBD primary
storage.
- System VM template 4.22.0 (Debian 12, HAProxy 2.6.12). The routers were
created on this build. Their `configure.py` and
`/root/health_checks/haproxy_check.py` match the build's.
- Two isolated networks:
- N1 on `DefaultIsolatedNetworkOfferingWithSourceNatService`, with
backends B1 and B2 (Ubuntu 24.04, nginx). nginx logs `X-Forwarded-For`,
`$connection` and `$connection_requests`, and sets `JSESSIONID=sess-<name>`.
- N2 on a copy of that offering with `keepaliveenabled=true`, with backend
B3.
- I moved N2's router onto N1's KVM host, so that one agent JVM generated
both routers' configs.
- The client was a host on the public network.
**1. An http rule with all three settings**
`createLoadBalancerRule ... publicport=80 keepalive=true idletimeout=600000
keepalivetimeout=15000`, with B1 and B2:
```
listen <P1>-80
bind <P1>:80
mode http
option http-keep-alive
timeout http-keep-alive 15000
timeout client 600000
timeout server 600000
balance roundrobin
server <P1>-80_0 <B2>:80 check
server <P1>-80_1 <B1>:80 check
```
- `haproxy -c` reports the file valid.
- `listLoadBalancerRules` returns `keepalive=true, idletimeout=600000,
keepalivetimeout=15000`.
- `curl -v` with four URLs on one command line: `Connected`, then `Re-using
existing connection` three times. The answers alternate B2, B1, B2, B1.
- nginx on each backend shows the client address in XFF, and two requests on
one backend connection: `xff="<client>" conn=2 reqs=1` and then `conn=2 reqs=2`.
So keep-alive and `X-Forwarded-For` work together on one rule. They could
not before.
**2. A tcp rule with only an idle timeout** (`publicport=3306
idletimeout=120000`)
```
listen <P1>-3306
bind <P1>:3306
timeout client 120000
timeout server 120000
balance roundrobin
server <P1>-3306_0 <B1>:3306 check
```
**3. Rules that set nothing** render as before:
- Port 80: `mode http`, `option httpclose`, `balance`, `server` lines. The
client gets `connection: close` on every response.
- Port 8080: `bind`, `balance`, `server` only.
- The `defaults` section keeps `option httpclose` and `timeout client/server
50000`.
**4. `no option forceclose` is gone (keepalive with LbCookie stickiness)**
- On rule 1 (per-rule keepalive), and on an N2 rule (keepalive from the
offering, which is the old trigger), I added an `LbCookie` policy.
- Both render `mode http`, `option http-keep-alive`, the `cookie ... insert`
line and the per-server cookies.
- `grep -c forceclose` is 0, and `haproxy -c` passes.
- haproxy is listening, and the haproxy cookie pins the client: four
requests with it all go to the same backend.
- For comparison, the same file with `no option forceclose` in place of
`option http-keep-alive` fails `haproxy -c`:
```
[ALERT] config : parsing [/tmp/fc.cfg:36]: option 'forceclose' is not
supported any more since HAProxy 2.0 ...
[ALERT] config : Fatal errors found in configuration.
```
**5. AppCookie renders as a stick table.**
Since #13315 (merged 10-21; this PR's branch does not include it yet),
`createLBStickinessPolicy methodname=AppCookie` is refused for Virtual Router
networks:
```
AppCookie based stickiness (HAProxy 'appsession') is not supported on a
Virtual Router, as this directive was removed in HAProxy 1.6
```
So the generator path is reached only by an AppCookie policy stored before
an upgrade. To test that case, I inserted one
(`cookie-name=JSESSIONID&length=52&holdtime=3h`) into
`load_balancer_stickiness_policies` for a port 80 rule. Then I regenerated the
config by editing another rule on the network.
```
stick-table type string len 52 size 10k expire 3h
stick store-response res.cook(JSESSIONID)
stick match req.cook(JSESSIONID)
```
- There is no `appsession`, and `haproxy -c` passes.
- Six requests with `Cookie: JSESSIONID=sess-B1` all go to B1, and six with
`sess-B2` all go to B2. Without a cookie they alternate.
- `show table` holds both keys.
- The same file with `appsession JSESSIONID len 52 timeout 3h` fails
`haproxy -c` with `'appsession' is not supported anymore since HAProxy 1.6`.
The PR description says AppCookie "is reachable from the API". That is no
longer true on main because of #13315. The fix still matters for policies
stored before that change.
**6. No leak between configs from one agent**
- I regenerated N2's config (keepalive offering: its `defaults` has `no
option httpclose`), and then N1's config, on the same agent.
- N1's `defaults` still has `option httpclose`, and its port 80 rule without
settings still has `option httpclose`.
- The 600000 and 120000 rule timeouts never reach either `defaults`, which
stays at 50000.
**7. Health check**
- `getRouterHealthCheckResults performfreshchecks=true`: `haproxy_check.py`
reports `All checks pass`.
- I changed `timeout client 45000` to `45001` by hand in the rule's listen
section. The next fresh check failed with:
```
Expected 'timeout client 45000' in listen <P1>-80 but found 45001
```
- A real re-apply (an algorithm change on another rule) rewrote the file,
and the check passed again.
**8. Back to defaults**
- `updateLoadBalancerRule cleanupconnectionsettings=true` returns success,
and the three fields come back empty (retested on the fixed build, see below).
The rule renders `mode http` / `option httpclose`, with the LbCookie lines kept.
- With `cleanupconnectionsettings=true keepalive=true`, the update returns
success and only keepalive remains: `option http-keep-alive`, no timeouts.
Connection reuse works again.
**9. Validation**
`idletimeout=-1` is refused on create (`idletimeout must be 0 or greater,
got [-1]. 0 means no timeout.`) and on update. `keepalivetimeout=-1` is refused
on update with the same wording.
**10. Rollback when the router update fails**
I reran this step on the build with both update fixes, so that only an
unreachable router can make the update fail.
- `updateLoadBalancerRule idletimeout=30000 keepalivetimeout=20000` returned
success.
- I stopped the agent of the router's host, so the update could not reach
the router.
- `updateLoadBalancerRule idletimeout=45000 keepalivetimeout=25000` then
failed with `Failed to update load balancer rule`.
- `listLoadBalancerRules` and `firewall_rule_details` still held the old
30000 / 20000, and the management server logged `LB Rollback rule`.
- With the agent back, the same update returned success, and the router
shows `timeout client/server 45000` and `timeout http-keep-alive 25000`.
The test plan's first idea for this step was `chattr +i
/etc/haproxy/haproxy.cfg` on the router. That does not fail the update: the
router logs `Could not copy /etc/haproxy/haproxy.cfg.new to
/etc/haproxy/haproxy.cfg`, reloads the old file and reports success. So the
management server has nothing to roll back. That is router-side behaviour,
unchanged by this PR.
**Result:** pass.
- Every generated config passes `haproxy -c` and is live.
- Keep-alive and XFF work together.
- Rules without settings render as before.
- There is no leak between configs.
- The health check notices drift.
- Reset to defaults and rollback behave as described.
**Two update bugs the lab found, both fixed in the PR**
The first run found two ways `updateLoadBalancerRule` returned job result
530 although the change had been applied:
- A rule created through the API without `protocol` has a NULL
`lb_protocol`, and the SSL check after the apply threw an NPE on it. The PR now
compares null-safely.
- An update that changes only connection settings failed with `Failed to
update load balancer rule`. These settings live in `firewall_rule_details`, so
no column of the rule changed and the DAO update reported no row updated. The
PR now treats a settings change as a successful update.
I retested on a build with both fixes, on two rules: one created without
`protocol`, and one with `protocol=tcp` as the UI sends it. Every update below
returned success. `listLoadBalancerRules` showed the new values, the router's
listen section had them, and `haproxy -c` passed after each.
| Rule | Update | Router listen section after it |
|---|---|---|
| no protocol | `keepalive=true idletimeout=600000 keepalivetimeout=15000` |
`option http-keep-alive`, `timeout http-keep-alive 15000`, `timeout
client/server 600000` |
| no protocol | `idletimeout=120000` | `timeout client 120000`, keep-alive
timeout kept at 15000 |
| no protocol | `keepalivetimeout=20000` | `timeout http-keep-alive 20000` |
| no protocol | `algorithm=leastconn` | `balance leastconn`, timeouts kept |
| no protocol | `cleanupconnectionsettings=true` | no timeouts, no
`http-keep-alive` |
| no protocol | `cleanupconnectionsettings=true keepalive=true` | `option
http-keep-alive` only |
| no protocol | `idletimeout=45000` | `timeout client 45000`, keep-alive
kept |
| `protocol=tcp` | `idletimeout=90000` | `timeout client/server 90000` |
| `protocol=tcp` | `keepalive=true` | timeouts kept |
| `protocol=tcp` | `keepalivetimeout=12000` | timeouts kept |
| `protocol=tcp` | `cleanupconnectionsettings=true` | no timeouts |
| `protocol=tcp` | `algorithm=source idletimeout=70000` | `balance source`,
`timeout client/server 70000` |
- The management server log has no `getLbProtocol` / `NullPointerException`
line over the retest.
- Requests through the rule still alternate between both backends.
- An update that names only the rule id and changes nothing still fails with
`Failed to update load balancer rule`, as on stock. That is the only `Failed to
update` line in the log.
**Not covered**
- The UI edit form (blank = "Use default", clearing a field sends
`cleanupconnectionsettings`). I had no browser on the lab network.
- SSL offload rules, and the `tcp-proxy` protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
