richevanscybermyte opened a new issue, #13735: URL: https://github.com/apache/cloudstack/issues/13735
##### ISSUE TYPE * Bug Report ##### COMPONENT NAME ~~~ SystemVM / Virtual Router health checks (systemvm/debian/root/health_checks/iptables_check.py) ~~~ ##### CLOUDSTACK VERSION ~~~ 4.22.0.0 and 4.22.1.0 (management), systemvm template 4.22.0. The affected code is unchanged on current main: https://github.com/apache/cloudstack/blob/main/systemvm/debian/root/health_checks/iptables_check.py ~~~ ##### CONFIGURATION Redundant VPC virtual routers, KVM. Advanced zone. Default health check settings (`router.health.checks.enabled=true`, `router.health.checks.advanced.interval=10`). The affected VPC has ~200 port-forwarding/LB rules (iptables-save output: 77,029 bytes). ##### OS / ENVIRONMENT KVM hosts on Rocky Linux 9.5/9.8 (qemu-kvm 9.0/10.1 — reproduced on both). Debian-based system VM template, 512 MB RAM routers. ##### SUMMARY `iptables_check.py` runs the fetch command with `Popen(cmd, shell=True, stdout=PIPE)` and then calls `pout.wait()` **before** reading the pipe: ```python fetchIpTableEntriesCmd = "iptables-save | grep " + destIp pout = Popen(fetchIpTableEntriesCmd, shell=True, stdout=PIPE) if pout.wait() != 0: # <-- deadlocks: wait() before the pipe is drained ... ipTablesMatchingEntries = pout.communicate()[0].decode().strip().split('\n') ``` This is the deadlock explicitly warned about in the Python `subprocess` documentation: when the command's output exceeds the OS pipe capacity (64 KiB on Linux), the child blocks writing to the full pipe and the parent blocks forever in `wait()`. On routers with large rule sets the check therefore hangs **on every run**. Because the advanced health check fires every `router.health.checks.advanced.interval` (10 min default), a new stuck process chain (`monitorServices.py advanced` -> `sh` -> `iptables_check.py`) accumulates every interval — ~13 MB RSS each, roughly 85–90 MB/h. On a 512 MB router, RAM and swap are exhausted after ~8.5–9 h of uptime. At that point keepalived's `heartbeat` track script can no longer fork within its timeout (`VRRP_Script(heartbeat) timed_out` -> `Entering FAULT STATE`), the router demotes, wipes `/etc/keepalived/keepalived.conf`, and the guest reboots — taking the VPC down. The impact is amplified on redundant VPCs: both routers of a pair are created together, so their exhaustion clocks are synchronized and **both routers fail within minutes of each other**, defeating the redundancy. We experienced a full-VPC outage roughly every 9 hours until the check was excluded. ##### STEPS TO REPRODUCE ~~~ 1. Create a VPC with enough port-forwarding/LB rules that the health check's "iptables-save | grep <ip>" output exceeds 64 KiB inside the router (our affected router: iptables-save = 77,029 bytes total). 2. Leave router.health.checks.enabled=true with default advanced interval (10 min). 3. Inside the router, watch: ps -eo pid,ppid,etimes,rss,args | grep -E "monitorServices|iptables_check" 4. A new stuck chain appears every 600 s and never exits (parent and child both in wchan do_wait; the shell child is blocked writing to the full pipe). 5. Watch "free -k": used memory climbs linearly; after ~8.5-9 h the router enters keepalived FAULT, demotes/wipes, and reboots. ~~~ Observed accumulation (etimes exactly 600 s apart, none ever exiting): ~~~ 90114 7551 /usr/bin/python /root/monitorServices.py advanced 90136 7551 /bin/sh -c ./health_checks/iptables_check.py advanced 90137 7551 /usr/bin/python ./health_checks/iptables_check.py advanced 92506 6952 /usr/bin/python /root/monitorServices.py advanced ... 6351 (repeats every 600s) ... 5751 ... 5152 ... 4552 ~~~ Control group: three routers in other VPCs on the same platform with small rule sets (iptables-save 1.2–9.3 KB) run the identical check to completion in <1 s and have 152-day uptimes. The only variable is iptables output size vs the 64 KiB pipe buffer. ##### EXPECTED RESULTS ~~~ The health check completes (or fails fast) regardless of iptables rule-set size. ~~~ ##### ACTUAL RESULTS ~~~ The check hangs permanently on every run once the piped output exceeds 64 KiB. Stuck process chains accumulate every advanced-check interval until the router exhausts RAM+swap (~8.5-9 h on a 512 MB router), keepalived's track script starves, the router demotes/wipes its keepalived config and reboots. Redundant pairs fail near-simultaneously because their clocks are synchronized at creation, causing a recurring full-VPC outage. ~~~ ##### WORKAROUND Set `router.health.checks.to.exclude = iptables_check.py` (global setting, dynamic). Note: in our testing the value had to be the script filename **with** the `.py` extension — the bare name `iptables_check` was silently ignored and the check kept running (possibly a second, minor bug in how the exclusion list is matched). Accumulated stuck processes must be killed manually inside affected routers. ##### SUGGESTED FIX Do not call `wait()` while `stdout=PIPE` is unread. Use `communicate()` first and check `returncode` afterwards, e.g.: ```python pout = Popen(fetchIpTableEntriesCmd, shell=True, stdout=PIPE) stdout, _ = pout.communicate() if pout.returncode != 0: ... ipTablesMatchingEntries = stdout.decode().strip().split('\n') ``` The same `Popen(stdout=PIPE)` + `wait()` pattern exists in other health check scripts (e.g. `gateways_check.py`, `cpu_usage_check.py`). Their outputs are normally tiny so they do not currently hang, but they share the same latent hazard and may be worth fixing in the same pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
