GitHub user mwaag created a discussion: Webserver.service health-checks keep failing for many vrouters
Cloudstack: 4.18.2.4 (in preparation to upgrade to a newer version) Hypervisor: VMWare ESX 7.0.3 Management-Server OS: Ubuntu 24.04.3 LTS vRouter version: 4.18.1 vRouter Software version: 4.18.2.4 Hi there, we have a lot of virtual routers raising an alert because of a failing health check. The health check in question is always 'webserver.service' which keeps failing on many vrouters creating a lot of alerts : <img width="1548" height="726" alt="11_37_12-000320" src="https://github.com/user-attachments/assets/0cddd54c-c36c-40fd-9f72-3c66f771d026"; /> <img width="1469" height="630" alt="11_41_37-000321" src="https://github.com/user-attachments/assets/d80a90b6-fa2e-4992-a0b1-223b643832d1"; /> We find this in managment.log as well: 2025-12-01 21:16:52,482 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-860476b6) (logid:7813569b) Health checks failed: 1 failing checks on router r-178445-VM / 826d1cc5-3a3e-486c-9731-5ea140d9f18c. Checking failed health checks to see if router needs recreate We usually find nothing in cloud.log on the vrouter, tried to grep for 'health' 'fail' 'check' 'apache2' 'webserver', zero matches. Systemctl-output: root@r-178445-VM:~# systemctl is-active apache2 failed root@r-178445-VM:~# systemctl status apache2 ● apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2025-12-01 20:21:10 UTC; 34s ago Docs: https://httpd.apache.org/docs/2.4/ Process: 115818 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE) CPU: 11ms Dec 01 20:21:10 r-178445-VM systemd[1]: Starting The Apache HTTP Server... Dec 01 20:21:10 r-178445-VM apachectl[115821]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Dec 01 20:21:10 r-178445-VM apachectl[115821]: no listening sockets available, shutting down Dec 01 20:21:10 r-178445-VM apachectl[115821]: AH00015: Unable to open logs Dec 01 20:21:10 r-178445-VM apachectl[115818]: Action 'start' failed. Dec 01 20:21:10 r-178445-VM apachectl[115818]: The Apache error log may have more information. Dec 01 20:21:10 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 01 20:21:10 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 01 20:21:10 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server. apache2-journal: Dec 01 20:18:11 r-178445-VM systemd[1]: Starting The Apache HTTP Server... Dec 01 20:18:11 r-178445-VM apachectl[115680]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Dec 01 20:18:11 r-178445-VM apachectl[115680]: no listening sockets available, shutting down Dec 01 20:18:11 r-178445-VM apachectl[115680]: AH00015: Unable to open logs Dec 01 20:18:11 r-178445-VM apachectl[115677]: Action 'start' failed. Dec 01 20:18:11 r-178445-VM apachectl[115677]: The Apache error log may have more information. Dec 01 20:18:11 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 01 20:18:11 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 01 20:18:11 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server. Dec 01 20:21:06 r-178445-VM systemd[1]: Starting The Apache HTTP Server... Dec 01 20:21:06 r-178445-VM apachectl[115773]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Dec 01 20:21:06 r-178445-VM apachectl[115773]: no listening sockets available, shutting down Dec 01 20:21:06 r-178445-VM apachectl[115773]: AH00015: Unable to open logs Dec 01 20:21:06 r-178445-VM apachectl[115770]: Action 'start' failed. Dec 01 20:21:06 r-178445-VM apachectl[115770]: The Apache error log may have more information. Dec 01 20:21:06 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 01 20:21:06 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 01 20:21:06 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server. Dec 01 20:21:07 r-178445-VM systemd[1]: Starting The Apache HTTP Server... Dec 01 20:21:07 r-178445-VM apachectl[115785]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Dec 01 20:21:07 r-178445-VM apachectl[115785]: no listening sockets available, shutting down Dec 01 20:21:07 r-178445-VM apachectl[115785]: AH00015: Unable to open logs Dec 01 20:21:07 r-178445-VM apachectl[115782]: Action 'start' failed. Dec 01 20:21:07 r-178445-VM apachectl[115782]: The Apache error log may have more information. Dec 01 20:21:07 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 01 20:21:07 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 01 20:21:07 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server. Dec 01 20:21:08 r-178445-VM systemd[1]: Starting The Apache HTTP Server... Dec 01 20:21:08 r-178445-VM apachectl[115797]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Dec 01 20:21:08 r-178445-VM apachectl[115797]: no listening sockets available, shutting down Dec 01 20:21:08 r-178445-VM apachectl[115797]: AH00015: Unable to open logs Dec 01 20:21:08 r-178445-VM apachectl[115794]: Action 'start' failed. Dec 01 20:21:08 r-178445-VM apachectl[115794]: The Apache error log may have more information. Dec 01 20:21:08 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 01 20:21:08 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 01 20:21:08 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server. Dec 01 20:21:09 r-178445-VM systemd[1]: Starting The Apache HTTP Server... Dec 01 20:21:09 r-178445-VM apachectl[115809]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Dec 01 20:21:09 r-178445-VM apachectl[115809]: no listening sockets available, shutting down Dec 01 20:21:09 r-178445-VM apachectl[115809]: AH00015: Unable to open logs Dec 01 20:21:09 r-178445-VM apachectl[115806]: Action 'start' failed. Dec 01 20:21:09 r-178445-VM apachectl[115806]: The Apache error log may have more information. Dec 01 20:21:09 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 01 20:21:09 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 01 20:21:09 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server. Dec 01 20:21:10 r-178445-VM systemd[1]: Starting The Apache HTTP Server... Dec 01 20:21:10 r-178445-VM apachectl[115821]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Dec 01 20:21:10 r-178445-VM apachectl[115821]: no listening sockets available, shutting down Dec 01 20:21:10 r-178445-VM apachectl[115821]: AH00015: Unable to open logs Dec 01 20:21:10 r-178445-VM apachectl[115818]: Action 'start' failed. Dec 01 20:21:10 r-178445-VM apachectl[115818]: The Apache error log may have more information. Dec 01 20:21:10 r-178445-VM systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 01 20:21:10 r-178445-VM systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 01 20:21:10 r-178445-VM systemd[1]: Failed to start The Apache HTTP Server. Syntax seems to be ok: root@r-178445-VM:~# apache2ctl configtest AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Syntax OK Any other processes bound to port 80/443? root@r-178445-VM:# ss -tulpn | grep -E ':80|:443' (no output) root@r-178445-VM:# lsof -nP -iTCP:80 -sTCP:LISTEN (no output) Checking rights on Log-dir: root@r-178445-VM:# ls -ld /var/log/apache2 drwxr-x--- 2 root adm 4096 Mar 18 2025 /var/log/apache2 root@r-178445-VM:# test -w /var/log/apache2 && echo "log-dir writable" || echo "log-dir NOT writable" log-dir writable root@r-178445-VM:# uptime 20:31:29 up 265 days, 5:05, 1 user, load average: 0.00, 0.00, 0.00 Currently we have 52 of 398 vRouters in total showing this symptom. We tried recreate one vrouter with vpc-restart with cleanup, but unfortunately with the same result. Can we do some more troubleshooting? Unfortunately this keeps us away from filling alert.email.addresses because the traffic is just to high as every router send an alarm every 10 minutes or so. Any hints, how we can further troubleshoot / eleminate this issues? GitHub link: https://github.com/apache/cloudstack/discussions/12170 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
