GitHub user sbrueseke added a comment to the discussion: Health checks failed
on VR
We did some deeper investigation and it looks like this has something to do
with our backups (volume-snapshots). It looks like one of our nfs-servers
(secondary storage) has some cpu stalls in the same timeframe when health
checks are happening. We do not have any idea how this is related. What we also
are able to see in the metrics of all instances, system vms and virtual routers
is, that iops are going up.
Our primary storage is not on nfs and not related. But we do use sds which uses
network (different nic), too. Is it possible that the amount of backups is too
mich for the tcpstack on the kvm hosts?
This I found in the agent log:
2025-01-31 02:01:19,168 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null)
(logid:) Seq 19-7793760630141465558: Processing: { Ans: , MgmtId:
165989263229575, via: 19, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success:
Creating file in VR, with ip: 169.254.88.32, file:
monitor_service.json.772738ed-480e-456f-96d9-f2deb211485b","null - success:
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
","null - success: Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
GitHub link:
https://github.com/apache/cloudstack/discussions/10249#discussioncomment-12015892
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscr...@cloudstack.apache.org
