Hi, It's a mystery to me to find out that the dns_check.py for my deployment is
failing every 10 minutes. I tried searching Google but I cannot find anything
relevant to my scenario. Please advise.
Log snippet:
2021-10-27 15:58:51,569 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-12:null) (logid:) SeqA 6-1926377: Sending Seq 6-1926377:
{ Ans: , MgmtId: 1743772850, via: 6, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0"}}] }
2021-10-27 15:58:55,789 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-19c33859) (logid:f3458df2) Zone 1 is ready to launch
secondary storage VM
2021-10-27 15:58:55,929 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-0e2ba419) (logid:fa5ecc2a) Zone 1 is ready to launch
console proxy
2021-10-27 15:58:58,894 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) Ping from 3(LAB)
2021-10-27 15:58:58,894 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) Process host VM state
report from ping process. host: 3
2021-10-27 15:58:59,030 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-392:ctx-44d6234c) (logid:17ec443d) Seq 3-6314046677573480412:
Response Received:
2021-10-27 15:58:59,044 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) Process VM state report.
host: 3, number of records in report: 24
2021-10-27 15:58:59,044 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 98, power state: PowerOn
2021-10-27 15:58:59,045 DEBUG [c.c.a.t.Request] (DirectAgent-392:ctx-44d6234c)
(logid:17ec443d) Seq 3-6314046677573480412: Processing: { Ans: , MgmtId:
1743772850, via: 3(LAB), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":["dns_check.py"],"monitoringResults":"{"basic":{"lastRun":
{"duration": "11.6007211208", "start": "2021-10-27 07:57:01.679916", "end":
"2021-10-27 07:57:13.280637"}, "disk_space_check.py": {"lastRunDuration":
"32.3760509491", "lastUpdate": "1635321429961", "message": "Sufficient free
space is 1032 MB", "success": "true"}, "gateways_check.py": {"lastRunDuration":
"8066.79701805", "lastUpdate": "1635321421894", "message": "All 1 gateways are
reachable via ping", "success": "true"}, "ssh.service": {"lastRunDuration":
"30.956029892", "lastUpdate": "1635321421681", "message": "service is running",
"success": "true"}, "dhcp.service": {"lastRunDuration": "9.65595245361",
"lastUpdate": "1635321421681", "message": "servi
ce is running", "success": "true"}, "memory_usage_check.py":
{"lastRunDuration": "54.6729564667", "lastUpdate": "1635321421773", "message":
"Memory Usage within limits with current at 55.1192%", "success": "true"},
"cpu_usage_check.py": {"lastRunDuration": "3215.58308601", "lastUpdate":
"1635321430064", "message": "CPU Usage within limits with current at 1.0%",
"success": "true"}, "router_version_check.py": {"lastRunDuration":
"33.4870815277", "lastUpdate": "1635321421740", "message": "Template and
scripts version match successful", "success": "true"}, "webserver.service":
{"lastRunDuration": "20.1549530029", "lastUpdate": "1635321421681", "message":
"service is running", "success": "true"}},"advanced":{"lastRun": {"duration":
"0.335355997086", "start": "2021-10-27 07:50:01.912077", "end": "2021-10-27
07:50:02.247433"}, "haproxy_check.py": {"lastRunDuration": "29.464006424",
"lastUpdate": "1635321002030", "message": "No data provided to check,
skipping", "success": "true"}, "dhcp_ch
eck.py": {"lastRunDuration": "32.4671268463", "lastUpdate": "1635321001997",
"message": "All 40 VMs are present in dhcphosts.txt", "success": "true"},
"dns_check.py": {"lastRunDuration": "35.9070301056", "lastUpdate":
"1635321002164", "message": "Missing entries for VMs in /etc/hosts
-\n103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.225
VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421", "success": "false"},
"iptables_check.py": {"lastRunDuration": "41.7759418488", "lastUpdate":
"1635321002122", "message": "No portforwarding rules provided to check,
skipping", "success":
"true"}}}","result":"true","details":"{"basic":{"lastRun": {"duration":
"11.6007211208", "start": "2021-10-27 07:57:01.679916", "end": "2021-10-27
07:57:13.280637"}, "disk_space_check.py": {"lastRunDuration": "32.3760509491",
"lastUpdate": "1635321429961", "message": "Sufficient free space is
1032 MB", "success": "true"}, "gateways_check.py": {"lastRunDuration":
"8066.79701805", "lastUpdate": "1635321421894", "message": "All 1 gateways are
reachable via ping", "success": "true"}, "ssh.service": {"lastRunDuration":
"30.956029892", "lastUpdate": "1635321421681", "message": "service is running",
"success": "true"}, "dhcp.service": {"lastRunDuration": "9.65595245361",
"lastUpdate": "1635321421681", "message": "service is running", "success":
"true"}, "memory_usage_check.py": {"lastRunDuration": "54.6729564667",
"lastUpdate": "1635321421773", "message": "Memory Usage within limits with
current at 55.1192%", "success": "true"}, "cpu_usage_check.py":
{"lastRunDuration": "3215.58308601", "lastUpdate": "1635321430064", "message":
"CPU Usage within limits with current at 1.0%", "success": "true"},
"router_version_check.py": {"lastRunDuration": "33.4870815277", "lastUpdate":
"1635321421740", "message": "Template and scripts version match successful",
"success": "true"}, "webserver
.service": {"lastRunDuration": "20.1549530029", "lastUpdate": "1635321421681",
"message": "service is running", "success": "true"}},"advanced":{"lastRun":
{"duration": "0.335355997086", "start": "2021-10-27 07:50:01.912077", "end":
"2021-10-27 07:50:02.247433"}, "haproxy_check.py": {"lastRunDuration":
"29.464006424", "lastUpdate": "1635321002030", "message": "No data provided to
check, skipping", "success": "true"}, "dhcp_check.py": {"lastRunDuration":
"32.4671268463", "lastUpdate": "1635321001997", "message": "All 40 VMs are
present in dhcphosts.txt", "success": "true"}, "dns_check.py":
{"lastRunDuration": "35.9070301056", "lastUpdate": "1635321002164", "message":
"Missing entries for VMs in /etc/hosts -\n103.239.221.225
VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.225
VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421", "success": "false"},
"iptables_check.py": {"lastRunD
uration": "41.7759418488", "lastUpdate": "1635321002122", "message": "No
portforwarding rules provided to check, skipping", "success":
"true"}}}","wait":"0"}}] }
2021-10-27 15:58:59,045 DEBUG [c.c.a.t.Request]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Seq
3-6314046677573480412: Received: { Ans: , MgmtId: 1743772850, via: 3(LAB),
Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } }
2021-10-27 15:58:59,045 DEBUG [c.c.a.m.AgentManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Details from executing
class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand:
{"basic":{"lastRun": {"duration": "11.6007211208", "start": "2021-10-27
07:57:01.679916", "end": "2021-10-27 07:57:13.280637"}, "disk_space_check.py":
{"lastRunDuration": "32.3760509491", "lastUpdate": "1635321429961", "message":
"Sufficient free space is 1032 MB", "success": "true"}, "gateways_check.py":
{"lastRunDuration": "8066.79701805", "lastUpdate": "1635321421894", "message":
"All 1 gateways are reachable via ping", "success": "true"}, "ssh.service":
{"lastRunDuration": "30.956029892", "lastUpdate": "1635321421681", "message":
"service is running", "success": "true"}, "dhcp.service": {"lastRunDuration":
"9.65595245361", "lastUpdate": "1635321421681", "message": "service is
running", "success": "true"}, "memory_usage_check.py": {"lastRunDuration":
"54.6729564667", "lastUpdate": "16
35321421773", "message": "Memory Usage within limits with current at
55.1192%", "success": "true"}, "cpu_usage_check.py": {"lastRunDuration":
"3215.58308601", "lastUpdate": "1635321430064", "message": "CPU Usage within
limits with current at 1.0%", "success": "true"}, "router_version_check.py":
{"lastRunDuration": "33.4870815277", "lastUpdate": "1635321421740", "message":
"Template and scripts version match successful", "success": "true"},
"webserver.service": {"lastRunDuration": "20.1549530029", "lastUpdate":
"1635321421681", "message": "service is running", "success":
"true"}},"advanced":{"lastRun": {"duration": "0.335355997086", "start":
"2021-10-27 07:50:01.912077", "end": "2021-10-27 07:50:02.247433"},
"haproxy_check.py": {"lastRunDuration": "29.464006424", "lastUpdate":
"1635321002030", "message": "No data provided to check, skipping", "success":
"true"}, "dhcp_check.py": {"lastRunDuration": "32.4671268463", "lastUpdate":
"1635321001997", "message": "All 40 VMs are present in
dhcphosts.txt", "success": "true"}, "dns_check.py": {"lastRunDuration":
"35.9070301056", "lastUpdate": "1635321002164", "message": "Missing entries for
VMs in /etc/hosts -\n103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5,
103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421", "success": "false"},
"iptables_check.py": {"lastRunDuration": "41.7759418488", "lastUpdate":
"1635321002122", "message": "No portforwarding rules provided to check,
skipping", "success": "true"}}}
2021-10-27 15:58:59,052 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 2, power state: PowerOn
2021-10-27 15:58:59,059 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 613, power state: PowerOn
2021-10-27 15:58:59,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 677, power state: PowerOn
2021-10-27 15:58:59,068 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Parsing and updating DB
health check data for router: 613 with data: {"basic":{"lastRun": {"duration":
"11.6007211208", "start": "2021-10-27 07:57:01.679916", "end": "2021-10-27
07:57:13.280637"}, "disk_space_check.py": {"lastRunDuration": "32.3760509491",
"lastUpdate": "1635321429961", "message": "Sufficient free space is 1032 MB",
"success": "true"}, "gateways_check.py": {"lastRunDuration": "8066.79701805",
"lastUpdate": "1635321421894", "message": "All 1 gateways are reachable via
ping", "success": "true"}, "ssh.service": {"lastRunDuration": "30.956029892",
"lastUpdate": "1635321421681", "message": "service is running", "success":
"true"}, "dhcp.service": {"lastRunDuration": "9.65595245361", "lastUpdate":
"1635321421681", "message": "service is running", "success": "true"},
"memory_usage_check.py": {"lastRunDuration": "54.6729564667", "lastUpdate":
"1635
321421773", "message": "Memory Usage within limits with current at 55.1192%",
"success": "true"}, "cpu_usage_check.py": {"lastRunDuration": "3215.58308601",
"lastUpdate": "1635321430064", "message": "CPU Usage within limits with current
at 1.0%", "success": "true"}, "router_version_check.py": {"lastRunDuration":
"33.4870815277", "lastUpdate": "1635321421740", "message": "Template and
scripts version match successful", "success": "true"}, "webserver.service":
{"lastRunDuration": "20.1549530029", "lastUpdate": "1635321421681", "message":
"service is running", "success": "true"}},"advanced":{"lastRun": {"duration":
"0.335355997086", "start": "2021-10-27 07:50:01.912077", "end": "2021-10-27
07:50:02.247433"}, "haproxy_check.py": {"lastRunDuration": "29.464006424",
"lastUpdate": "1635321002030", "message": "No data provided to check,
skipping", "success": "true"}, "dhcp_check.py": {"lastRunDuration":
"32.4671268463", "lastUpdate": "1635321001997", "message": "All 40 VMs are
present in dh
cphosts.txt", "success": "true"}, "dns_check.py": {"lastRunDuration":
"35.9070301056", "lastUpdate": "1635321002164", "message": "Missing entries for
VMs in /etc/hosts -\n103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5,
103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421", "success": "false"},
"iptables_check.py": {"lastRunDuration": "41.7759418488", "lastUpdate":
"1635321002122", "message": "No portforwarding rules provided to check,
skipping", "success": "true"}}}
2021-10-27 15:58:59,072 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found check types
executed on VR basic, start: 2021-10-27 07:57:01.679916, end: 2021-10-27
07:57:13.280637, duration: 11.6007211208
2021-10-27 15:58:59,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 518, power state: PowerOn
2021-10-27 15:58:59,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 233, power state: PowerOn
2021-10-27 15:58:59,083 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@735fcbf9- check type:
basic,check name: disk_space_check.py, check result: true, check last update:
Wed Oct 27 15:57:09 SGT 2021, details: Sufficient free space is 1032 MB which
took running duration (ms) 32.3760509491
2021-10-27 15:58:59,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 651, power state: PowerOn
2021-10-27 15:58:59,092 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@6d56e337- check type:
basic,check name: gateways_check.py, check result: true, check last update: Wed
Oct 27 15:57:01 SGT 2021, details: All 1 gateways are reachable via ping which
took running duration (ms) 8066.79701805
2021-10-27 15:58:59,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 588, power state: PowerOn
2021-10-27 15:58:59,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 12, power state: PowerOn
2021-10-27 15:58:59,102 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@503c8360- check type:
basic,check name: ssh.service, check result: true, check last update: Wed Oct
27 15:57:01 SGT 2021, details: service is running which took running duration
(ms) 30.956029892
2021-10-27 15:58:59,105 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 589, power state: PowerOn
2021-10-27 15:58:59,112 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@37d29ee7- check type:
basic,check name: dhcp.service, check result: true, check last update: Wed Oct
27 15:57:01 SGT 2021, details: service is running which took running duration
(ms) 9.65595245361
2021-10-27 15:58:59,112 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 654, power state: PowerOn
2021-10-27 15:58:59,119 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 14, power state: PowerOn
2021-10-27 15:58:59,122 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@61d49321- check type:
basic,check name: memory_usage_check.py, check result: true, check last update:
Wed Oct 27 15:57:01 SGT 2021, details: Memory Usage within limits with current
at 55.1192% which took running duration (ms) 54.6729564667
2021-10-27 15:58:59,125 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 590, power state: PowerOn
2021-10-27 15:58:59,131 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 655, power state: PowerOn
2021-10-27 15:58:59,134 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@13be7d12- check type:
basic,check name: cpu_usage_check.py, check result: true, check last update:
Wed Oct 27 15:57:10 SGT 2021, details: CPU Usage within limits with current at
1.0% which took running duration (ms) 3215.58308601
2021-10-27 15:58:59,137 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 593, power state: PowerOn
2021-10-27 15:58:59,144 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 595, power state: PowerOn
2021-10-27 15:58:59,145 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@5d50fbd2- check type:
basic,check name: router_version_check.py, check result: true, check last
update: Wed Oct 27 15:57:01 SGT 2021, details: Template and scripts version
match successful which took running duration (ms) 33.4870815277
2021-10-27 15:58:59,150 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 276, power state: PowerOn
2021-10-27 15:58:59,156 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 662, power state: PowerOn
2021-10-27 15:58:59,157 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@60583d5c- check type:
basic,check name: webserver.service, check result: true, check last update: Wed
Oct 27 15:57:01 SGT 2021, details: service is running which took running
duration (ms) 20.1549530029
2021-10-27 15:58:59,157 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found check types
executed on VR advanced, start: 2021-10-27 07:50:01.912077, end: 2021-10-27
07:50:02.247433, duration: 0.335355997086
2021-10-27 15:58:59,163 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 567, power state: PowerOn
2021-10-27 15:58:59,168 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@c260431- check type:
advanced,check name: haproxy_check.py, check result: true, check last update:
Wed Oct 27 15:50:02 SGT 2021, details: No data provided to check, skipping
which took running duration (ms) 29.464006424
2021-10-27 15:58:59,170 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 249, power state: PowerOn
2021-10-27 15:58:59,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 251, power state: PowerOn
2021-10-27 15:58:59,178 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@3f97396c- check type:
advanced,check name: dhcp_check.py, check result: true, check last update: Wed
Oct 27 15:50:01 SGT 2021, details: All 40 VMs are present in dhcphosts.txt
which took running duration (ms) 32.4671268463
2021-10-27 15:58:59,184 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 571, power state: PowerOn
2021-10-27 15:58:59,189 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@6f0bdabb- check type:
advanced,check name: dns_check.py, check result: false, check last update: Wed
Oct 27 15:50:02 SGT 2021, details: Missing entries for VMs in /etc/hosts -
103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.225
VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230
VM-94b471ec-65ef-4f57-8774-c523a689d421 which took running duration (ms)
35.9070301056
2021-10-27 15:58:59,191 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 667, power state: PowerOn
2021-10-27 15:58:59,198 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. host:
3, vm id: 637, power state: PowerOn
2021-10-27 15:58:59,199 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check
com.cloud.network.dao.RouterHealthCheckResultVO@6ac966ba- check type:
advanced,check name: iptables_check.py, check result: true, check last update:
Wed Oct 27 15:50:02 SGT 2021, details: No portforwarding rules provided to
check, skipping which took running duration (ms) 41.7759418488
2021-10-27 15:58:59,202 WARN [c.c.a.AlertManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) AlertType:: 9 |
dataCenterId:: 1 | podId:: 1 | clusterId:: null | message:: Health checks
failed: 1 failing checks on router 5d6063ef-804d-4223-887a-187800f4b6c3
2021-10-27 15:58:59,216 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Health checks failed: 1
failing checks on router 5d6063ef-804d-4223-887a-187800f4b6c3. Checking failed
health checks to see if router needs recreate
2021-10-27 15:58:59,235 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) Done with process of VM
state report. host: 3
