Hi Wei, I am using version 4.15 as of the moment.
Do you recommend for me to upgrade then? On 2021/10/27 08:16:04, Wei ZHOU <ustcweiz...@gmail.com> wrote: > Hi, > > Which cloudstack version do you use ? > > A similar issue has been fixed in 4.16.0.0-rc2. see > https://github.com/apache/cloudstack/pull/5554 > > -Wei > > On Wed, 27 Oct 2021 at 10:09, Paul <poco8...@gmail.com> wrote: > > > Hi, It's a mystery to me to find out that the dns_check.py for my > > deployment is failing every 10 minutes. I tried searching Google but I > > cannot find anything relevant to my scenario. Please advise. > > > > Log snippet: > > > > 2021-10-27 15:58:51,569 DEBUG [c.c.a.m.AgentManagerImpl] > > (AgentManager-Handler-12:null) (logid:) SeqA 6-1926377: Sending Seq > > 6-1926377: { Ans: , MgmtId: 1743772850, via: 6, Ver: v1, Flags: 100010, > > [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0"}}] } > > 2021-10-27 15:58:55,789 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > > (secstorage-1:ctx-19c33859) (logid:f3458df2) Zone 1 is ready to launch > > secondary storage VM > > 2021-10-27 15:58:55,929 DEBUG [c.c.c.ConsoleProxyManagerImpl] > > (consoleproxy-1:ctx-0e2ba419) (logid:fa5ecc2a) Zone 1 is ready to launch > > console proxy > > 2021-10-27 15:58:58,894 DEBUG [c.c.a.m.DirectAgentAttache] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) Ping from 3(LAB) > > 2021-10-27 15:58:58,894 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) Process host VM > > state report from ping process. host: 3 > > 2021-10-27 15:58:59,030 DEBUG [c.c.a.m.DirectAgentAttache] > > (DirectAgent-392:ctx-44d6234c) (logid:17ec443d) Seq 3-6314046677573480412: > > Response Received: > > 2021-10-27 15:58:59,044 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) Process VM state > > report. host: 3, number of records in report: 24 > > 2021-10-27 15:58:59,044 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 98, power state: PowerOn > > 2021-10-27 15:58:59,045 DEBUG [c.c.a.t.Request] > > (DirectAgent-392:ctx-44d6234c) (logid:17ec443d) Seq 3-6314046677573480412: > > Processing: { Ans: , MgmtId: 1743772850, via: 3(LAB), Ver: v1, Flags: 10, > > [{"com.cloud.agent.api.routing.GetRouterMonitorResultsAnswer":{"failingChecks":["dns_check.py"],"monitoringResults":"{"basic":{"lastRun": > > {"duration": "11.6007211208", "start": "2021-10-27 07:57:01.679916", "end": > > "2021-10-27 07:57:13.280637"}, "disk_space_check.py": {"lastRunDuration": > > "32.3760509491", "lastUpdate": "1635321429961", "message": "Sufficient free > > space is 1032 MB", "success": "true"}, "gateways_check.py": > > {"lastRunDuration": "8066.79701805", "lastUpdate": "1635321421894", > > "message": "All 1 gateways are reachable via ping", "success": "true"}, > > "ssh.service": {"lastRunDuration": "30.956029892", "lastUpdate": > > "1635321421681", "message": "service is running", "success": "true"}, > > "dhcp.service": {"lastRunDuration": "9.65595245361", "lastUpdate": > > "1635321421681", "message": "servi > > ce is running", "success": "true"}, "memory_usage_check.py": > > {"lastRunDuration": "54.6729564667", "lastUpdate": "1635321421773", > > "message": "Memory Usage within limits with current at 55.1192%", > > "success": "true"}, "cpu_usage_check.py": {"lastRunDuration": > > "3215.58308601", "lastUpdate": "1635321430064", "message": "CPU Usage > > within limits with current at 1.0%", "success": "true"}, > > "router_version_check.py": {"lastRunDuration": "33.4870815277", > > "lastUpdate": "1635321421740", "message": "Template and scripts version > > match successful", "success": "true"}, "webserver.service": > > {"lastRunDuration": "20.1549530029", "lastUpdate": "1635321421681", > > "message": "service is running", "success": "true"}},"advanced":{"lastRun": > > {"duration": "0.335355997086", "start": "2021-10-27 07:50:01.912077", > > "end": "2021-10-27 07:50:02.247433"}, "haproxy_check.py": > > {"lastRunDuration": "29.464006424", "lastUpdate": "1635321002030", > > "message": "No data provided to check, skipping", "success": "true"}, > > "dhcp_ch > > eck.py": {"lastRunDuration": "32.4671268463", "lastUpdate": > > "1635321001997", "message": "All 40 VMs are present in dhcphosts.txt", > > "success": "true"}, "dns_check.py": {"lastRunDuration": "35.9070301056", > > "lastUpdate": "1635321002164", "message": "Missing entries for VMs in > > /etc/hosts -\n103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, > > 103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421", "success": "false"}, > > "iptables_check.py": {"lastRunDuration": "41.7759418488", "lastUpdate": > > "1635321002122", "message": "No portforwarding rules provided to check, > > skipping", "success": > > "true"}}}","result":"true","details":"{"basic":{"lastRun": {"duration": > > "11.6007211208", "start": "2021-10-27 07:57:01.679916", "end": "2021-10-27 > > 07:57:13.280637"}, "disk_space_check.py": {"lastRunDuration": > > "32.3760509491", "lastUpdate": "1635321429961", "message": "Sufficient free > > space is > > 1032 MB", "success": "true"}, "gateways_check.py": {"lastRunDuration": > > "8066.79701805", "lastUpdate": "1635321421894", "message": "All 1 gateways > > are reachable via ping", "success": "true"}, "ssh.service": > > {"lastRunDuration": "30.956029892", "lastUpdate": "1635321421681", > > "message": "service is running", "success": "true"}, "dhcp.service": > > {"lastRunDuration": "9.65595245361", "lastUpdate": "1635321421681", > > "message": "service is running", "success": "true"}, > > "memory_usage_check.py": {"lastRunDuration": "54.6729564667", "lastUpdate": > > "1635321421773", "message": "Memory Usage within limits with current at > > 55.1192%", "success": "true"}, "cpu_usage_check.py": {"lastRunDuration": > > "3215.58308601", "lastUpdate": "1635321430064", "message": "CPU Usage > > within limits with current at 1.0%", "success": "true"}, > > "router_version_check.py": {"lastRunDuration": "33.4870815277", > > "lastUpdate": "1635321421740", "message": "Template and scripts version > > match successful", "success": "true"}, "webserver > > .service": {"lastRunDuration": "20.1549530029", "lastUpdate": > > "1635321421681", "message": "service is running", "success": > > "true"}},"advanced":{"lastRun": {"duration": "0.335355997086", "start": > > "2021-10-27 07:50:01.912077", "end": "2021-10-27 07:50:02.247433"}, > > "haproxy_check.py": {"lastRunDuration": "29.464006424", "lastUpdate": > > "1635321002030", "message": "No data provided to check, skipping", > > "success": "true"}, "dhcp_check.py": {"lastRunDuration": "32.4671268463", > > "lastUpdate": "1635321001997", "message": "All 40 VMs are present in > > dhcphosts.txt", "success": "true"}, "dns_check.py": {"lastRunDuration": > > "35.9070301056", "lastUpdate": "1635321002164", "message": "Missing entries > > for VMs in /etc/hosts -\n103.239.221.225 > > VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.225 > > VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421", "success": "false"}, > > "iptables_check.py": {"lastRunD > > uration": "41.7759418488", "lastUpdate": "1635321002122", "message": "No > > portforwarding rules provided to check, skipping", "success": > > "true"}}}","wait":"0"}}] } > > 2021-10-27 15:58:59,045 DEBUG [c.c.a.t.Request] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Seq > > 3-6314046677573480412: Received: { Ans: , MgmtId: 1743772850, via: 3(LAB), > > Ver: v1, Flags: 10, { GetRouterMonitorResultsAnswer } } > > 2021-10-27 15:58:59,045 DEBUG [c.c.a.m.AgentManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Details from > > executing class com.cloud.agent.api.routing.GetRouterMonitorResultsCommand: > > {"basic":{"lastRun": {"duration": "11.6007211208", "start": "2021-10-27 > > 07:57:01.679916", "end": "2021-10-27 07:57:13.280637"}, > > "disk_space_check.py": {"lastRunDuration": "32.3760509491", "lastUpdate": > > "1635321429961", "message": "Sufficient free space is 1032 MB", "success": > > "true"}, "gateways_check.py": {"lastRunDuration": "8066.79701805", > > "lastUpdate": "1635321421894", "message": "All 1 gateways are reachable via > > ping", "success": "true"}, "ssh.service": {"lastRunDuration": > > "30.956029892", "lastUpdate": "1635321421681", "message": "service is > > running", "success": "true"}, "dhcp.service": {"lastRunDuration": > > "9.65595245361", "lastUpdate": "1635321421681", "message": "service is > > running", "success": "true"}, "memory_usage_check.py": {"lastRunDuration": > > "54.6729564667", "lastUpdate": "16 > > 35321421773", "message": "Memory Usage within limits with current at > > 55.1192%", "success": "true"}, "cpu_usage_check.py": {"lastRunDuration": > > "3215.58308601", "lastUpdate": "1635321430064", "message": "CPU Usage > > within limits with current at 1.0%", "success": "true"}, > > "router_version_check.py": {"lastRunDuration": "33.4870815277", > > "lastUpdate": "1635321421740", "message": "Template and scripts version > > match successful", "success": "true"}, "webserver.service": > > {"lastRunDuration": "20.1549530029", "lastUpdate": "1635321421681", > > "message": "service is running", "success": "true"}},"advanced":{"lastRun": > > {"duration": "0.335355997086", "start": "2021-10-27 07:50:01.912077", > > "end": "2021-10-27 07:50:02.247433"}, "haproxy_check.py": > > {"lastRunDuration": "29.464006424", "lastUpdate": "1635321002030", > > "message": "No data provided to check, skipping", "success": "true"}, > > "dhcp_check.py": {"lastRunDuration": "32.4671268463", "lastUpdate": > > "1635321001997", "message": "All 40 VMs are present in > > dhcphosts.txt", "success": "true"}, "dns_check.py": {"lastRunDuration": > > "35.9070301056", "lastUpdate": "1635321002164", "message": "Missing entries > > for VMs in /etc/hosts -\n103.239.221.225 > > VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.225 > > VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421", "success": "false"}, > > "iptables_check.py": {"lastRunDuration": "41.7759418488", "lastUpdate": > > "1635321002122", "message": "No portforwarding rules provided to check, > > skipping", "success": "true"}}} > > 2021-10-27 15:58:59,052 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 2, power state: PowerOn > > 2021-10-27 15:58:59,059 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 613, power state: PowerOn > > 2021-10-27 15:58:59,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 677, power state: PowerOn > > 2021-10-27 15:58:59,068 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Parsing and updating > > DB health check data for router: 613 with data: {"basic":{"lastRun": > > {"duration": "11.6007211208", "start": "2021-10-27 07:57:01.679916", "end": > > "2021-10-27 07:57:13.280637"}, "disk_space_check.py": {"lastRunDuration": > > "32.3760509491", "lastUpdate": "1635321429961", "message": "Sufficient free > > space is 1032 MB", "success": "true"}, "gateways_check.py": > > {"lastRunDuration": "8066.79701805", "lastUpdate": "1635321421894", > > "message": "All 1 gateways are reachable via ping", "success": "true"}, > > "ssh.service": {"lastRunDuration": "30.956029892", "lastUpdate": > > "1635321421681", "message": "service is running", "success": "true"}, > > "dhcp.service": {"lastRunDuration": "9.65595245361", "lastUpdate": > > "1635321421681", "message": "service is running", "success": "true"}, > > "memory_usage_check.py": {"lastRunDuration": "54.6729564667", "lastUpdate": > > "1635 > > 321421773", "message": "Memory Usage within limits with current at > > 55.1192%", "success": "true"}, "cpu_usage_check.py": {"lastRunDuration": > > "3215.58308601", "lastUpdate": "1635321430064", "message": "CPU Usage > > within limits with current at 1.0%", "success": "true"}, > > "router_version_check.py": {"lastRunDuration": "33.4870815277", > > "lastUpdate": "1635321421740", "message": "Template and scripts version > > match successful", "success": "true"}, "webserver.service": > > {"lastRunDuration": "20.1549530029", "lastUpdate": "1635321421681", > > "message": "service is running", "success": "true"}},"advanced":{"lastRun": > > {"duration": "0.335355997086", "start": "2021-10-27 07:50:01.912077", > > "end": "2021-10-27 07:50:02.247433"}, "haproxy_check.py": > > {"lastRunDuration": "29.464006424", "lastUpdate": "1635321002030", > > "message": "No data provided to check, skipping", "success": "true"}, > > "dhcp_check.py": {"lastRunDuration": "32.4671268463", "lastUpdate": > > "1635321001997", "message": "All 40 VMs are present in dh > > cphosts.txt", "success": "true"}, "dns_check.py": {"lastRunDuration": > > "35.9070301056", "lastUpdate": "1635321002164", "message": "Missing entries > > for VMs in /etc/hosts -\n103.239.221.225 > > VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.225 > > VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421", "success": "false"}, > > "iptables_check.py": {"lastRunDuration": "41.7759418488", "lastUpdate": > > "1635321002122", "message": "No portforwarding rules provided to check, > > skipping", "success": "true"}}} > > 2021-10-27 15:58:59,072 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found check types > > executed on VR basic, start: 2021-10-27 07:57:01.679916, end: 2021-10-27 > > 07:57:13.280637, duration: 11.6007211208 > > 2021-10-27 15:58:59,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 518, power state: PowerOn > > 2021-10-27 15:58:59,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 233, power state: PowerOn > > 2021-10-27 15:58:59,083 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@735fcbf9- check type: > > basic,check name: disk_space_check.py, check result: true, check last > > update: Wed Oct 27 15:57:09 SGT 2021, details: Sufficient free space is > > 1032 MB which took running duration (ms) 32.3760509491 > > 2021-10-27 15:58:59,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 651, power state: PowerOn > > 2021-10-27 15:58:59,092 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@6d56e337- check type: > > basic,check name: gateways_check.py, check result: true, check last update: > > Wed Oct 27 15:57:01 SGT 2021, details: All 1 gateways are reachable via > > ping which took running duration (ms) 8066.79701805 > > 2021-10-27 15:58:59,093 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 588, power state: PowerOn > > 2021-10-27 15:58:59,099 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 12, power state: PowerOn > > 2021-10-27 15:58:59,102 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@503c8360- check type: > > basic,check name: ssh.service, check result: true, check last update: Wed > > Oct 27 15:57:01 SGT 2021, details: service is running which took running > > duration (ms) 30.956029892 > > 2021-10-27 15:58:59,105 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 589, power state: PowerOn > > 2021-10-27 15:58:59,112 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@37d29ee7- check type: > > basic,check name: dhcp.service, check result: true, check last update: Wed > > Oct 27 15:57:01 SGT 2021, details: service is running which took running > > duration (ms) 9.65595245361 > > 2021-10-27 15:58:59,112 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 654, power state: PowerOn > > 2021-10-27 15:58:59,119 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 14, power state: PowerOn > > 2021-10-27 15:58:59,122 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@61d49321- check type: > > basic,check name: memory_usage_check.py, check result: true, check last > > update: Wed Oct 27 15:57:01 SGT 2021, details: Memory Usage within limits > > with current at 55.1192% which took running duration (ms) 54.6729564667 > > 2021-10-27 15:58:59,125 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 590, power state: PowerOn > > 2021-10-27 15:58:59,131 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 655, power state: PowerOn > > 2021-10-27 15:58:59,134 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@13be7d12- check type: > > basic,check name: cpu_usage_check.py, check result: true, check last > > update: Wed Oct 27 15:57:10 SGT 2021, details: CPU Usage within limits with > > current at 1.0% which took running duration (ms) 3215.58308601 > > 2021-10-27 15:58:59,137 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 593, power state: PowerOn > > 2021-10-27 15:58:59,144 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 595, power state: PowerOn > > 2021-10-27 15:58:59,145 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@5d50fbd2- check type: > > basic,check name: router_version_check.py, check result: true, check last > > update: Wed Oct 27 15:57:01 SGT 2021, details: Template and scripts version > > match successful which took running duration (ms) 33.4870815277 > > 2021-10-27 15:58:59,150 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 276, power state: PowerOn > > 2021-10-27 15:58:59,156 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 662, power state: PowerOn > > 2021-10-27 15:58:59,157 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@60583d5c- check type: > > basic,check name: webserver.service, check result: true, check last update: > > Wed Oct 27 15:57:01 SGT 2021, details: service is running which took > > running duration (ms) 20.1549530029 > > 2021-10-27 15:58:59,157 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found check types > > executed on VR advanced, start: 2021-10-27 07:50:01.912077, end: 2021-10-27 > > 07:50:02.247433, duration: 0.335355997086 > > 2021-10-27 15:58:59,163 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 567, power state: PowerOn > > 2021-10-27 15:58:59,168 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@c260431- check type: > > advanced,check name: haproxy_check.py, check result: true, check last > > update: Wed Oct 27 15:50:02 SGT 2021, details: No data provided to check, > > skipping which took running duration (ms) 29.464006424 > > 2021-10-27 15:58:59,170 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 249, power state: PowerOn > > 2021-10-27 15:58:59,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 251, power state: PowerOn > > 2021-10-27 15:58:59,178 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@3f97396c- check type: > > advanced,check name: dhcp_check.py, check result: true, check last update: > > Wed Oct 27 15:50:01 SGT 2021, details: All 40 VMs are present in > > dhcphosts.txt which took running duration (ms) 32.4671268463 > > 2021-10-27 15:58:59,184 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 571, power state: PowerOn > > 2021-10-27 15:58:59,189 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@6f0bdabb- check type: > > advanced,check name: dns_check.py, check result: false, check last update: > > Wed Oct 27 15:50:02 SGT 2021, details: Missing entries for VMs in > > /etc/hosts - > > 103.239.221.225 VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.225 > > VM-59c7fd13-32f7-4ace-958f-18c84f2b50f5, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421, 103.239.221.230 > > VM-94b471ec-65ef-4f57-8774-c523a689d421 which took running duration (ms) > > 35.9070301056 > > 2021-10-27 15:58:59,191 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 667, power state: PowerOn > > 2021-10-27 15:58:59,198 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) VM state report. > > host: 3, vm id: 637, power state: PowerOn > > 2021-10-27 15:58:59,199 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Found health check > > com.cloud.network.dao.RouterHealthCheckResultVO@6ac966ba- check type: > > advanced,check name: iptables_check.py, check result: true, check last > > update: Wed Oct 27 15:50:02 SGT 2021, details: No portforwarding rules > > provided to check, skipping which took running duration (ms) 41.7759418488 > > 2021-10-27 15:58:59,202 WARN [c.c.a.AlertManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) AlertType:: 9 | > > dataCenterId:: 1 | podId:: 1 | clusterId:: null | message:: Health checks > > failed: 1 failing checks on router 5d6063ef-804d-4223-887a-187800f4b6c3 > > 2021-10-27 15:58:59,216 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > (RouterStatusMonitor-1:ctx-cca75a19) (logid:17ec443d) Health checks failed: > > 1 failing checks on router 5d6063ef-804d-4223-887a-187800f4b6c3. Checking > > failed health checks to see if router needs recreate > > 2021-10-27 15:58:59,235 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > > (DirectAgentCronJob-319:ctx-1b85b80c) (logid:eb936b4c) Done with process of > > VM state report. host: 3 > > >
