I upgraded Cloudstack 4.13 to 4.15.2, still the same error: Monitor ComputeCapacityListener says there is an error in the connect process for 260 due to null java.lang.NullPointerException at com.cloud.capacity.CapacityManagerImpl.updateCapacityForHost(CapacityManagerImpl.java:686)
Could it be related to that system vms are still running 4.11.2 template? (4.11.2 is running for 4 years or so)
*Any ideas how to upgrade console and storage system vms to 4.15.2 template if Cloudstack cannot manage hosts?*
Some last ERRORs and WARNs below: ============================================2024-07-23 12:45:43,982 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-81:ctx-4d88cf7c) (logid:c62194f2) Monitor ComputeCapacityListener says there is an error in the connect process for 248 due to null 2024-07-23 12:45:45,386 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-82:ctx-c83208d5) (logid:ea38f7dd) Monitor ComputeCapacityListener says there is an error in the connect process for 254 due to null 2024-07-23 12:45:46,075 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-83:ctx-5fb35072) (logid:b7df75ba) Monitor ComputeCapacityListener says there is an error in the connect process for 257 due to null 2024-07-23 12:45:46,106 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-84:ctx-70f90b00) (logid:f7e53041) Monitor ComputeCapacityListener says there is an error in the connect process for 260 due to null 2024-07-23 12:46:04,076 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-fa39a8a4) (logid:7d53f056) Caught exception in recalculating capacity 2024-07-23 12:47:04,014 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-15eb0c8f) (logid:ed70ae4e) Caught exception in recalculating capacity 2024-07-23 12:47:39,287 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-85:ctx-54145195) (logid:da6c27fe) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:47:39,479 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-86:ctx-60a73df9) (logid:708313f4) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:47:39,674 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-87:ctx-49add049) (logid:7940fff8) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:47:39,727 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-88:ctx-f708573c) (logid:422a27af) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:47:43,918 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-85:ctx-54145195) (logid:da6c27fe) Monitor ComputeCapacityListener says there is an error in the connect process for 248 due to null 2024-07-23 12:47:45,347 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-86:ctx-60a73df9) (logid:708313f4) Monitor ComputeCapacityListener says there is an error in the connect process for 254 due to null 2024-07-23 12:47:45,800 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-88:ctx-f708573c) (logid:422a27af) Monitor ComputeCapacityListener says there is an error in the connect process for 260 due to null 2024-07-23 12:47:46,282 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-87:ctx-49add049) (logid:7940fff8) Monitor ComputeCapacityListener says there is an error in the connect process for 257 due to null 2024-07-23 12:48:04,031 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-1fd8ad07) (logid:c61cdeb1) Caught exception in recalculating capacity 2024-07-23 12:49:04,017 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-a200d404) (logid:1a7fe8c3) Caught exception in recalculating capacity 2024-07-23 12:49:39,286 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-89:ctx-2bf30748) (logid:dd025da3) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:49:39,482 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-90:ctx-e71c3cb3) (logid:4323fafb) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:49:39,710 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-91:ctx-67526ba6) (logid:b14219d0) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:49:39,720 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-92:ctx-4002a251) (logid:657202f4) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:49:41,401 ERROR [c.c.c.ClusterServiceServletContainer] (Thread-18:null) (logid:) Unexpected exception 2024-07-23 12:50:10,905 ERROR [c.c.c.ClusterManagerImpl] (main:null) (logid:) Unable to ping management server at 10.10.10.11:9090 due to ConnectException 2024-07-23 12:50:27,179 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-1:ctx-a0261ab7) (logid:48ebda4b) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:50:27,285 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-2:ctx-108df46d) (logid:dcb8b5d7) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:50:27,508 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-3:ctx-6f71596b) (logid:d976c278) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:50:27,522 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-4:ctx-d1e37186) (logid:62dcf805) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:50:32,768 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-1:ctx-a0261ab7) (logid:48ebda4b) Monitor ComputeCapacityListener says there is an error in the connect process for 248 due to null 2024-07-23 12:50:33,773 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-108df46d) (logid:dcb8b5d7) Monitor ComputeCapacityListener says there is an error in the connect process for 254 due to null 2024-07-23 12:50:34,103 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-4:ctx-d1e37186) (logid:62dcf805) Monitor ComputeCapacityListener says there is an error in the connect process for 260 due to null 2024-07-23 12:50:34,236 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-3:ctx-6f71596b) (logid:d976c278) Monitor ComputeCapacityListener says there is an error in the connect process for 257 due to null 2024-07-23 12:50:48,919 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null) (logid:) Seq 209-1213720099576348683: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"da9f1feb-8a0d-46e3-a0a4-cf836bebd442","downloadPct":"0","errorString":" HTTP Server returned 404 (expected 200 OK) ","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/9fbc3b37-b8ad-3eee-b2af-053832061fe7/template/tmpl/283/725/dnld3368347414585741593tmp_","installPath":"template/tmpl/283/725","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"{SHA-512}03995bc65699d3c92b7ea83b8a77c74482a3d9e2e015d2b5755b01a38e4232cc720fd82265cf351c0989f5d04bcdb9c5e4ecc8014a1a9acfabfa61d29ede73e3","result":"true","details":" HTTP Server returned 404 (expected 200 OK) ","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:50:48,974 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-1:ctx-625b0cc1) (logid:2abe9268) Failed to register template: 80745d5a-8359-488f-b225-fc09670f7fa7 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:50:51,371 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-151ef859) (logid:21052d7c) Caught exception in recalculating capacity 2024-07-23 12:51:05,839 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 209-1213720099576348686: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"2c3ac366-f515-4d71-aa9c-d3716722f543","downloadPct":"0","errorString":" HTTP Server returned 404 (expected 200 OK) ","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/1283f316-9cc8-3dfb-a7ed-9fa292d06202/template/tmpl/2/768/dnld8746222714158484790tmp_","installPath":"template/tmpl/2/768","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"{SHA-512}353a997e71515806e66ebdf5489e522a11463b6db0d3fe341ef2502714343e7d0061fb8f464ec60b6d18598ca9c0810429c434d7e6d016644d7b28c128d18039","result":"true","details":" HTTP Server returned 404 (expected 200 OK) ","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:05,871 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 209-1213720099576348687: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"9064837b-afed-4601-9221-c8dc16038cfe","downloadPct":"0","errorString":" HTTP Server returned 404 (expected 200 OK) ","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/1283f316-9cc8-3dfb-a7ed-9fa292d06202/template/tmpl/2/776/dnld379950261993851161tmp_","installPath":"template/tmpl/2/776","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"{SHA-512}59cbc2e14ca5777cd95a769e7e1f416a220a2231356192b9dce34ccf0e1e9e045f877f76961b78e3b69f48aacb14c9a731f2221910dc7557e2e7921e44c1ef23","result":"true","details":" HTTP Server returned 404 (expected 200 OK) ","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:05,889 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-1:ctx-36fb0b56) (logid:4b755a2e) Failed to register template: e51131e3-39af-4749-8a3a-c5a32de583d0 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:05,919 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-107abf64) (logid:d7925f2b) Failed to register template: 30feaccd-5efe-4b1a-b636-1d3831604ab4 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:05,988 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:null) (logid:) Seq 209-1213720099576348688: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"b3fad476-da59-4f20-9817-fac5eeb38f97","downloadPct":"0","errorString":" HTTP Server returned 404 (expected 200 OK) ","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/1283f316-9cc8-3dfb-a7ed-9fa292d06202/template/tmpl/283/725/dnld3817215856530157180tmp_","installPath":"template/tmpl/283/725","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"{SHA-512}03995bc65699d3c92b7ea83b8a77c74482a3d9e2e015d2b5755b01a38e4232cc720fd82265cf351c0989f5d04bcdb9c5e4ecc8014a1a9acfabfa61d29ede73e3","result":"true","details":" HTTP Server returned 404 (expected 200 OK) ","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:06,033 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-0a4277db) (logid:22736af0) Failed to register template: 80745d5a-8359-488f-b225-fc09670f7fa7 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:06,093 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:null) (logid:) Seq 209-1213720099576348689: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"b098f322-65c0-4f50-ba08-4a08bb2820e5","downloadPct":"0","errorString":" HTTP Server returned 404 (expected 200 OK) ","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/1283f316-9cc8-3dfb-a7ed-9fa292d06202/template/tmpl/2/769/dnld3767042060573065314tmp_","installPath":"template/tmpl/2/769","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"{SHA-512}11d733d626d1c7d3b20cfcccc516caff2cbc57c81769d56434aab958d4d9b3af59106bc0796252aeefede8353e2582378e08c65e35a36769d5cf673c5444f80e","result":"true","details":" HTTP Server returned 404 (expected 200 OK) ","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:06,142 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-70dc58c0) (logid:b82fd5e7) Failed to register template: a51ae863-0e4b-4204-8b00-d5fc70f07e13 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:06,290 DEBUG [c.c.a.t.Request] (AgentManager-Handler-16:null) (logid:) Seq 209-1213720099576348691: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"bf93bda9-6c7e-4a2e-ae8c-0888cb3d2612","downloadPct":"0","errorString":" HTTP Server returned 404 (expected 200 OK) ","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/1283f316-9cc8-3dfb-a7ed-9fa292d06202/template/tmpl/2/771/dnld6274861868355105828tmp_","installPath":"template/tmpl/2/771","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"{SHA-512}7381a9fcf6f15cebd176b85ca1470ce924b3589d0ffb48c26e171e0bc61a6bb0fe0f883fe8307fbf10cfd841c10ead1d0cb75b6342a0844c7afe5e193fefe6f8","result":"true","details":" HTTP Server returned 404 (expected 200 OK) ","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:06,333 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-49ad1cf2) (logid:e424ef32) Failed to register template: 1bc63305-d743-4f18-98f5-8a75a5333653 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:06,518 DEBUG [c.c.a.t.Request] (AgentManager-Handler-18:null) (logid:) Seq 209-1213720099576348693: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"e8024447-81ff-45f6-8f1f-9c0e00e12d33","downloadPct":"0","errorString":" HTTP Server returned 404 (expected 200 OK) ","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/1283f316-9cc8-3dfb-a7ed-9fa292d06202/template/tmpl/2/765/dnld6085301853043105192tmp_","installPath":"template/tmpl/2/765","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"{SHA-512}02fd0259aff4355166b8bfd76b067bf299197a0f512553595651341da4740288bebc6dfae0b6a769d27a79839573aaa11d2dc9f2c988aca037108b07bb88650d","result":"true","details":" HTTP Server returned 404 (expected 200 OK) ","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:06,565 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-b28d85ca) (logid:ea60a1f1) Failed to register template: e31467df-3857-41a4-a21e-22f0dc7008df with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:16,243 DEBUG [c.c.a.t.Request] (AgentManager-Handler-24:null) (logid:) Seq 209-1213720099576348696: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"7c0218aa-fbc8-41e3-a18c-a4581029eea3","downloadPct":"0","errorString":"No route to host (Host unreachable)","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/1283f316-9cc8-3dfb-a7ed-9fa292d06202/template/tmpl/2/295/dnld4964797074958599302tmp_","installPath":"template/tmpl/2/295","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"56ac4fdc1c42b539a0787ccf036fe186","result":"true","details":"No route to host (Host unreachable)","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:16,292 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-a8a51b7a) (logid:60ebd03c) Failed to register template: 9f402c90-f381-4bc6-a161-4b22a525ae63 with error: No route to host (Host unreachable) 2024-07-23 12:51:31,071 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:null) (logid:) Seq 209-1213720099576348703: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"8b8b9177-cb93-42d1-8a22-8cf70b7ae48f","downloadPct":"0","errorString":"No route to host (Host unreachable)","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/a6fcad21-ac13-3517-ad51-6e0e25df9c14/template/tmpl/2/295/dnld1256438286177965066tmp_","installPath":"template/tmpl/2/295","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"56ac4fdc1c42b539a0787ccf036fe186","result":"true","details":"No route to host (Host unreachable)","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:31,115 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-5fcbbff5) (logid:4f7997ec) Failed to register template: 9f402c90-f381-4bc6-a161-4b22a525ae63 with error: No route to host (Host unreachable) 2024-07-23 12:51:41,239 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) (logid:) Seq 209-1213720099576348708: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"26740efd-2c93-4266-8776-cf9270b47a24","downloadPct":"0","errorString":"No route to host (Host unreachable)","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/a6fcad21-ac13-3517-ad51-6e0e25df9c14/template/tmpl/2/292/dnld5104612569867744036tmp_","installPath":"template/tmpl/2/292","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"fe8e8e44394ddfda4f793c15c803ca3c","result":"true","details":"No route to host (Host unreachable)","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:51:41,285 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-24c2c462) (logid:cfc2ffea) Failed to register template: 7163a1aa-344b-4afa-bb2b-5bfa182b53d2 with error: No route to host (Host unreachable) 2024-07-23 12:51:51,366 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-b295000c) (logid:0c47527f) Caught exception in recalculating capacity 2024-07-23 12:52:23,423 DEBUG [c.c.a.t.Request] (AgentManager-Handler-20:null) (logid:) Seq 209-1213720099576348712: Processing: { Ans: , MgmtId: 95530460204, via: 209, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"69a34a01-4605-4229-9aeb-53bf0e5f9ff7","downloadPct":"0","errorString":"No route to host (Host unreachable)","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"/mnt/SecStorage/1283f316-9cc8-3dfb-a7ed-9fa292d06202/template/tmpl/2/292/dnld6745697132157670126tmp_","installPath":"template/tmpl/2/292","templateSize":"(0 bytes) 0","templatePhySicalSize":"(0 bytes) 0","checkSum":"fe8e8e44394ddfda4f793c15c803ca3c","result":"true","details":"No route to host (Host unreachable)","wait":"0","bypassHostMaintenance":"false"}}] } 2024-07-23 12:52:23,470 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-2:ctx-1389f748) (logid:b8306c21) Failed to register template: 7163a1aa-344b-4afa-bb2b-5bfa182b53d2 with error: No route to host (Host unreachable) 2024-07-23 12:52:26,764 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-5:ctx-86f1dfff) (logid:e29bf016) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:52:26,951 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-6:ctx-f4397018) (logid:383b301a) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:52:27,186 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-7:ctx-31c90c8b) (logid:2873a550) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:52:27,227 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-8:ctx-77a2b38a) (logid:21a72f49) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:52:31,579 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-5:ctx-86f1dfff) (logid:e29bf016) Monitor ComputeCapacityListener says there is an error in the connect process for 248 due to null 2024-07-23 12:52:32,906 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-6:ctx-f4397018) (logid:383b301a) Monitor ComputeCapacityListener says there is an error in the connect process for 254 due to null 2024-07-23 12:52:33,587 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-8:ctx-77a2b38a) (logid:21a72f49) Monitor ComputeCapacityListener says there is an error in the connect process for 260 due to null 2024-07-23 12:52:34,241 ERROR [c.c.a.m.AgentManagerImpl] (AgentTaskPool-7:ctx-31c90c8b) (logid:2873a550) Monitor ComputeCapacityListener says there is an error in the connect process for 257 due to null 2024-07-23 12:52:51,393 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-9ed2899e) (logid:097abe87) Caught exception in recalculating capacity 2024-07-23 12:53:51,348 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-e45a9c36) (logid:6ab275d6) Caught exception in recalculating capacity 2024-07-23 12:54:27,523 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-9:ctx-75be1d9b) (logid:56e2daed) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:54:27,677 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-10:ctx-f5b43b18) (logid:d234b7a5) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:54:27,903 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-11:ctx-4710e6da) (logid:c1682411) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output: 2024-07-23 12:54:27,952 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-12:ctx-d2dc7f4d) (logid:4aa1ca2f) SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code in return. Result output:
=============== WARN2024-07-23 12:46:08,056 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 77-3187422636271468859: Timed out on null 2024-07-23 12:46:08,068 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 170-8501107246615232844: Timed out on null 2024-07-23 12:46:08,077 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 74-82190693199511797: Timed out on null 2024-07-23 12:46:08,090 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 74-82190693199511798: Timed out on null 2024-07-23 12:46:08,099 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 77-3187422636271468860: Timed out on null 2024-07-23 12:46:08,108 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 170-8501107246615232845: Timed out on null 2024-07-23 12:46:08,123 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 77-3187422636271468861: Timed out on null 2024-07-23 12:46:08,132 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 74-82190693199511799: Timed out on null 2024-07-23 12:46:08,142 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 170-8501107246615232846: Timed out on null 2024-07-23 12:46:08,157 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 170-8501107246615232847: Timed out on null 2024-07-23 12:46:08,166 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 77-3187422636271468862: Timed out on null 2024-07-23 12:46:08,175 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 74-82190693199511800: Timed out on null 2024-07-23 12:46:08,187 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 74-82190693199511801: Timed out on null 2024-07-23 12:46:08,195 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 170-8501107246615232848: Timed out on null 2024-07-23 12:46:08,204 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-3da30bf6) (logid:04003b4a) Seq 77-3187422636271468863: Timed out on null 2024-07-23 12:46:53,591 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Seq 18-8490692672476938282: Timed out on null 2024-07-23 12:46:53,591 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Operation timed out: Commands 8490692672476938282 to Host 18 timed out after 172800 2024-07-23 12:46:53,591 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Unable to obtain host 18 statistics. 2024-07-23 12:46:53,591 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) The Host stats is null for host: 18 2024-07-23 12:46:53,616 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Seq 74-82190693199511802: Timed out on null 2024-07-23 12:46:53,616 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Operation timed out: Commands 82190693199511802 to Host 74 timed out after 172800 2024-07-23 12:46:53,616 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Unable to obtain host 74 statistics. 2024-07-23 12:46:53,616 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) The Host stats is null for host: 74 2024-07-23 12:46:53,641 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Seq 77-3187422636271468864: Timed out on null 2024-07-23 12:46:53,641 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Operation timed out: Commands 3187422636271468864 to Host 77 timed out after 172800 2024-07-23 12:46:53,641 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Unable to obtain host 77 statistics. 2024-07-23 12:46:53,641 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) The Host stats is null for host: 77 2024-07-23 12:46:53,667 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Seq 170-8501107246615232849: Timed out on null 2024-07-23 12:46:53,667 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Operation timed out: Commands 8501107246615232849 to Host 170 timed out after 172800 2024-07-23 12:46:53,667 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) Unable to obtain host 170 statistics. 2024-07-23 12:46:53,667 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-9c1143f2) (logid:8e9da756) The Host stats is null for host: 170 2024-07-23 12:46:55,167 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-5175a997) (logid:b5f2a49a) Seq 77-3187422636271468865: Timed out on null 2024-07-23 12:46:55,167 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-5175a997) (logid:b5f2a49a) Operation timed out: Commands 3187422636271468865 to Host 77 timed out after 172800 2024-07-23 12:46:55,167 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-1:ctx-5175a997) (logid:b5f2a49a) Unable to obtain VM statistics. 2024-07-23 12:46:55,257 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-5175a997) (logid:b5f2a49a) Seq 170-8501107246615232850: Timed out on null 2024-07-23 12:46:55,258 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-5175a997) (logid:b5f2a49a) Operation timed out: Commands 8501107246615232850 to Host 170 timed out after 172800 2024-07-23 12:46:55,258 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-1:ctx-5175a997) (logid:b5f2a49a) Unable to obtain VM statistics. 2024-07-23 12:47:04,038 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-15eb0c8f) (logid:ed70ae4e) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:47:08,509 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 74-82190693199511803: Timed out on null 2024-07-23 12:47:08,517 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 170-8501107246615232851: Timed out on null 2024-07-23 12:47:08,527 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 77-3187422636271468866: Timed out on null 2024-07-23 12:47:08,538 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 74-82190693199511804: Timed out on null 2024-07-23 12:47:08,547 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 77-3187422636271468867: Timed out on null 2024-07-23 12:47:08,557 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 170-8501107246615232852: Timed out on null 2024-07-23 12:47:08,570 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 77-3187422636271468868: Timed out on null 2024-07-23 12:47:08,581 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 170-8501107246615232853: Timed out on null 2024-07-23 12:47:08,590 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 74-82190693199511805: Timed out on null 2024-07-23 12:47:08,604 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 77-3187422636271468869: Timed out on null 2024-07-23 12:47:08,613 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 170-8501107246615232854: Timed out on null 2024-07-23 12:47:08,621 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 74-82190693199511806: Timed out on null 2024-07-23 12:47:08,632 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 74-82190693199511807: Timed out on null 2024-07-23 12:47:08,641 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 170-8501107246615232855: Timed out on null 2024-07-23 12:47:08,649 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-dc96032a) (logid:67656f73) Seq 77-3187422636271468870: Timed out on null 2024-07-23 12:47:33,781 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-f0104260) (logid:c73ee5a4) Seq 77-3187422636271468871: Timed out on null 2024-07-23 12:47:33,781 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-f0104260) (logid:c73ee5a4) Operation timed out: Commands 3187422636271468871 to Host 77 timed out after 172800 2024-07-23 12:47:33,781 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-3:ctx-f0104260) (logid:c73ee5a4) Unable to obtain VM network statistics. 2024-07-23 12:47:33,848 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-f0104260) (logid:c73ee5a4) Seq 170-8501107246615232856: Timed out on null 2024-07-23 12:47:33,848 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-f0104260) (logid:c73ee5a4) Operation timed out: Commands 8501107246615232856 to Host 170 timed out after 172800 2024-07-23 12:47:33,848 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-3:ctx-f0104260) (logid:c73ee5a4) Unable to obtain VM network statistics. 2024-07-23 12:47:39,439 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-85:ctx-54145195) (logid:da6c27fe) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:47:39,620 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-86:ctx-60a73df9) (logid:708313f4) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:47:39,821 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-87:ctx-49add049) (logid:7940fff8) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:47:39,871 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-88:ctx-f708573c) (logid:422a27af) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:47:43,941 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-85:ctx-54145195) (logid:da6c27fe) Unable to connect due to 2024-07-23 12:47:45,370 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-86:ctx-60a73df9) (logid:708313f4) Unable to connect due to 2024-07-23 12:47:45,824 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-88:ctx-f708573c) (logid:422a27af) Unable to connect due to 2024-07-23 12:47:46,303 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-87:ctx-49add049) (logid:7940fff8) Unable to connect due to 2024-07-23 12:47:53,699 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Seq 18-8490692672476938283: Timed out on null 2024-07-23 12:47:53,699 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Operation timed out: Commands 8490692672476938283 to Host 18 timed out after 172800 2024-07-23 12:47:53,699 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Unable to obtain host 18 statistics. 2024-07-23 12:47:53,699 WARN [c.c.s.StatsCollector] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) The Host stats is null for host: 18 2024-07-23 12:47:53,723 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Seq 74-82190693199511808: Timed out on null 2024-07-23 12:47:53,724 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Operation timed out: Commands 82190693199511808 to Host 74 timed out after 172800 2024-07-23 12:47:53,724 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Unable to obtain host 74 statistics. 2024-07-23 12:47:53,724 WARN [c.c.s.StatsCollector] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) The Host stats is null for host: 74 2024-07-23 12:47:53,747 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Seq 77-3187422636271468872: Timed out on null 2024-07-23 12:47:53,747 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Operation timed out: Commands 3187422636271468872 to Host 77 timed out after 172800 2024-07-23 12:47:53,747 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Unable to obtain host 77 statistics. 2024-07-23 12:47:53,747 WARN [c.c.s.StatsCollector] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) The Host stats is null for host: 77 2024-07-23 12:47:53,770 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Seq 170-8501107246615232857: Timed out on null 2024-07-23 12:47:53,770 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Operation timed out: Commands 8501107246615232857 to Host 170 timed out after 172800 2024-07-23 12:47:53,770 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) Unable to obtain host 170 statistics. 2024-07-23 12:47:53,770 WARN [c.c.s.StatsCollector] (StatsCollector-6:ctx-0dbe1005) (logid:c4ba2650) The Host stats is null for host: 170 2024-07-23 12:47:55,312 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-7a4a70a5) (logid:57a72fb2) Seq 77-3187422636271468873: Timed out on null 2024-07-23 12:47:55,312 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-7a4a70a5) (logid:57a72fb2) Operation timed out: Commands 3187422636271468873 to Host 77 timed out after 172800 2024-07-23 12:47:55,312 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-4:ctx-7a4a70a5) (logid:57a72fb2) Unable to obtain VM statistics. 2024-07-23 12:47:55,401 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-7a4a70a5) (logid:57a72fb2) Seq 170-8501107246615232858: Timed out on null 2024-07-23 12:47:55,401 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-7a4a70a5) (logid:57a72fb2) Operation timed out: Commands 8501107246615232858 to Host 170 timed out after 172800 2024-07-23 12:47:55,401 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-4:ctx-7a4a70a5) (logid:57a72fb2) Unable to obtain VM statistics. 2024-07-23 12:48:04,060 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-1fd8ad07) (logid:c61cdeb1) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:48:08,974 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 77-3187422636271468874: Timed out on null 2024-07-23 12:48:08,983 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 74-82190693199511809: Timed out on null 2024-07-23 12:48:08,991 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 170-8501107246615232859: Timed out on null 2024-07-23 12:48:09,004 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 77-3187422636271468875: Timed out on null 2024-07-23 12:48:09,013 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 74-82190693199511810: Timed out on null 2024-07-23 12:48:09,045 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 170-8501107246615232860: Timed out on null 2024-07-23 12:48:09,086 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 77-3187422636271468876: Timed out on null 2024-07-23 12:48:09,095 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 170-8501107246615232861: Timed out on null 2024-07-23 12:48:09,103 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 74-82190693199511811: Timed out on null 2024-07-23 12:48:09,117 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 77-3187422636271468877: Timed out on null 2024-07-23 12:48:09,126 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 170-8501107246615232862: Timed out on null 2024-07-23 12:48:09,135 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 74-82190693199511812: Timed out on null 2024-07-23 12:48:09,148 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 170-8501107246615232863: Timed out on null 2024-07-23 12:48:09,158 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 77-3187422636271468878: Timed out on null 2024-07-23 12:48:09,166 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-94398385) (logid:c2a85e86) Seq 74-82190693199511813: Timed out on null 2024-07-23 12:48:53,804 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Seq 18-8490692672476938284: Timed out on null 2024-07-23 12:48:53,804 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Operation timed out: Commands 8490692672476938284 to Host 18 timed out after 172800 2024-07-23 12:48:53,804 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Unable to obtain host 18 statistics. 2024-07-23 12:48:53,804 WARN [c.c.s.StatsCollector] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) The Host stats is null for host: 18 2024-07-23 12:48:53,829 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Seq 74-82190693199511814: Timed out on null 2024-07-23 12:48:53,829 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Operation timed out: Commands 82190693199511814 to Host 74 timed out after 172800 2024-07-23 12:48:53,829 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Unable to obtain host 74 statistics. 2024-07-23 12:48:53,829 WARN [c.c.s.StatsCollector] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) The Host stats is null for host: 74 2024-07-23 12:48:53,854 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Seq 77-3187422636271468879: Timed out on null 2024-07-23 12:48:53,854 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Operation timed out: Commands 3187422636271468879 to Host 77 timed out after 172800 2024-07-23 12:48:53,854 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Unable to obtain host 77 statistics. 2024-07-23 12:48:53,854 WARN [c.c.s.StatsCollector] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) The Host stats is null for host: 77 2024-07-23 12:48:53,878 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Seq 170-8501107246615232864: Timed out on null 2024-07-23 12:48:53,878 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Operation timed out: Commands 8501107246615232864 to Host 170 timed out after 172800 2024-07-23 12:48:53,878 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) Unable to obtain host 170 statistics. 2024-07-23 12:48:53,879 WARN [c.c.s.StatsCollector] (StatsCollector-3:ctx-ac998c45) (logid:f462f295) The Host stats is null for host: 170 2024-07-23 12:48:55,452 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-c81ba658) (logid:bd789796) Seq 77-3187422636271468880: Timed out on null 2024-07-23 12:48:55,452 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-c81ba658) (logid:bd789796) Operation timed out: Commands 3187422636271468880 to Host 77 timed out after 172800 2024-07-23 12:48:55,452 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-5:ctx-c81ba658) (logid:bd789796) Unable to obtain VM statistics. 2024-07-23 12:48:55,541 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-c81ba658) (logid:bd789796) Seq 170-8501107246615232865: Timed out on null 2024-07-23 12:48:55,541 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-c81ba658) (logid:bd789796) Operation timed out: Commands 8501107246615232865 to Host 170 timed out after 172800 2024-07-23 12:48:55,541 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-5:ctx-c81ba658) (logid:bd789796) Unable to obtain VM statistics. 2024-07-23 12:49:04,041 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-a200d404) (logid:1a7fe8c3) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:49:09,489 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 170-8501107246615232866: Timed out on null 2024-07-23 12:49:09,499 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 77-3187422636271468881: Timed out on null 2024-07-23 12:49:09,508 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 74-82190693199511815: Timed out on null 2024-07-23 12:49:09,521 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 77-3187422636271468882: Timed out on null 2024-07-23 12:49:09,530 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 170-8501107246615232867: Timed out on null 2024-07-23 12:49:09,540 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 74-82190693199511816: Timed out on null 2024-07-23 12:49:09,555 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 74-82190693199511817: Timed out on null 2024-07-23 12:49:09,565 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 77-3187422636271468883: Timed out on null 2024-07-23 12:49:09,573 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 170-8501107246615232868: Timed out on null 2024-07-23 12:49:09,589 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 77-3187422636271468884: Timed out on null 2024-07-23 12:49:09,598 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 74-82190693199511818: Timed out on null 2024-07-23 12:49:09,607 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 170-8501107246615232869: Timed out on null 2024-07-23 12:49:09,619 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 170-8501107246615232870: Timed out on null 2024-07-23 12:49:09,628 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 74-82190693199511819: Timed out on null 2024-07-23 12:49:09,638 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-5b4eba45) (logid:315e6dcb) Seq 77-3187422636271468885: Timed out on null 2024-07-23 12:49:33,782 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-aa34ff3c) (logid:e1fd920b) Seq 77-3187422636271468886: Timed out on null 2024-07-23 12:49:33,782 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-aa34ff3c) (logid:e1fd920b) Operation timed out: Commands 3187422636271468886 to Host 77 timed out after 172800 2024-07-23 12:49:33,782 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-4:ctx-aa34ff3c) (logid:e1fd920b) Unable to obtain VM network statistics. 2024-07-23 12:49:33,847 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-aa34ff3c) (logid:e1fd920b) Seq 170-8501107246615232871: Timed out on null 2024-07-23 12:49:33,848 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-aa34ff3c) (logid:e1fd920b) Operation timed out: Commands 8501107246615232871 to Host 170 timed out after 172800 2024-07-23 12:49:33,848 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-4:ctx-aa34ff3c) (logid:e1fd920b) Unable to obtain VM network statistics. 2024-07-23 12:49:39,439 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-89:ctx-2bf30748) (logid:dd025da3) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:49:39,636 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-90:ctx-e71c3cb3) (logid:4323fafb) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:49:39,880 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-91:ctx-67526ba6) (logid:b14219d0) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:49:39,897 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-92:ctx-4002a251) (logid:657202f4) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:49:42,611 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-89:ctx-2bf30748) (logid:dd025da3) Monitor StoragePoolMonitor says there is an error in the connect process for 248 due to Unable to connect host 248 to storage pool id 152 due to com.cloud.utils.exception.CloudRuntimeException: Unable to get an answer to the modify storage pool command152 2024-07-23 12:49:42,631 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-89:ctx-2bf30748) (logid:dd025da3) Unable to connect due to 2024-07-23 12:49:43,039 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-90:ctx-e71c3cb3) (logid:4323fafb) Monitor StoragePoolMonitor says there is an error in the connect process for 254 due to Unable to connect host 254 to storage pool id 152 due to com.cloud.utils.exception.CloudRuntimeException: Unable to get an answer to the modify storage pool command152 2024-07-23 12:49:43,060 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-90:ctx-e71c3cb3) (logid:4323fafb) Unable to connect due to 2024-07-23 12:49:43,355 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-92:ctx-4002a251) (logid:657202f4) Monitor StoragePoolMonitor says there is an error in the connect process for 260 due to Unable to connect host 260 to storage pool id 152 due to com.cloud.utils.exception.CloudRuntimeException: Unable to get an answer to the modify storage pool command152 2024-07-23 12:49:43,380 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-92:ctx-4002a251) (logid:657202f4) Unable to connect due to 2024-07-23 12:49:43,401 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-91:ctx-67526ba6) (logid:b14219d0) Monitor StoragePoolMonitor says there is an error in the connect process for 257 due to Unable to connect host 257 to storage pool id 152 due to com.cloud.utils.exception.CloudRuntimeException: Unable to get an answer to the modify storage pool command152 2024-07-23 12:49:43,424 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-91:ctx-67526ba6) (logid:b14219d0) Unable to connect due to 2024-07-23 12:49:43,550 WARN [c.c.u.d.GlobalLock] (SpringContextShutdownHook:null) (logid:) Unmatched Global lock secstorage.scan.lock reference usage detected, check your code! 2024-07-23 12:49:43,550 WARN [c.c.u.d.GlobalLock] (SpringContextShutdownHook:null) (logid:) Unmatched Global lock secStorageVm.alloc reference usage detected, check your code! 2024-07-23 12:49:43,555 WARN [c.c.u.d.GlobalLock] (SpringContextShutdownHook:null) (logid:) Unmatched Global lock consoleproxy.scan.lock reference usage detected, check your code! 2024-07-23 12:49:43,555 WARN [c.c.u.d.GlobalLock] (SpringContextShutdownHook:null) (logid:) Unmatched Global lock consoleproxy.alloc reference usage detected, check your code! 2024-07-23 12:50:11,750 WARN [c.c.s.d.DownloadMonitorImpl] (main:null) (logid:) Only realhostip.com ssl cert is supported, ignoring self-signed and other certs 2024-07-23 12:50:23,947 WARN [c.c.a.AlertManagerImpl] (Cluster-Notification-1:ctx-b7dc5179) (logid:7a296679) AlertType:: 14 | dataCenterId:: 0 | podId:: 0 | clusterId:: null | message:: Management server node 10.10.10.11 is up 2024-07-23 12:50:24,018 WARN [c.c.c.ClusterManagerImpl] (Cluster-Notification-1:ctx-b7dc5179) (logid:7a296679) Notifying management server join event took 87 ms 2024-07-23 12:50:27,394 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-1:ctx-a0261ab7) (logid:48ebda4b) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:50:27,467 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-2:ctx-108df46d) (logid:dcb8b5d7) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:50:27,692 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-3:ctx-6f71596b) (logid:d976c278) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:50:27,710 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-4:ctx-d1e37186) (logid:62dcf805) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:50:32,802 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-1:ctx-a0261ab7) (logid:48ebda4b) Unable to connect due to 2024-07-23 12:50:33,801 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-2:ctx-108df46d) (logid:dcb8b5d7) Unable to connect due to 2024-07-23 12:50:34,126 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-4:ctx-d1e37186) (logid:62dcf805) Unable to connect due to 2024-07-23 12:50:34,258 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-3:ctx-6f71596b) (logid:d976c278) Unable to connect due to 2024-07-23 12:50:36,401 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-1da0fe13) (logid:55e9ece8) Seq 77-8169811199026790401: Timed out on null 2024-07-23 12:50:36,401 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 170-5956291982174191617: Timed out on null 2024-07-23 12:50:36,401 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Seq 18-2597451085085933569: Timed out on null 2024-07-23 12:50:36,401 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Seq 77-8169811199026790402: Timed out on null 2024-07-23 12:50:36,402 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Operation timed out: Commands 2597451085085933569 to Host 18 timed out after 172800 2024-07-23 12:50:36,403 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Operation timed out: Commands 8169811199026790402 to Host 77 timed out after 240 2024-07-23 12:50:36,403 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-1da0fe13) (logid:55e9ece8) Operation timed out: Commands 8169811199026790401 to Host 77 timed out after 172800 2024-07-23 12:50:36,403 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Unable to obtain host 18 statistics. 2024-07-23 12:50:36,403 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-1da0fe13) (logid:55e9ece8) Unable to obtain VM statistics. 2024-07-23 12:50:36,404 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-77748330) (logid:33ee9741) The Host stats is null for host: 18 2024-07-23 12:50:36,418 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 77-8169811199026790403: Timed out on null 2024-07-23 12:50:36,443 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 74-5907596811203248129: Timed out on null 2024-07-23 12:50:36,445 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Seq 74-5907596811203248130: Timed out on null 2024-07-23 12:50:36,446 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Operation timed out: Commands 5907596811203248130 to Host 74 timed out after 172800 2024-07-23 12:50:36,446 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Unable to obtain host 74 statistics. 2024-07-23 12:50:36,446 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-77748330) (logid:33ee9741) The Host stats is null for host: 74 2024-07-23 12:50:36,461 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 74-5907596811203248131: Timed out on null 2024-07-23 12:50:36,474 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 170-5956291982174191618: Timed out on null 2024-07-23 12:50:36,475 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Seq 77-8169811199026790404: Timed out on null 2024-07-23 12:50:36,475 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Operation timed out: Commands 8169811199026790404 to Host 77 timed out after 172800 2024-07-23 12:50:36,475 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Unable to obtain host 77 statistics. 2024-07-23 12:50:36,475 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-77748330) (logid:33ee9741) The Host stats is null for host: 77 2024-07-23 12:50:36,486 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Seq 170-5956291982174191619: Timed out on null 2024-07-23 12:50:36,487 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Operation timed out: Commands 5956291982174191619 to Host 170 timed out after 240 2024-07-23 12:50:36,488 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 77-8169811199026790405: Timed out on null 2024-07-23 12:50:36,507 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 74-5907596811203248132: Timed out on null 2024-07-23 12:50:36,508 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Seq 170-5956291982174191620: Timed out on null 2024-07-23 12:50:36,509 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Operation timed out: Commands 5956291982174191620 to Host 170 timed out after 172800 2024-07-23 12:50:36,509 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-77748330) (logid:33ee9741) Unable to obtain host 170 statistics. 2024-07-23 12:50:36,509 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-77748330) (logid:33ee9741) The Host stats is null for host: 170 2024-07-23 12:50:36,520 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 77-8169811199026790406: Timed out on null 2024-07-23 12:50:36,530 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-1da0fe13) (logid:55e9ece8) Seq 170-5956291982174191621: Timed out on null 2024-07-23 12:50:36,530 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-1da0fe13) (logid:55e9ece8) Operation timed out: Commands 5956291982174191621 to Host 170 timed out after 172800 2024-07-23 12:50:36,531 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-1da0fe13) (logid:55e9ece8) Unable to obtain VM statistics. 2024-07-23 12:50:36,533 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 170-5956291982174191622: Timed out on null 2024-07-23 12:50:36,553 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 74-5907596811203248133: Timed out on null 2024-07-23 12:50:36,567 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 77-8169811199026790407: Timed out on null 2024-07-23 12:50:36,578 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 170-5956291982174191623: Timed out on null 2024-07-23 12:50:36,590 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Seq 170-5956291982174191624: Timed out on null 2024-07-23 12:50:36,590 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Operation timed out: Commands 5956291982174191624 to Host 170 timed out after 240 2024-07-23 12:50:36,594 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 74-5907596811203248134: Timed out on null 2024-07-23 12:50:36,605 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 77-8169811199026790408: Timed out on null 2024-07-23 12:50:36,616 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-686ab399) (logid:6322172f) Seq 170-5956291982174191625: Timed out on null 2024-07-23 12:50:36,699 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Seq 170-5956291982174191626: Timed out on null 2024-07-23 12:50:36,699 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Operation timed out: Commands 5956291982174191626 to Host 170 timed out after 240 2024-07-23 12:50:36,752 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Seq 77-8169811199026790409: Timed out on null 2024-07-23 12:50:36,752 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Operation timed out: Commands 8169811199026790409 to Host 77 timed out after 240 2024-07-23 12:50:36,818 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Seq 170-5956291982174191627: Timed out on null 2024-07-23 12:50:36,818 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Operation timed out: Commands 5956291982174191627 to Host 170 timed out after 240 2024-07-23 12:50:36,903 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Seq 170-5956291982174191628: Timed out on null 2024-07-23 12:50:36,903 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-a3fb5a0c) (logid:603966f2) Operation timed out: Commands 5956291982174191628 to Host 170 timed out after 240 2024-07-23 12:50:48,961 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-1:ctx-625b0cc1) (logid:2abe9268) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: 80745d5a-8359-488f-b225-fc09670f7fa7 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:50:51,402 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-151ef859) (logid:21052d7c) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:51:05,875 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-1:ctx-36fb0b56) (logid:4b755a2e) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: e51131e3-39af-4749-8a3a-c5a32de583d0 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:05,909 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-107abf64) (logid:d7925f2b) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: 30feaccd-5efe-4b1a-b636-1d3831604ab4 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:06,022 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-0a4277db) (logid:22736af0) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: 80745d5a-8359-488f-b225-fc09670f7fa7 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:06,130 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-70dc58c0) (logid:b82fd5e7) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: a51ae863-0e4b-4204-8b00-d5fc70f07e13 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:06,322 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-49ad1cf2) (logid:e424ef32) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: 1bc63305-d743-4f18-98f5-8a75a5333653 with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:06,556 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-b28d85ca) (logid:ea60a1f1) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: e31467df-3857-41a4-a21e-22f0dc7008df with error: HTTP Server returned 404 (expected 200 OK) 2024-07-23 12:51:16,281 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-a8a51b7a) (logid:60ebd03c) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: 9f402c90-f381-4bc6-a161-4b22a525ae63 with error: No route to host (Host unreachable) 2024-07-23 12:51:31,105 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-5fcbbff5) (logid:4f7997ec) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: 9f402c90-f381-4bc6-a161-4b22a525ae63 with error: No route to host (Host unreachable) 2024-07-23 12:51:36,543 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Seq 18-2597451085085933570: Timed out on null 2024-07-23 12:51:36,543 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Operation timed out: Commands 2597451085085933570 to Host 18 timed out after 172800 2024-07-23 12:51:36,543 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Unable to obtain host 18 statistics. 2024-07-23 12:51:36,544 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) The Host stats is null for host: 18 2024-07-23 12:51:36,571 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Seq 74-5907596811203248135: Timed out on null 2024-07-23 12:51:36,571 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Operation timed out: Commands 5907596811203248135 to Host 74 timed out after 172800 2024-07-23 12:51:36,571 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Unable to obtain host 74 statistics. 2024-07-23 12:51:36,572 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) The Host stats is null for host: 74 2024-07-23 12:51:36,592 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-5f0b91f0) (logid:94dba04d) Seq 77-8169811199026790410: Timed out on null 2024-07-23 12:51:36,593 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-5f0b91f0) (logid:94dba04d) Operation timed out: Commands 8169811199026790410 to Host 77 timed out after 172800 2024-07-23 12:51:36,593 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-1:ctx-5f0b91f0) (logid:94dba04d) Unable to obtain VM statistics. 2024-07-23 12:51:36,599 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Seq 77-8169811199026790411: Timed out on null 2024-07-23 12:51:36,600 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Operation timed out: Commands 8169811199026790411 to Host 77 timed out after 172800 2024-07-23 12:51:36,600 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Unable to obtain host 77 statistics. 2024-07-23 12:51:36,600 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) The Host stats is null for host: 77 2024-07-23 12:51:36,627 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Seq 170-5956291982174191629: Timed out on null 2024-07-23 12:51:36,627 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Operation timed out: Commands 5956291982174191629 to Host 170 timed out after 172800 2024-07-23 12:51:36,627 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) Unable to obtain host 170 statistics. 2024-07-23 12:51:36,627 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-4ec1a0a2) (logid:d81bc9bc) The Host stats is null for host: 170 2024-07-23 12:51:36,687 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-5f0b91f0) (logid:94dba04d) Seq 170-5956291982174191630: Timed out on null 2024-07-23 12:51:36,687 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-5f0b91f0) (logid:94dba04d) Operation timed out: Commands 5956291982174191630 to Host 170 timed out after 172800 2024-07-23 12:51:36,687 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-1:ctx-5f0b91f0) (logid:94dba04d) Unable to obtain VM statistics. 2024-07-23 12:51:36,935 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 77-8169811199026790412: Timed out on null 2024-07-23 12:51:36,947 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 74-5907596811203248136: Timed out on null 2024-07-23 12:51:36,958 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 170-5956291982174191631: Timed out on null 2024-07-23 12:51:36,987 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 170-5956291982174191632: Timed out on null 2024-07-23 12:51:36,998 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 74-5907596811203248137: Timed out on null 2024-07-23 12:51:37,008 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 77-8169811199026790413: Timed out on null 2024-07-23 12:51:37,024 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 170-5956291982174191633: Timed out on null 2024-07-23 12:51:37,036 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 74-5907596811203248138: Timed out on null 2024-07-23 12:51:37,048 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 77-8169811199026790414: Timed out on null 2024-07-23 12:51:37,068 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 74-5907596811203248139: Timed out on null 2024-07-23 12:51:37,080 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 170-5956291982174191634: Timed out on null 2024-07-23 12:51:37,090 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 77-8169811199026790415: Timed out on null 2024-07-23 12:51:37,104 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 74-5907596811203248140: Timed out on null 2024-07-23 12:51:37,113 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 77-8169811199026790416: Timed out on null 2024-07-23 12:51:37,124 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-ece3fe51) (logid:190a25ed) Seq 170-5956291982174191635: Timed out on null 2024-07-23 12:51:41,275 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-24c2c462) (logid:cfc2ffea) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: 7163a1aa-344b-4afa-bb2b-5bfa182b53d2 with error: No route to host (Host unreachable) 2024-07-23 12:51:51,397 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-b295000c) (logid:0c47527f) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:52:21,052 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-4d492bb8) (logid:899d246d) Seq 77-8169811199026790417: Timed out on null 2024-07-23 12:52:21,052 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-4d492bb8) (logid:899d246d) Operation timed out: Commands 8169811199026790417 to Host 77 timed out after 172800 2024-07-23 12:52:21,052 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-3:ctx-4d492bb8) (logid:899d246d) Unable to obtain VM network statistics. 2024-07-23 12:52:21,122 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-4d492bb8) (logid:899d246d) Seq 170-5956291982174191636: Timed out on null 2024-07-23 12:52:21,123 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-4d492bb8) (logid:899d246d) Operation timed out: Commands 5956291982174191636 to Host 170 timed out after 172800 2024-07-23 12:52:21,123 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-3:ctx-4d492bb8) (logid:899d246d) Unable to obtain VM network statistics. 2024-07-23 12:52:23,457 WARN [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-2:ctx-1389f748) (logid:b8306c21) AlertType:: 28 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: Failed to register template: 7163a1aa-344b-4afa-bb2b-5bfa182b53d2 with error: No route to host (Host unreachable) 2024-07-23 12:52:26,931 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-5:ctx-86f1dfff) (logid:e29bf016) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:52:27,122 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-6:ctx-f4397018) (logid:383b301a) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:52:27,416 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-8:ctx-77a2b38a) (logid:21a72f49) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:52:27,561 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-7:ctx-31c90c8b) (logid:2873a550) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:52:31,604 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-5:ctx-86f1dfff) (logid:e29bf016) Unable to connect due to 2024-07-23 12:52:32,928 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-6:ctx-f4397018) (logid:383b301a) Unable to connect due to 2024-07-23 12:52:33,611 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-8:ctx-77a2b38a) (logid:21a72f49) Unable to connect due to 2024-07-23 12:52:34,265 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-7:ctx-31c90c8b) (logid:2873a550) Unable to connect due to 2024-07-23 12:52:36,672 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Seq 18-2597451085085933571: Timed out on null 2024-07-23 12:52:36,672 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Operation timed out: Commands 2597451085085933571 to Host 18 timed out after 172800 2024-07-23 12:52:36,672 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Unable to obtain host 18 statistics. 2024-07-23 12:52:36,672 WARN [c.c.s.StatsCollector] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) The Host stats is null for host: 18 2024-07-23 12:52:36,706 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Seq 74-5907596811203248141: Timed out on null 2024-07-23 12:52:36,706 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Operation timed out: Commands 5907596811203248141 to Host 74 timed out after 172800 2024-07-23 12:52:36,706 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Unable to obtain host 74 statistics. 2024-07-23 12:52:36,707 WARN [c.c.s.StatsCollector] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) The Host stats is null for host: 74 2024-07-23 12:52:36,740 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Seq 77-8169811199026790418: Timed out on null 2024-07-23 12:52:36,740 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Operation timed out: Commands 8169811199026790418 to Host 77 timed out after 172800 2024-07-23 12:52:36,741 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Unable to obtain host 77 statistics. 2024-07-23 12:52:36,741 WARN [c.c.s.StatsCollector] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) The Host stats is null for host: 77 2024-07-23 12:52:36,751 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-d0da15e4) (logid:eacf0074) Seq 77-8169811199026790419: Timed out on null 2024-07-23 12:52:36,752 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-d0da15e4) (logid:eacf0074) Operation timed out: Commands 8169811199026790419 to Host 77 timed out after 172800 2024-07-23 12:52:36,752 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-4:ctx-d0da15e4) (logid:eacf0074) Unable to obtain VM statistics. 2024-07-23 12:52:36,768 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Seq 170-5956291982174191637: Timed out on null 2024-07-23 12:52:36,768 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Operation timed out: Commands 5956291982174191637 to Host 170 timed out after 172800 2024-07-23 12:52:36,769 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) Unable to obtain host 170 statistics. 2024-07-23 12:52:36,769 WARN [c.c.s.StatsCollector] (StatsCollector-6:ctx-d96f1974) (logid:bc6c3f63) The Host stats is null for host: 170 2024-07-23 12:52:36,846 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-d0da15e4) (logid:eacf0074) Seq 170-5956291982174191638: Timed out on null 2024-07-23 12:52:36,846 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-d0da15e4) (logid:eacf0074) Operation timed out: Commands 5956291982174191638 to Host 170 timed out after 172800 2024-07-23 12:52:36,846 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-4:ctx-d0da15e4) (logid:eacf0074) Unable to obtain VM statistics. 2024-07-23 12:52:37,461 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 77-8169811199026790420: Timed out on null 2024-07-23 12:52:37,484 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 170-5956291982174191639: Timed out on null 2024-07-23 12:52:37,497 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 74-5907596811203248142: Timed out on null 2024-07-23 12:52:37,517 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 77-8169811199026790421: Timed out on null 2024-07-23 12:52:37,531 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 74-5907596811203248143: Timed out on null 2024-07-23 12:52:37,543 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 170-5956291982174191640: Timed out on null 2024-07-23 12:52:37,559 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 170-5956291982174191641: Timed out on null 2024-07-23 12:52:37,569 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 77-8169811199026790422: Timed out on null 2024-07-23 12:52:37,578 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 74-5907596811203248144: Timed out on null 2024-07-23 12:52:37,592 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 170-5956291982174191642: Timed out on null 2024-07-23 12:52:37,620 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 77-8169811199026790423: Timed out on null 2024-07-23 12:52:37,641 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 74-5907596811203248145: Timed out on null 2024-07-23 12:52:37,664 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 170-5956291982174191643: Timed out on null 2024-07-23 12:52:37,676 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 77-8169811199026790424: Timed out on null 2024-07-23 12:52:37,691 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-2e3c19cf) (logid:9553c22c) Seq 74-5907596811203248146: Timed out on null 2024-07-23 12:52:51,417 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-9ed2899e) (logid:097abe87) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:53:36,804 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Seq 18-2597451085085933572: Timed out on null 2024-07-23 12:53:36,804 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Operation timed out: Commands 2597451085085933572 to Host 18 timed out after 172800 2024-07-23 12:53:36,804 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Unable to obtain host 18 statistics. 2024-07-23 12:53:36,804 WARN [c.c.s.StatsCollector] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) The Host stats is null for host: 18 2024-07-23 12:53:36,832 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Seq 74-5907596811203248147: Timed out on null 2024-07-23 12:53:36,832 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Operation timed out: Commands 5907596811203248147 to Host 74 timed out after 172800 2024-07-23 12:53:36,832 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Unable to obtain host 74 statistics. 2024-07-23 12:53:36,832 WARN [c.c.s.StatsCollector] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) The Host stats is null for host: 74 2024-07-23 12:53:36,861 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Seq 77-8169811199026790425: Timed out on null 2024-07-23 12:53:36,861 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Operation timed out: Commands 8169811199026790425 to Host 77 timed out after 172800 2024-07-23 12:53:36,861 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Unable to obtain host 77 statistics. 2024-07-23 12:53:36,861 WARN [c.c.s.StatsCollector] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) The Host stats is null for host: 77 2024-07-23 12:53:36,890 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Seq 170-5956291982174191644: Timed out on null 2024-07-23 12:53:36,890 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Operation timed out: Commands 5956291982174191644 to Host 170 timed out after 172800 2024-07-23 12:53:36,890 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) Unable to obtain host 170 statistics. 2024-07-23 12:53:36,890 WARN [c.c.s.StatsCollector] (StatsCollector-3:ctx-69eec79b) (logid:2d5e0b91) The Host stats is null for host: 170 2024-07-23 12:53:36,906 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-20b9ae0b) (logid:a88b78e3) Seq 77-8169811199026790426: Timed out on null 2024-07-23 12:53:36,906 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-20b9ae0b) (logid:a88b78e3) Operation timed out: Commands 8169811199026790426 to Host 77 timed out after 172800 2024-07-23 12:53:36,906 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-5:ctx-20b9ae0b) (logid:a88b78e3) Unable to obtain VM statistics. 2024-07-23 12:53:37,006 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-20b9ae0b) (logid:a88b78e3) Seq 170-5956291982174191645: Timed out on null 2024-07-23 12:53:37,006 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-20b9ae0b) (logid:a88b78e3) Operation timed out: Commands 5956291982174191645 to Host 170 timed out after 172800 2024-07-23 12:53:37,006 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-5:ctx-20b9ae0b) (logid:a88b78e3) Unable to obtain VM statistics. 2024-07-23 12:53:38,019 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 74-5907596811203248148: Timed out on null 2024-07-23 12:53:38,029 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 77-8169811199026790427: Timed out on null 2024-07-23 12:53:38,040 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 170-5956291982174191646: Timed out on null 2024-07-23 12:53:38,054 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 74-5907596811203248149: Timed out on null 2024-07-23 12:53:38,064 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 170-5956291982174191647: Timed out on null 2024-07-23 12:53:38,074 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 77-8169811199026790428: Timed out on null 2024-07-23 12:53:38,089 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 77-8169811199026790429: Timed out on null 2024-07-23 12:53:38,098 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 170-5956291982174191648: Timed out on null 2024-07-23 12:53:38,108 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 74-5907596811203248150: Timed out on null 2024-07-23 12:53:38,123 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 74-5907596811203248151: Timed out on null 2024-07-23 12:53:38,133 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 170-5956291982174191649: Timed out on null 2024-07-23 12:53:38,143 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 77-8169811199026790430: Timed out on null 2024-07-23 12:53:38,160 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 77-8169811199026790431: Timed out on null 2024-07-23 12:53:38,171 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 170-5956291982174191650: Timed out on null 2024-07-23 12:53:38,181 WARN [c.c.a.m.AgentAttache] (StatsCollector-6:ctx-ddbb7aaf) (logid:979182ad) Seq 74-5907596811203248152: Timed out on null 2024-07-23 12:53:51,373 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-e45a9c36) (logid:6ab275d6) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:54:21,041 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-7bd66ef3) (logid:ee703965) Seq 77-8169811199026790432: Timed out on null 2024-07-23 12:54:21,041 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-7bd66ef3) (logid:ee703965) Operation timed out: Commands 8169811199026790432 to Host 77 timed out after 172800 2024-07-23 12:54:21,041 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-4:ctx-7bd66ef3) (logid:ee703965) Unable to obtain VM network statistics. 2024-07-23 12:54:21,114 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-7bd66ef3) (logid:ee703965) Seq 170-5956291982174191651: Timed out on null 2024-07-23 12:54:21,114 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-7bd66ef3) (logid:ee703965) Operation timed out: Commands 5956291982174191651 to Host 170 timed out after 172800 2024-07-23 12:54:21,114 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-4:ctx-7bd66ef3) (logid:ee703965) Unable to obtain VM network statistics. 2024-07-23 12:54:27,675 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-9:ctx-75be1d9b) (logid:56e2daed) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:54:27,833 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-10:ctx-f5b43b18) (logid:d234b7a5) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:54:28,071 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-11:ctx-4710e6da) (logid:c1682411) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:54:28,113 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-12:ctx-d2dc7f4d) (logid:4aa1ca2f) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:54:32,266 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-9:ctx-75be1d9b) (logid:56e2daed) Unable to connect due to 2024-07-23 12:54:33,581 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-10:ctx-f5b43b18) (logid:d234b7a5) Unable to connect due to 2024-07-23 12:54:34,228 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-12:ctx-d2dc7f4d) (logid:4aa1ca2f) Unable to connect due to 2024-07-23 12:54:34,332 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-11:ctx-4710e6da) (logid:c1682411) Unable to connect due to 2024-07-23 12:54:36,921 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Seq 18-2597451085085933573: Timed out on null 2024-07-23 12:54:36,921 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Operation timed out: Commands 2597451085085933573 to Host 18 timed out after 172800 2024-07-23 12:54:36,921 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Unable to obtain host 18 statistics. 2024-07-23 12:54:36,921 WARN [c.c.s.StatsCollector] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) The Host stats is null for host: 18 2024-07-23 12:54:36,943 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Seq 74-5907596811203248153: Timed out on null 2024-07-23 12:54:36,943 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Operation timed out: Commands 5907596811203248153 to Host 74 timed out after 172800 2024-07-23 12:54:36,943 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Unable to obtain host 74 statistics. 2024-07-23 12:54:36,943 WARN [c.c.s.StatsCollector] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) The Host stats is null for host: 74 2024-07-23 12:54:36,969 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Seq 77-8169811199026790433: Timed out on null 2024-07-23 12:54:36,969 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Operation timed out: Commands 8169811199026790433 to Host 77 timed out after 172800 2024-07-23 12:54:36,969 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Unable to obtain host 77 statistics. 2024-07-23 12:54:36,969 WARN [c.c.s.StatsCollector] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) The Host stats is null for host: 77 2024-07-23 12:54:36,994 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Seq 170-5956291982174191652: Timed out on null 2024-07-23 12:54:36,994 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Operation timed out: Commands 5956291982174191652 to Host 170 timed out after 172800 2024-07-23 12:54:36,995 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) Unable to obtain host 170 statistics. 2024-07-23 12:54:36,995 WARN [c.c.s.StatsCollector] (StatsCollector-2:ctx-a09ed5be) (logid:5a200921) The Host stats is null for host: 170 2024-07-23 12:54:37,062 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-f3266c0f) (logid:0ad5fdef) Seq 77-8169811199026790434: Timed out on null 2024-07-23 12:54:37,062 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-f3266c0f) (logid:0ad5fdef) Operation timed out: Commands 8169811199026790434 to Host 77 timed out after 172800 2024-07-23 12:54:37,062 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-3:ctx-f3266c0f) (logid:0ad5fdef) Unable to obtain VM statistics. 2024-07-23 12:54:37,148 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-f3266c0f) (logid:0ad5fdef) Seq 170-5956291982174191653: Timed out on null 2024-07-23 12:54:37,148 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-3:ctx-f3266c0f) (logid:0ad5fdef) Operation timed out: Commands 5956291982174191653 to Host 170 timed out after 172800 2024-07-23 12:54:37,148 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-3:ctx-f3266c0f) (logid:0ad5fdef) Unable to obtain VM statistics. 2024-07-23 12:54:38,494 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 77-8169811199026790435: Timed out on null 2024-07-23 12:54:38,503 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 74-5907596811203248154: Timed out on null 2024-07-23 12:54:38,512 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 170-5956291982174191654: Timed out on null 2024-07-23 12:54:38,524 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 77-8169811199026790436: Timed out on null 2024-07-23 12:54:38,534 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 170-5956291982174191655: Timed out on null 2024-07-23 12:54:38,542 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 74-5907596811203248155: Timed out on null 2024-07-23 12:54:38,556 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 170-5956291982174191656: Timed out on null 2024-07-23 12:54:38,565 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 74-5907596811203248156: Timed out on null 2024-07-23 12:54:38,574 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 77-8169811199026790437: Timed out on null 2024-07-23 12:54:38,588 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 77-8169811199026790438: Timed out on null 2024-07-23 12:54:38,597 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 170-5956291982174191657: Timed out on null 2024-07-23 12:54:38,606 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 74-5907596811203248157: Timed out on null 2024-07-23 12:54:38,619 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 77-8169811199026790439: Timed out on null 2024-07-23 12:54:38,629 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 170-5956291982174191658: Timed out on null 2024-07-23 12:54:38,637 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-ff05f0fc) (logid:062da384) Seq 74-5907596811203248158: Timed out on null 2024-07-23 12:54:52,689 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-d9cba012) (logid:5542d4d7) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:55:37,029 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Seq 18-2597451085085933574: Timed out on null 2024-07-23 12:55:37,029 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Operation timed out: Commands 2597451085085933574 to Host 18 timed out after 172800 2024-07-23 12:55:37,029 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Unable to obtain host 18 statistics. 2024-07-23 12:55:37,029 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) The Host stats is null for host: 18 2024-07-23 12:55:37,057 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Seq 74-5907596811203248159: Timed out on null 2024-07-23 12:55:37,057 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Operation timed out: Commands 5907596811203248159 to Host 74 timed out after 172800 2024-07-23 12:55:37,057 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Unable to obtain host 74 statistics. 2024-07-23 12:55:37,057 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) The Host stats is null for host: 74 2024-07-23 12:55:37,084 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Seq 77-8169811199026790440: Timed out on null 2024-07-23 12:55:37,084 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Operation timed out: Commands 8169811199026790440 to Host 77 timed out after 172800 2024-07-23 12:55:37,084 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Unable to obtain host 77 statistics. 2024-07-23 12:55:37,084 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) The Host stats is null for host: 77 2024-07-23 12:55:37,110 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Seq 170-5956291982174191659: Timed out on null 2024-07-23 12:55:37,110 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Operation timed out: Commands 5956291982174191659 to Host 170 timed out after 172800 2024-07-23 12:55:37,110 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) Unable to obtain host 170 statistics. 2024-07-23 12:55:37,110 WARN [c.c.s.StatsCollector] (StatsCollector-1:ctx-4b03c221) (logid:8ba482f8) The Host stats is null for host: 170 2024-07-23 12:55:37,205 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b658797a) (logid:d9150295) Seq 77-8169811199026790441: Timed out on null 2024-07-23 12:55:37,205 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-b658797a) (logid:d9150295) Operation timed out: Commands 8169811199026790441 to Host 77 timed out after 172800 2024-07-23 12:55:37,205 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-b658797a) (logid:d9150295) Unable to obtain VM statistics. 2024-07-23 12:55:37,300 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-b658797a) (logid:d9150295) Seq 170-5956291982174191660: Timed out on null 2024-07-23 12:55:37,300 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-b658797a) (logid:d9150295) Operation timed out: Commands 5956291982174191660 to Host 170 timed out after 172800 2024-07-23 12:55:37,301 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-2:ctx-b658797a) (logid:d9150295) Unable to obtain VM statistics. 2024-07-23 12:55:38,948 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 77-8169811199026790442: Timed out on null 2024-07-23 12:55:38,957 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 170-5956291982174191661: Timed out on null 2024-07-23 12:55:38,965 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 74-5907596811203248160: Timed out on null 2024-07-23 12:55:38,979 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 170-5956291982174191662: Timed out on null 2024-07-23 12:55:38,989 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 77-8169811199026790443: Timed out on null 2024-07-23 12:55:38,999 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 74-5907596811203248161: Timed out on null 2024-07-23 12:55:39,015 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 77-8169811199026790444: Timed out on null 2024-07-23 12:55:39,025 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 170-5956291982174191663: Timed out on null 2024-07-23 12:55:39,035 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 74-5907596811203248162: Timed out on null 2024-07-23 12:55:39,049 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 74-5907596811203248163: Timed out on null 2024-07-23 12:55:39,059 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 170-5956291982174191664: Timed out on null 2024-07-23 12:55:39,068 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 77-8169811199026790445: Timed out on null 2024-07-23 12:55:39,082 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 74-5907596811203248164: Timed out on null 2024-07-23 12:55:39,092 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 170-5956291982174191665: Timed out on null 2024-07-23 12:55:39,102 WARN [c.c.a.m.AgentAttache] (StatsCollector-3:ctx-31eb1df1) (logid:c1f3245d) Seq 77-8169811199026790446: Timed out on null 2024-07-23 12:55:51,442 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-7c336beb) (logid:9ae32c87) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC 2024-07-23 12:56:21,041 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-7c85a26e) (logid:c6a24d77) Seq 77-8169811199026790447: Timed out on null 2024-07-23 12:56:21,041 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-7c85a26e) (logid:c6a24d77) Operation timed out: Commands 8169811199026790447 to Host 77 timed out after 172800 2024-07-23 12:56:21,041 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-5:ctx-7c85a26e) (logid:c6a24d77) Unable to obtain VM network statistics. 2024-07-23 12:56:21,101 WARN [c.c.a.m.AgentAttache] (StatsCollector-5:ctx-7c85a26e) (logid:c6a24d77) Seq 170-5956291982174191666: Timed out on null 2024-07-23 12:56:21,101 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-5:ctx-7c85a26e) (logid:c6a24d77) Operation timed out: Commands 5956291982174191666 to Host 170 timed out after 172800 2024-07-23 12:56:21,101 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-5:ctx-7c85a26e) (logid:c6a24d77) Unable to obtain VM network statistics. 2024-07-23 12:56:26,887 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-13:ctx-3856c228) (logid:bedc7c1b) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:56:27,088 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-14:ctx-cb15223c) (logid:7c0c1d3a) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:56:27,269 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-15:ctx-bfff22e0) (logid:714ab2ea) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:56:27,374 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-16:ctx-a0f4ebc6) (logid:65af0757) defaulting to xenserver650 resource for product brand: XCP-ng with product version: 8.1.0 2024-07-23 12:56:31,504 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-13:ctx-3856c228) (logid:bedc7c1b) Unable to connect due to 2024-07-23 12:56:32,795 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-14:ctx-cb15223c) (logid:7c0c1d3a) Unable to connect due to 2024-07-23 12:56:33,508 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-16:ctx-a0f4ebc6) (logid:65af0757) Unable to connect due to 2024-07-23 12:56:33,595 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-15:ctx-bfff22e0) (logid:714ab2ea) Unable to connect due to 2024-07-23 12:56:37,143 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Seq 18-2597451085085933575: Timed out on null 2024-07-23 12:56:37,143 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Operation timed out: Commands 2597451085085933575 to Host 18 timed out after 172800 2024-07-23 12:56:37,143 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Unable to obtain host 18 statistics. 2024-07-23 12:56:37,143 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) The Host stats is null for host: 18 2024-07-23 12:56:37,176 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Seq 74-5907596811203248165: Timed out on null 2024-07-23 12:56:37,177 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Operation timed out: Commands 5907596811203248165 to Host 74 timed out after 172800 2024-07-23 12:56:37,177 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Unable to obtain host 74 statistics. 2024-07-23 12:56:37,177 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) The Host stats is null for host: 74 2024-07-23 12:56:37,203 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Seq 77-8169811199026790448: Timed out on null 2024-07-23 12:56:37,203 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Operation timed out: Commands 8169811199026790448 to Host 77 timed out after 172800 2024-07-23 12:56:37,203 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Unable to obtain host 77 statistics. 2024-07-23 12:56:37,203 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) The Host stats is null for host: 77 2024-07-23 12:56:37,228 WARN [c.c.a.m.AgentAttache] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Seq 170-5956291982174191667: Timed out on null 2024-07-23 12:56:37,228 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Operation timed out: Commands 5956291982174191667 to Host 170 timed out after 172800 2024-07-23 12:56:37,228 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) Unable to obtain host 170 statistics. 2024-07-23 12:56:37,228 WARN [c.c.s.StatsCollector] (StatsCollector-4:ctx-caa3a01a) (logid:e13a2b22) The Host stats is null for host: 170 2024-07-23 12:56:37,354 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-61225aa8) (logid:a07c40fb) Seq 77-8169811199026790449: Timed out on null 2024-07-23 12:56:37,355 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-61225aa8) (logid:a07c40fb) Operation timed out: Commands 8169811199026790449 to Host 77 timed out after 172800 2024-07-23 12:56:37,355 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-1:ctx-61225aa8) (logid:a07c40fb) Unable to obtain VM statistics. 2024-07-23 12:56:37,454 WARN [c.c.a.m.AgentAttache] (StatsCollector-1:ctx-61225aa8) (logid:a07c40fb) Seq 170-5956291982174191668: Timed out on null 2024-07-23 12:56:37,454 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-61225aa8) (logid:a07c40fb) Operation timed out: Commands 5956291982174191668 to Host 170 timed out after 172800 2024-07-23 12:56:37,454 WARN [c.c.v.UserVmManagerImpl] (StatsCollector-1:ctx-61225aa8) (logid:a07c40fb) Unable to obtain VM statistics. 2024-07-23 12:56:39,416 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 77-8169811199026790450: Timed out on null 2024-07-23 12:56:39,426 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 74-5907596811203248166: Timed out on null 2024-07-23 12:56:39,439 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 170-5956291982174191669: Timed out on null 2024-07-23 12:56:39,453 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 170-5956291982174191670: Timed out on null 2024-07-23 12:56:39,462 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 74-5907596811203248167: Timed out on null 2024-07-23 12:56:39,471 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 77-8169811199026790451: Timed out on null 2024-07-23 12:56:39,486 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 77-8169811199026790452: Timed out on null 2024-07-23 12:56:39,495 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 74-5907596811203248168: Timed out on null 2024-07-23 12:56:39,504 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 170-5956291982174191671: Timed out on null 2024-07-23 12:56:39,517 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 77-8169811199026790453: Timed out on null 2024-07-23 12:56:39,526 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 74-5907596811203248169: Timed out on null 2024-07-23 12:56:39,535 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 170-5956291982174191672: Timed out on null 2024-07-23 12:56:39,547 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 77-8169811199026790454: Timed out on null 2024-07-23 12:56:39,556 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 74-5907596811203248170: Timed out on null 2024-07-23 12:56:39,565 WARN [c.c.a.m.AgentAttache] (StatsCollector-2:ctx-326a8c12) (logid:29c3fc96) Seq 170-5956291982174191673: Timed out on null 2024-07-23 12:56:51,373 WARN [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-14f68dfb) (logid:223ec41a) AlertType:: 24 | dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System Alert: Number of unallocated shared network IPs is low in availability zone LTC-DC
Janis On 2024-07-04 3:48, Nux wrote:
Janis,No clue, it's been a while since I used Xenserver and you are also on quite an old version as well, right? There have been many bugs fixed since 4.13.Would it be possible to include a much larger fragment from the logs or the full logs?Also, have you checked the Xcp logs, anything there, is XenCenter showing anything out of the ordinary?HTH On 2024-07-03 14:36, Janis Viklis | Files.fm wrote:If I set valid management server id, it returns to NULL after next host check cycle.I wonder could it bet somehow related to total or cluster resources. (but i tried to find and check/change all overprovisionig multipliers)2024-07-03 16:30:16,036 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-af9f7c42) (logid:31d432e5) Found 32 VMs on host 248 2024-07-03 16:30:16,039 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-af9f7c42) (logid:31d432e5) Found 0 VMs are Migrating from host 248 2024-07-03 16:30:16,138 ERROR [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-af9f7c42) (logid:31d432e5) Caught exception in recalculating capacityjava.lang.NullPointerExceptionat com.cloud.capacity.CapacityManagerImpl.updateCapacityForHost(CapacityManagerImpl.java:677) at com.cloud.alert.AlertManagerImpl.recalculateCapacity(AlertManagerImpl.java:279) at com.cloud.alert.AlertManagerImpl.checkForAlerts(AlertManagerImpl.java:432) at com.cloud.alert.AlertManagerImpl$CapacityChecker.runInContext(AlertManagerImpl.java:422) at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Janis On 2024-07-03 16:27, Nux wrote:Hello,What happens if you update the 4 problematic hosts with a valid mgmt id?On 2024-07-03 14:23, Janis Viklis | Files.fm wrote:mgmt_server_id is NULL just for those 4 hosts, other hosts ar fine. Looking at logs, cs1 management server starts to connect pools at first: 2024-07-01 16:31:29,617 DEBUG [c.c.s.l.StoragePoolMonitor] (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host 248 connected, connecting host to shared pool id 152 and sending storage pool...------------------------------------------------------------------------DB Tables: cloud.host and cloud.mshost: SELECT id, status, Type, mgmt_server_id FROM cloud.host where ID in (74,77,170, 248, 254, 257, 260) : 260 Alert Routing 257 Alert Routing 254 Alert Routing 248 Alert Routing 170 Up Routing 95534596974 77 Up Routing 95534596974 74 Up Routing 95534596974 179 95534596974 1720012401793 localhost b34f493a-42c0-47a8-ada4-04be4cdd8c49 Up 4.13.1.0 10.10.10.11 9090 2024-07-03 13:13:47 0 178 95536034244 1718828790629 cs2.failiem.lv 70420423-b362-4335-b083-8ad1342ce485 Down 4.13.1.0 10.10.10.12 9090 2024-06-19 20:39:19 1 176 95530190206 1719663483676 localhost 96a155b6-7041-48ff-9f20-268ea77c5098 Down 4.13.1.0 10.10.10.13 9090 2024-06-29 12:24:28 1 175 95536505104 1719666507512 localhost c8e6fefa-7464-4bb7-a379-5eafb55c666d Down 4.13.1.0 10.10.10.11 9090 2024-06-29 13:38:00 0 174 95534962877 1682516323955 localhost 45a057c6-6d50-41a9-bbad-cab370c01832 Down 4.13.1.0 10.10.10.11 9090 2024-06-15 08:36:06 1 172 95529749065 1658756353180 localhost 535277d3-33df-4b2a-9f1d-07f05084d473 Down 4.13.1.0 10.10.10.13 9090 2024-06-15 07:53:32 1 170 95529797928 1603725530943 localhost 5892611f-7af8-4686-8818-95ade086e6cf Down 4.13.1.0 10.10.10.13 9090 2020-11-03 04:05:40 1 167 95534560846 1658756323907 localhost e7ffd55a-77b7-4848-90de-5b5f10cc4500 Down 4.13.1.0 10.10.10.11 9090 2023-04-17 09:50:14 1 163 95534279505 1582559260879 cs1.failiem.lv 8c254697-9783-11ea-900f-00163e4db64e Down 4.11.1.0 10.10.10.11 9090 2020-05-16 14:07:09 1 161 95531601526 1582559325515 cs3.failiem.lv 8c25457e-9783-11ea-900f-00163e4db64e Down 4.11.1.0 10.10.10.13 9090 2020-05-16 14:07:21 1 Janis On 2024-07-03 13:11, Nux wrote:A shot in the dark, haven't checked the log files properly. For these hosts in the disconnected state, if you check them in the DB cloud.host table (type="Routing" btw), which mgmt_server_id are they reporting? Then check cloud.mshost table and see whether the management server with that id is in there and marked as UP etc. HTH On 2024-07-03 06:57, Janis Viklis | Files.fm wrote: (sorry, some bad formatting in previous email) Could anyone have any ideas why this error occurs and how to debug it? (248 is a host id) Monitor ComputeCapacityListener says there is an error in the connect process for 248 due to null Janis On 2024-07-01 21:44, Janis Viklis | Files.fm wrote: Hi, looking for help after 2 weeks: What could be the reason that suddenly after restarting the 4.13.1 Management server, all 4 XEN (xcp-ng 8.1) hosts of one Intel cluster disconnects and goes into "Alert state" with an error: Monitor ComputeCapacityListener says there is an error in the connect process for 248 due to null I can't find the reason for 2 weeks. The other AMD Xenserver 6.5 cluster is working just fine. Everything seems ok: network is working, I restarted: toolstack, both system vms (SSVM, consolev), one of the hosts, then removed and added back. Previously there were 3 management servers via Haproxy and Galera Mariadb, I left only one. (tried upgrade to 3.14.1, didn't help). I can manage hosts via Xencenter. There ar 5 storage pools and 3 secondary. Thanks, hoping on some clues or directions, Janis. Below is LOG output:
