Hello, I have a suspicion that the situation may be related to a mtu issue on the tunnel.
Standby. Antoine Boucher [email protected] Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system. On Mar 5, 2022, at 13:22, Antoine Boucher <[email protected]> wrote: Hi Wei, Here are my corresponding log for a disconnection event. Management server is 10.101.2.40 and the kit1-kvm01, host 15 is 10.91.1.11 <KVM Host 15 - 2022-03-05 070009 2.zip> <Management Server 2022-03-05 07:00:09.zip> Antoine Boucher [email protected] [o] +1-226-505-9734 www.haltondc.com “Data security made simple and affordable” <HDClogo7-small.png> Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system. > On Mar 4, 2022, at 2:29 PM, Wei ZHOU <[email protected]> wrote: > > Hi, > > It would be good to upload management server log as well. > > -Wei > > On Fri, 4 Mar 2022 at 19:56, Antoine Boucher <[email protected]> wrote: > >> Any one? >> >> Killing the Java processes does not recover from the situation. The Java >> processes end-up restarting and using 100% of all CPUs. >> >> Regards, >> Antoine >> >> >> >> Confidentiality Warning: This message and any attachments are intended >> only for the use of the intended recipient(s), are confidential, and may be >> privileged. If you are not the intended recipient, you are hereby notified >> that any review, retransmission, conversion to hard copy, copying, >> circulation or other use of this message and any attachments is strictly >> prohibited. If you are not the intended recipient, please notify the sender >> immediately by return e-mail, and delete this message and any attachments >> from your system. >> >> >>> On Mar 3, 2022, at 9:17 AM, Antoine Boucher <[email protected]> >> wrote: >>> >>> I have 2 zones hosted from the same management server, one larger local >> to management server zone and one smaller “remote” zone using an IPSec >> tunnel. >>> >>> All is well except that recently one of the “remote” KVM host keeps >> loosing connectivity with the management server. It used to happened >> every few days 24 but now it happens every few hours. >>> >>> Other than rebooting, I’m only able to recover by killing the Java >> process and restarting the CloudStack-Agent. >>> >>> The tunnel is monitored and used by many other processes without any >> interruptions. >>> >>> Has anyone experienced anything similar before? >>> >>> >>> Here are my host logs, the management server is 10.101.2.40 and the kvm >> host is 10.91.1.11: >>> >>> 2022-03-03 08:21:09,477 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-3:null) (logid:7ff2c4c1) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:21:09,482 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-3:null) (logid:7ff2c4c1) Asking libvirt to refresh >> storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c >>> 2022-03-03 08:22:03,802 WARN >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:null) (logid:4fc5e215) Expected 1 answers while >> executing SetMonitorServiceCommand but received 3 >>> 2022-03-03 08:22:07,740 INFO >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-5:null) (logid:ba7d1810) Fetching health check result >> for 169.254.4.222 and executing fresh checks: false >>> 2022-03-03 08:22:13,881 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-3:null) (logid:f0ed50d5) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:22:13,886 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-3:null) (logid:f0ed50d5) Asking libvirt to refresh >> storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c >>> 2022-03-03 08:22:17,641 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:22:17,651 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:22:17,659 INFO [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:26:45,714 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >> (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting >> reconnection while we still have 0 commands in progress. >>> 2022-03-03 08:26:45,715 INFO [utils.nio.NioClient] >> (Agent-Handler-4:null) (logid:3959c447) NioClient connection closed >>> 2022-03-03 08:26:45,716 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >> (logid:3959c447) Reconnecting to host:10.101.2.40 >>> 2022-03-03 08:26:45,716 INFO [utils.nio.NioClient] >> (Agent-Handler-4:null) (logid:3959c447) Connecting to 10.101.2.40:8250 >>> 2022-03-03 08:26:45,733 INFO [utils.nio.Link] (Agent-Handler-4:null) >> (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties >>> 2022-03-03 08:27:15,762 WARN [utils.nio.Link] (Agent-Handler-4:null) >> (logid:3959c447) This SSL engine was forced to close inbound due to end of >> stream. >>> javax.net.ssl.SSLException: closing inbound before receiving peer's >> close_notify >>> at >> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133) >>> at >> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) >>> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:339) >>> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:295) >>> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:286) >>> at >> java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733) >>> at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490) >>> at com.cloud.utils.nio.Link.doHandshake(Link.java:618) >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:64) >>> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) >>> at com.cloud.agent.Agent.reconnect(Agent.java:536) >>> at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) >>> at com.cloud.utils.nio.Task.call(Task.java:83) >>> at com.cloud.utils.nio.Task.call(Task.java:29) >>> at >> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>> at java.base/java.lang.Thread.run(Thread.java:829) >>> 2022-03-03 08:27:15,768 INFO [utils.nio.NioClient] >> (Agent-Handler-4:null) (logid:3959c447) SSL: Handshake done >>> 2022-03-03 08:27:15,768 INFO [utils.nio.NioClient] >> (Agent-Handler-4:null) (logid:3959c447) Connected to 10.101.2.40:8250 >>> 2022-03-03 08:27:15,771 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting >> reconnection while we still have 0 commands in progress. >>> 2022-03-03 08:27:15,772 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) NioClient connection closed >>> 2022-03-03 08:27:15,772 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:3959c447) Reconnecting to host:10.101.2.40 >>> 2022-03-03 08:27:15,773 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) Connecting to 10.101.2.40:8250 >>> 2022-03-03 08:27:15,783 INFO [utils.linux.KVMHostInfo] >> (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file >> [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds >> to a CPU speed of [3000] MHz. >>> 2022-03-03 08:27:15,791 INFO [utils.nio.Link] (Agent-Handler-2:null) >> (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties >>> 2022-03-03 08:27:15,890 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt >>> 2022-03-03 08:27:15,890 ERROR [kvm.resource.LibvirtConnection] >> (Agent-Handler-1:null) (logid:3959c447) Connection with libvirtd is broken: >> invalid connection pointer in virConnectGetVersion >>> 2022-03-03 08:27:15,893 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. >>> 2022-03-03 08:27:15,893 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:27:15,907 WARN [cloud.agent.Agent] (Agent-Handler-1:null) >> (logid:3959c447) Unable to send reques: null >>> 2022-03-03 08:27:20,770 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >> (logid:3959c447) Reconnecting to host:10.101.2.40 >>> 2022-03-03 08:27:20,770 INFO [utils.nio.NioClient] >> (Agent-Handler-4:null) (logid:3959c447) Connecting to 10.101.2.40:8250 >>> 2022-03-03 08:27:20,790 INFO [utils.nio.Link] (Agent-Handler-4:null) >> (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties >>> 2022-03-03 08:27:45,814 WARN [utils.nio.Link] (Agent-Handler-2:null) >> (logid:3959c447) This SSL engine was forced to close inbound due to end of >> stream. >>> javax.net.ssl.SSLException: closing inbound before receiving peer's >> close_notify >>> at >> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133) >>> at >> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) >>> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:339) >>> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:295) >>> at >> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:286) >>> at >> java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733) >>> at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490) >>> at com.cloud.utils.nio.Link.doHandshake(Link.java:618) >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:64) >>> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) >>> at com.cloud.agent.Agent.reconnect(Agent.java:536) >>> at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) >>> at com.cloud.utils.nio.Task.call(Task.java:83) >>> at com.cloud.utils.nio.Task.call(Task.java:29) >>> at >> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>> at java.base/java.lang.Thread.run(Thread.java:829) >>> 2022-03-03 08:27:45,816 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done >>> 2022-03-03 08:27:45,816 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 >>> 2022-03-03 08:27:45,818 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting >> reconnection while we still have 0 commands in progress. >>> 2022-03-03 08:27:45,819 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) NioClient connection closed >>> 2022-03-03 08:27:45,819 ERROR [utils.nio.NioConnection] >> (Agent-Handler-4:null) (logid:3959c447) Unable to initialize the threads. >>> java.nio.channels.ClosedChannelException >>> at java.base/sun.nio.ch >> .SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:180) >>> at java.base/sun.nio.ch >> .SocketChannelImpl.read(SocketChannelImpl.java:341) >>> at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:485) >>> at com.cloud.utils.nio.Link.doHandshake(Link.java:618) >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:64) >>> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) >>> at com.cloud.agent.Agent.reconnect(Agent.java:536) >>> at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) >>> at com.cloud.utils.nio.Task.call(Task.java:83) >>> at com.cloud.utils.nio.Task.call(Task.java:29) >>> at >> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>> at java.base/java.lang.Thread.run(Thread.java:829) >>> 2022-03-03 08:27:45,819 INFO [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:3959c447) Reconnecting to host:10.101.2.40 >>> 2022-03-03 08:27:45,820 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) Connecting to 10.101.2.40:8250 >>> 2022-03-03 08:27:45,821 INFO [cloud.agent.Agent] (Agent-Handler-4:null) >> (logid:3959c447) Attempted to re-connect to the server, but received an >> unexpected exception, trying again... >>> com.cloud.utils.exception.NioConnectionException >>> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101) >>> at com.cloud.agent.Agent.reconnect(Agent.java:536) >>> at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) >>> at com.cloud.utils.nio.Task.call(Task.java:83) >>> at com.cloud.utils.nio.Task.call(Task.java:29) >>> at >> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>> at java.base/java.lang.Thread.run(Thread.java:829) >>> Caused by: java.nio.channels.ClosedChannelException >>> at java.base/sun.nio.ch >> .SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:180) >>> at java.base/sun.nio.ch >> .SocketChannelImpl.read(SocketChannelImpl.java:341) >>> at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:485) >>> at com.cloud.utils.nio.Link.doHandshake(Link.java:618) >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:64) >>> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) >>> ... 8 more >>> >>> ... >>> >>> 2022-03-03 08:30:06,027 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) NioClient connection closed >>> 2022-03-03 08:30:06,032 INFO [utils.nio.Link] (Agent-Handler-2:null) >> (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties >>> 2022-03-03 08:30:06,032 INFO [utils.nio.Link] (Agent-Handler-2:null) >> (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties >>> 2022-03-03 08:30:06,037 INFO [utils.nio.Link] (Agent-Handler-2:null) >> (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties >>> 2022-03-03 08:30:06,261 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done >>> 2022-03-03 08:30:06,261 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 >>> 2022-03-03 08:30:06,268 INFO [utils.linux.KVMHostInfo] >> (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file >> [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds >> to a CPU speed of [3000] MHz. >>> 2022-03-03 08:30:06,303 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done >>> 2022-03-03 08:30:06,303 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 >>> 2022-03-03 08:30:06,303 ERROR [kvm.resource.LibvirtConnection] >> (Agent-Handler-1:null) (logid:3959c447) Connection with libvirtd is broken: >> invalid connection pointer in virConnectGetVersion >>> 2022-03-03 08:30:06,309 INFO [utils.linux.KVMHostInfo] >> (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file >> [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds >> to a CPU speed of [3000] MHz. >>> 2022-03-03 08:30:06,334 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done >>> 2022-03-03 08:30:06,334 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 >>> 2022-03-03 08:30:06,335 ERROR [kvm.resource.LibvirtConnection] >> (Agent-Handler-1:null) (logid:3959c447) Connection with libvirtd is broken: >> invalid connection pointer in virConnectGetVersion >>> 2022-03-03 08:30:06,339 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done >>> 2022-03-03 08:30:06,339 INFO [utils.nio.NioClient] >> (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 >>> 2022-03-03 08:30:06,345 INFO [utils.linux.KVMHostInfo] >> (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file >> [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds >> to a CPU speed of [3000] MHz. >>> 2022-03-03 08:30:06,346 INFO [utils.linux.KVMHostInfo] >> (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file >> [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds >> to a CPU speed of [3000] MHz. >>> 2022-03-03 08:30:06,364 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt >>> 2022-03-03 08:30:06,365 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. >>> 2022-03-03 08:30:06,365 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:30:06,408 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt >>> 2022-03-03 08:30:06,409 ERROR [kvm.resource.LibvirtConnection] >> (Agent-Handler-1:null) (logid:3959c447) Connection with libvirtd is broken: >> invalid connection pointer in virConnectGetVersion >>> 2022-03-03 08:30:06,410 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. >>> 2022-03-03 08:30:06,410 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:30:06,436 INFO [cloud.agent.Agent] (Agent-Handler-5:null) >> (logid:754ee53b) Lost connection to host: 10.101.2.40. Attempting >> reconnection while we still have 0 commands in progress. >>> 2022-03-03 08:30:06,436 INFO [utils.nio.NioClient] >> (Agent-Handler-5:null) (logid:754ee53b) NioClient connection closed >>> 2022-03-03 08:30:06,436 INFO [cloud.agent.Agent] (Agent-Handler-5:null) >> (logid:754ee53b) Reconnecting to host:10.101.2.40 >>> 2022-03-03 08:30:06,436 INFO [utils.nio.NioClient] >> (Agent-Handler-5:null) (logid:754ee53b) Connecting to 10.101.2.40:8250 >>> 2022-03-03 08:30:06,436 WARN [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:3959c447) Threw away a startup answer because we're reconnecting. >>> 2022-03-03 08:30:06,446 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt >>> 2022-03-03 08:30:06,447 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. >>> 2022-03-03 08:30:06,447 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:30:06,451 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt >>> 2022-03-03 08:30:06,451 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. >>> 2022-03-03 08:30:06,451 INFO [kvm.storage.LibvirtStorageAdaptor] >> (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool >> cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt >>> 2022-03-03 08:30:06,459 INFO [utils.nio.Link] (Agent-Handler-5:null) >> (logid:754ee53b) Conf file found: /etc/cloudstack/agent/agent.properties >>> 2022-03-03 08:30:06,479 INFO [cloud.agent.Agent] (Agent-Handler-1:null) >> (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting >> reconnection while we still have 1 commands in progress. >>> 2022-03-03 08:30:06,479 INFO [utils.nio.NioClient] >> (Agent-Handler-1:null) (logid:3959c447) NioClient connection closed >>> 2022-03-03 08:30:06,479 INFO [cloud.agent.Agent] (Agent-Handler-1:null) >> (logid:3959c447) Reconnecting to host:10.101.2.40 >>> 2022-03-03 08:30:06,479 INFO [utils.nio.NioClient] >> (Agent-Handler-1:null) (logid:3959c447) Connecting to 10.101.2.40:8250 >>> 2022-03-03 08:30:06,482 WARN [cloud.agent.Agent] (Agent-Handler-2:null) >> (logid:3959c447) Threw away a startup answer because we're reconnecting. >>> 2022-03-03 08:30:06,485 INFO [cloud.agent.Agent] (Agent-Handler-5:null) >> (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting >> reconnection while we still have 2 commands in progress. >>> 2022-03-03 08:30:06,486 INFO [utils.nio.NioClient] >> (Agent-Handler-5:null) (logid:3959c447) NioClient connection closed >>> 2022-03-03 08:30:06,486 ERROR [utils.nio.NioConnection] >> (Agent-Handler-1:null) (logid:3959c447) Unable to initialize the threads. >>> java.nio.channels.AsynchronousCloseException >>> at >> java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) >>> at java.base/sun.nio.ch >> .SocketChannelImpl.endRead(SocketChannelImpl.java:331) >>> at java.base/sun.nio.ch >> .SocketChannelImpl.endConnect(SocketChannelImpl.java:659) >>> at java.base/sun.nio.ch >> .SocketChannelImpl.connect(SocketChannelImpl.java:695) >>> at com.cloud.utils.nio.NioClient.init(NioClient.java:56) >>> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) >>> at com.cloud.agent.Agent.reconnect(Agent.java:536) >>> at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) >>> at com.cloud.utils.nio.Task.call(Task.java:83) >>> at com.cloud.utils.nio.Task.call(Task.java:29) >>> at >> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >>> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >>> at java.base/java.lang.Thread.run(Thread.java:829) >>> 2022-03-03 08:30:06,486 INFO [cloud.agent.Agent] (Agent-Handler-5:null) >> (logid:3959c447) Reconnecting to host:10.101.2.40 >>> 2022-03-03 08:30:06,486 INFO [cloud.agent.Agent] (Agent-Handler-1:null) >> (logid:3959c447) Attempted to re-connect to the server, but received an >> unexpected exception, trying again... >>> com.cloud.utils.exception.NioConnectionException >>> ... >>> >>> >>> Thank you, >>> Antoine >> >>
