Hi, It would be good to upload management server log as well.
-Wei On Fri, 4 Mar 2022 at 19:56, Antoine Boucher <[email protected]> wrote: > Any one? > > Killing the Java processes does not recover from the situation. The Java > processes end-up restarting and using 100% of all CPUs. > > Regards, > Antoine > > > > Confidentiality Warning: This message and any attachments are intended > only for the use of the intended recipient(s), are confidential, and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, retransmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return e-mail, and delete this message and any attachments > from your system. > > > > On Mar 3, 2022, at 9:17 AM, Antoine Boucher <[email protected]> > wrote: > > > > I have 2 zones hosted from the same management server, one larger local > to management server zone and one smaller “remote” zone using an IPSec > tunnel. > > > > All is well except that recently one of the “remote” KVM host keeps > loosing connectivity with the management server. It used to happened > every few days 24 but now it happens every few hours. > > > > Other than rebooting, I’m only able to recover by killing the Java > process and restarting the CloudStack-Agent. > > > > The tunnel is monitored and used by many other processes without any > interruptions. > > > > Has anyone experienced anything similar before? > > > > > > Here are my host logs, the management server is 10.101.2.40 and the kvm > host is 10.91.1.11: > > > > 2022-03-03 08:21:09,477 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-3:null) (logid:7ff2c4c1) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:21:09,482 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-3:null) (logid:7ff2c4c1) Asking libvirt to refresh > storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c > > 2022-03-03 08:22:03,802 WARN > [resource.virtualnetwork.VirtualRoutingResource] > (agentRequest-Handler-1:null) (logid:4fc5e215) Expected 1 answers while > executing SetMonitorServiceCommand but received 3 > > 2022-03-03 08:22:07,740 INFO > [resource.virtualnetwork.VirtualRoutingResource] > (agentRequest-Handler-5:null) (logid:ba7d1810) Fetching health check result > for 169.254.4.222 and executing fresh checks: false > > 2022-03-03 08:22:13,881 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-3:null) (logid:f0ed50d5) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:22:13,886 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-3:null) (logid:f0ed50d5) Asking libvirt to refresh > storage pool cec8c1db-c9a0-42cc-96d3-5c06369d115c > > 2022-03-03 08:22:17,641 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:22:17,651 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:22:17,659 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-2:null) (logid:54076348) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:26:45,714 INFO [cloud.agent.Agent] (Agent-Handler-4:null) > (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting > reconnection while we still have 0 commands in progress. > > 2022-03-03 08:26:45,715 INFO [utils.nio.NioClient] > (Agent-Handler-4:null) (logid:3959c447) NioClient connection closed > > 2022-03-03 08:26:45,716 INFO [cloud.agent.Agent] (Agent-Handler-4:null) > (logid:3959c447) Reconnecting to host:10.101.2.40 > > 2022-03-03 08:26:45,716 INFO [utils.nio.NioClient] > (Agent-Handler-4:null) (logid:3959c447) Connecting to 10.101.2.40:8250 > > 2022-03-03 08:26:45,733 INFO [utils.nio.Link] (Agent-Handler-4:null) > (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties > > 2022-03-03 08:27:15,762 WARN [utils.nio.Link] (Agent-Handler-4:null) > (logid:3959c447) This SSL engine was forced to close inbound due to end of > stream. > > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify > > at > java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133) > > at > java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) > > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:339) > > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:295) > > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:286) > > at > java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733) > > at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490) > > at com.cloud.utils.nio.Link.doHandshake(Link.java:618) > > at com.cloud.utils.nio.NioClient.init(NioClient.java:64) > > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > > at com.cloud.agent.Agent.reconnect(Agent.java:536) > > at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) > > at com.cloud.utils.nio.Task.call(Task.java:83) > > at com.cloud.utils.nio.Task.call(Task.java:29) > > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2022-03-03 08:27:15,768 INFO [utils.nio.NioClient] > (Agent-Handler-4:null) (logid:3959c447) SSL: Handshake done > > 2022-03-03 08:27:15,768 INFO [utils.nio.NioClient] > (Agent-Handler-4:null) (logid:3959c447) Connected to 10.101.2.40:8250 > > 2022-03-03 08:27:15,771 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting > reconnection while we still have 0 commands in progress. > > 2022-03-03 08:27:15,772 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) NioClient connection closed > > 2022-03-03 08:27:15,772 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:3959c447) Reconnecting to host:10.101.2.40 > > 2022-03-03 08:27:15,773 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) Connecting to 10.101.2.40:8250 > > 2022-03-03 08:27:15,783 INFO [utils.linux.KVMHostInfo] > (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file > [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds > to a CPU speed of [3000] MHz. > > 2022-03-03 08:27:15,791 INFO [utils.nio.Link] (Agent-Handler-2:null) > (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties > > 2022-03-03 08:27:15,890 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt > > 2022-03-03 08:27:15,890 ERROR [kvm.resource.LibvirtConnection] > (Agent-Handler-1:null) (logid:3959c447) Connection with libvirtd is broken: > invalid connection pointer in virConnectGetVersion > > 2022-03-03 08:27:15,893 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. > > 2022-03-03 08:27:15,893 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:27:15,907 WARN [cloud.agent.Agent] (Agent-Handler-1:null) > (logid:3959c447) Unable to send reques: null > > 2022-03-03 08:27:20,770 INFO [cloud.agent.Agent] (Agent-Handler-4:null) > (logid:3959c447) Reconnecting to host:10.101.2.40 > > 2022-03-03 08:27:20,770 INFO [utils.nio.NioClient] > (Agent-Handler-4:null) (logid:3959c447) Connecting to 10.101.2.40:8250 > > 2022-03-03 08:27:20,790 INFO [utils.nio.Link] (Agent-Handler-4:null) > (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties > > 2022-03-03 08:27:45,814 WARN [utils.nio.Link] (Agent-Handler-2:null) > (logid:3959c447) This SSL engine was forced to close inbound due to end of > stream. > > javax.net.ssl.SSLException: closing inbound before receiving peer's > close_notify > > at > java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133) > > at > java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) > > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:339) > > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:295) > > at > java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:286) > > at > java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733) > > at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490) > > at com.cloud.utils.nio.Link.doHandshake(Link.java:618) > > at com.cloud.utils.nio.NioClient.init(NioClient.java:64) > > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > > at com.cloud.agent.Agent.reconnect(Agent.java:536) > > at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) > > at com.cloud.utils.nio.Task.call(Task.java:83) > > at com.cloud.utils.nio.Task.call(Task.java:29) > > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2022-03-03 08:27:45,816 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done > > 2022-03-03 08:27:45,816 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 > > 2022-03-03 08:27:45,818 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting > reconnection while we still have 0 commands in progress. > > 2022-03-03 08:27:45,819 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) NioClient connection closed > > 2022-03-03 08:27:45,819 ERROR [utils.nio.NioConnection] > (Agent-Handler-4:null) (logid:3959c447) Unable to initialize the threads. > > java.nio.channels.ClosedChannelException > > at java.base/sun.nio.ch > .SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:180) > > at java.base/sun.nio.ch > .SocketChannelImpl.read(SocketChannelImpl.java:341) > > at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:485) > > at com.cloud.utils.nio.Link.doHandshake(Link.java:618) > > at com.cloud.utils.nio.NioClient.init(NioClient.java:64) > > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > > at com.cloud.agent.Agent.reconnect(Agent.java:536) > > at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) > > at com.cloud.utils.nio.Task.call(Task.java:83) > > at com.cloud.utils.nio.Task.call(Task.java:29) > > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2022-03-03 08:27:45,819 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:3959c447) Reconnecting to host:10.101.2.40 > > 2022-03-03 08:27:45,820 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) Connecting to 10.101.2.40:8250 > > 2022-03-03 08:27:45,821 INFO [cloud.agent.Agent] (Agent-Handler-4:null) > (logid:3959c447) Attempted to re-connect to the server, but received an > unexpected exception, trying again... > > com.cloud.utils.exception.NioConnectionException > > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101) > > at com.cloud.agent.Agent.reconnect(Agent.java:536) > > at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) > > at com.cloud.utils.nio.Task.call(Task.java:83) > > at com.cloud.utils.nio.Task.call(Task.java:29) > > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > Caused by: java.nio.channels.ClosedChannelException > > at java.base/sun.nio.ch > .SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:180) > > at java.base/sun.nio.ch > .SocketChannelImpl.read(SocketChannelImpl.java:341) > > at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:485) > > at com.cloud.utils.nio.Link.doHandshake(Link.java:618) > > at com.cloud.utils.nio.NioClient.init(NioClient.java:64) > > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > > ... 8 more > > > > ... > > > > 2022-03-03 08:30:06,027 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) NioClient connection closed > > 2022-03-03 08:30:06,032 INFO [utils.nio.Link] (Agent-Handler-2:null) > (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties > > 2022-03-03 08:30:06,032 INFO [utils.nio.Link] (Agent-Handler-2:null) > (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties > > 2022-03-03 08:30:06,037 INFO [utils.nio.Link] (Agent-Handler-2:null) > (logid:3959c447) Conf file found: /etc/cloudstack/agent/agent.properties > > 2022-03-03 08:30:06,261 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done > > 2022-03-03 08:30:06,261 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 > > 2022-03-03 08:30:06,268 INFO [utils.linux.KVMHostInfo] > (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file > [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds > to a CPU speed of [3000] MHz. > > 2022-03-03 08:30:06,303 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done > > 2022-03-03 08:30:06,303 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 > > 2022-03-03 08:30:06,303 ERROR [kvm.resource.LibvirtConnection] > (Agent-Handler-1:null) (logid:3959c447) Connection with libvirtd is broken: > invalid connection pointer in virConnectGetVersion > > 2022-03-03 08:30:06,309 INFO [utils.linux.KVMHostInfo] > (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file > [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds > to a CPU speed of [3000] MHz. > > 2022-03-03 08:30:06,334 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done > > 2022-03-03 08:30:06,334 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 > > 2022-03-03 08:30:06,335 ERROR [kvm.resource.LibvirtConnection] > (Agent-Handler-1:null) (logid:3959c447) Connection with libvirtd is broken: > invalid connection pointer in virConnectGetVersion > > 2022-03-03 08:30:06,339 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) SSL: Handshake done > > 2022-03-03 08:30:06,339 INFO [utils.nio.NioClient] > (Agent-Handler-2:null) (logid:3959c447) Connected to 10.101.2.40:8250 > > 2022-03-03 08:30:06,345 INFO [utils.linux.KVMHostInfo] > (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file > [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds > to a CPU speed of [3000] MHz. > > 2022-03-03 08:30:06,346 INFO [utils.linux.KVMHostInfo] > (Agent-Handler-1:null) (logid:3959c447) Retrieved value [3000000] from file > [/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq]. This corresponds > to a CPU speed of [3000] MHz. > > 2022-03-03 08:30:06,364 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt > > 2022-03-03 08:30:06,365 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. > > 2022-03-03 08:30:06,365 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:30:06,408 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt > > 2022-03-03 08:30:06,409 ERROR [kvm.resource.LibvirtConnection] > (Agent-Handler-1:null) (logid:3959c447) Connection with libvirtd is broken: > invalid connection pointer in virConnectGetVersion > > 2022-03-03 08:30:06,410 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. > > 2022-03-03 08:30:06,410 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:30:06,436 INFO [cloud.agent.Agent] (Agent-Handler-5:null) > (logid:754ee53b) Lost connection to host: 10.101.2.40. Attempting > reconnection while we still have 0 commands in progress. > > 2022-03-03 08:30:06,436 INFO [utils.nio.NioClient] > (Agent-Handler-5:null) (logid:754ee53b) NioClient connection closed > > 2022-03-03 08:30:06,436 INFO [cloud.agent.Agent] (Agent-Handler-5:null) > (logid:754ee53b) Reconnecting to host:10.101.2.40 > > 2022-03-03 08:30:06,436 INFO [utils.nio.NioClient] > (Agent-Handler-5:null) (logid:754ee53b) Connecting to 10.101.2.40:8250 > > 2022-03-03 08:30:06,436 WARN [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:3959c447) Threw away a startup answer because we're reconnecting. > > 2022-03-03 08:30:06,446 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt > > 2022-03-03 08:30:06,447 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. > > 2022-03-03 08:30:06,447 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:30:06,451 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Attempting to create storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c (Filesystem) in libvirt > > 2022-03-03 08:30:06,451 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Found existing defined storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c, using it. > > 2022-03-03 08:30:06,451 INFO [kvm.storage.LibvirtStorageAdaptor] > (Agent-Handler-1:null) (logid:3959c447) Trying to fetch storage pool > cec8c1db-c9a0-42cc-96d3-5c06369d115c from libvirt > > 2022-03-03 08:30:06,459 INFO [utils.nio.Link] (Agent-Handler-5:null) > (logid:754ee53b) Conf file found: /etc/cloudstack/agent/agent.properties > > 2022-03-03 08:30:06,479 INFO [cloud.agent.Agent] (Agent-Handler-1:null) > (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting > reconnection while we still have 1 commands in progress. > > 2022-03-03 08:30:06,479 INFO [utils.nio.NioClient] > (Agent-Handler-1:null) (logid:3959c447) NioClient connection closed > > 2022-03-03 08:30:06,479 INFO [cloud.agent.Agent] (Agent-Handler-1:null) > (logid:3959c447) Reconnecting to host:10.101.2.40 > > 2022-03-03 08:30:06,479 INFO [utils.nio.NioClient] > (Agent-Handler-1:null) (logid:3959c447) Connecting to 10.101.2.40:8250 > > 2022-03-03 08:30:06,482 WARN [cloud.agent.Agent] (Agent-Handler-2:null) > (logid:3959c447) Threw away a startup answer because we're reconnecting. > > 2022-03-03 08:30:06,485 INFO [cloud.agent.Agent] (Agent-Handler-5:null) > (logid:3959c447) Lost connection to host: 10.101.2.40. Attempting > reconnection while we still have 2 commands in progress. > > 2022-03-03 08:30:06,486 INFO [utils.nio.NioClient] > (Agent-Handler-5:null) (logid:3959c447) NioClient connection closed > > 2022-03-03 08:30:06,486 ERROR [utils.nio.NioConnection] > (Agent-Handler-1:null) (logid:3959c447) Unable to initialize the threads. > > java.nio.channels.AsynchronousCloseException > > at > java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) > > at java.base/sun.nio.ch > .SocketChannelImpl.endRead(SocketChannelImpl.java:331) > > at java.base/sun.nio.ch > .SocketChannelImpl.endConnect(SocketChannelImpl.java:659) > > at java.base/sun.nio.ch > .SocketChannelImpl.connect(SocketChannelImpl.java:695) > > at com.cloud.utils.nio.NioClient.init(NioClient.java:56) > > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > > at com.cloud.agent.Agent.reconnect(Agent.java:536) > > at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1117) > > at com.cloud.utils.nio.Task.call(Task.java:83) > > at com.cloud.utils.nio.Task.call(Task.java:29) > > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2022-03-03 08:30:06,486 INFO [cloud.agent.Agent] (Agent-Handler-5:null) > (logid:3959c447) Reconnecting to host:10.101.2.40 > > 2022-03-03 08:30:06,486 INFO [cloud.agent.Agent] (Agent-Handler-1:null) > (logid:3959c447) Attempted to re-connect to the server, but received an > unexpected exception, trying again... > > com.cloud.utils.exception.NioConnectionException > > ... > > > > > > Thank you, > > Antoine > >
