Re: All VM domains in KVM suddenly terminated and VMs cannot start
looks like your ssl keys got corrupted, I suggest removing them and re-adding the hosts (as a first try. Hope this is not your production environment. On Sat, Sep 4, 2021 at 12:20 PM Fariborz Navidan wrote: > Hello, > > All VM domains in KVM suddenly terminated but CloudStack was thinking they > are running as states were not updated in the database. I just forcefully > stopped them and now they cannot start due to insufficient capacity. Below > is the agent log. i tried restarting systemd-machind, libvirtd and agent. > Also checked all ports. Port 8250 is open. > > in:null) (logid:) Connecting to 79.143.86.149:8250 > 2021-09-04 12:16:02,000 INFO [utils.nio.Link] (main:null) (logid:) Conf > file found: /etc/cloudstack/agent/agent.properties > > > > > > > > 2021-09-04 12:16:02,255 ERROR [utils.nio.Link] (main:null) (logid:) SSL > error caught during unwrap data: Received fatal alert: certificate_unknown, > for local address=/79.143.86.149:53860, remote address=/79.143.86.149:8250 > . > The client may have invalid ca-certificates. > 2021-09-04 12:16:02,255 ERROR [utils.nio.NioClient] (main:null) (logid:) > SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 > 2021-09-04 12:16:02,255 ERROR [utils.nio.NioConnection] (main:null) > (logid:) Unable to initialize the threads. > java.io.IOException: SSL Handshake failed while connecting to host: > 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > 2021-09-04 12:16:02,255 INFO [utils.nio.NioClient] (main:null) (logid:) > NioClient connection closed > 2021-09-04 12:16:02,255 INFO [cloud.agent.Agent] (main:null) (logid:) > Attempted to connect to the server, but received an unexpected exception, > trying again... > com.cloud.utils.exception.NioConnectionException: SSL Handshake failed > while connecting to host: 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > Caused by: java.io.IOException: SSL Handshake failed while connecting to > host: 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > ... 6 more > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2021-09-04 12:16:07,256 INFO [cloud.agent.Agent] (main:null) (logid:) > Connecting to host:79.143.86.149 > 2021-09-04 12:16:07,256 INFO [utils.nio.NioClient] (main:null) (logid:) > Connecting to 79.143.86.149:8250 > 2021-09-04 12:16:07,257 INFO [utils.nio.Link] (main:null) (logid:) Conf > file found: /etc/cloudstack/agent/agent.properties > 2021-09-04 12:16:07,521 ERROR [utils.nio.Link] (main:null) (logid:) SSL > error caught during unwrap data: Received fatal alert: certificate_unknown, > for local address=/79.143.86.149:53862, remote address=/79.143.86.149:8250 > . > The client may have invalid ca-certificates. > 2021-09-04 12:16:07,521 ERROR [utils.nio.NioClient] (main:null) (logid:) > SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 > 2021-09-04 12:16:07,521 ERROR [utils.nio.NioConnection] (main:null) > (logid:) Unable to initialize the threads. > java.io.IOException: SSL Handshake failed while connecting to host: > 79.143.86.149 port: 8250 > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) > at com.cloud.agent.Agent.start(Agent.java:293) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > 2021-09-04 12:16:07,522 INFO [utils.nio.NioClient] (main:null) (logid:) > NioClient connection closed > 2021-09-04 12:16:07,522 INFO [cloud.agent.Agent] (main:null) (logid:) > Attempted to connect to the server, but received an
All VM domains in KVM suddenly terminated and VMs cannot start
Hello, All VM domains in KVM suddenly terminated but CloudStack was thinking they are running as states were not updated in the database. I just forcefully stopped them and now they cannot start due to insufficient capacity. Below is the agent log. i tried restarting systemd-machind, libvirtd and agent. Also checked all ports. Port 8250 is open. in:null) (logid:) Connecting to 79.143.86.149:8250 2021-09-04 12:16:02,000 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties 2021-09-04 12:16:02,255 ERROR [utils.nio.Link] (main:null) (logid:) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/79.143.86.149:53860, remote address=/79.143.86.149:8250. The client may have invalid ca-certificates. 2021-09-04 12:16:02,255 ERROR [utils.nio.NioClient] (main:null) (logid:) SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 2021-09-04 12:16:02,255 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads. java.io.IOException: SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 at com.cloud.utils.nio.NioClient.init(NioClient.java:67) at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) at com.cloud.agent.Agent.start(Agent.java:293) at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) at com.cloud.agent.AgentShell.start(AgentShell.java:512) at com.cloud.agent.AgentShell.main(AgentShell.java:547) 2021-09-04 12:16:02,255 INFO [utils.nio.NioClient] (main:null) (logid:) NioClient connection closed 2021-09-04 12:16:02,255 INFO [cloud.agent.Agent] (main:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again... com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101) at com.cloud.agent.Agent.start(Agent.java:293) at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) at com.cloud.agent.AgentShell.start(AgentShell.java:512) at com.cloud.agent.AgentShell.main(AgentShell.java:547) Caused by: java.io.IOException: SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 at com.cloud.utils.nio.NioClient.init(NioClient.java:67) at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) ... 6 more 2021-09-04 12:16:07,256 INFO [cloud.agent.Agent] (main:null) (logid:) Connecting to host:79.143.86.149 2021-09-04 12:16:07,256 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 79.143.86.149:8250 2021-09-04 12:16:07,257 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties 2021-09-04 12:16:07,521 ERROR [utils.nio.Link] (main:null) (logid:) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/79.143.86.149:53862, remote address=/79.143.86.149:8250. The client may have invalid ca-certificates. 2021-09-04 12:16:07,521 ERROR [utils.nio.NioClient] (main:null) (logid:) SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 2021-09-04 12:16:07,521 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads. java.io.IOException: SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 at com.cloud.utils.nio.NioClient.init(NioClient.java:67) at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95) at com.cloud.agent.Agent.start(Agent.java:293) at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) at com.cloud.agent.AgentShell.start(AgentShell.java:512) at com.cloud.agent.AgentShell.main(AgentShell.java:547) 2021-09-04 12:16:07,522 INFO [utils.nio.NioClient] (main:null) (logid:) NioClient connection closed 2021-09-04 12:16:07,522 INFO [cloud.agent.Agent] (main:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again... com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 79.143.86.149 port: 8250 at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101) at com.cloud.agent.Agent.start(Agent.java:293) at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455) at
