Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt and qemu and ca-certificates to the same version as the other functional nodes. That didn’t seem to help. This is obviously an ssl issue but I don’t really know what to do about it.
2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.15.0.0 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id is 0 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties. 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver. 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) iscsi session clean up is disabled 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.30.59 : port = 8250 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.30.59:8250 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.30.59:8250 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 18796842-a137-475d-9799-9874240e3c0c, using it. 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 18796842-a137-475d-9799-9874240e3c0c from libvirt 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) (logid:) Default Builder inited. 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Proccess agent startup answer, agent id = 0 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill I also noticed this: 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to handle host connection: java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with hostOS: Rocky into a cluster,in which there are CentOS hosts added at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456) at com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719) at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946) at com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278) at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source) at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097) at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) The fact that this host has been converted to Rocky Linux is causing an issue? What’s the work around for this? Thanks -jeremy > On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <jer...@skidrow.la > (mailto:jer...@skidrow.la)> wrote: > It doesn’t error out but there is purposely no VMs running on these hosts to > test the upgrade of the underlying distro before having to stop any active > VMs. > > [jeremy@cm02 ~]$ sudo virsh list > Id Name State > -------------------- > > [jeremy@cm02 ~]$ > > > -jeremy > > > > > > On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <ustcweiz...@gmail.com > > (mailto:ustcweiz...@gmail.com)> wrote: > > Hi, > > > > Do virsh commands e.g. "virsh list" work ? > > > > -Wei > > > > On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <jer...@skidrow.la.invalid> > > wrote: > > > > > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5. > > > > > > I’m running Cloudstack 4.15.0.0. > > > > > > Trying to bring up cloudstack agent, I’m seeing some issues communicating > > > with libvirt: > > > > > > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > > > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt > > > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection] > > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: > > > invalid > > > connection pointer in virConnectGetVersion > > > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > > > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it. > > > > > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: > > > Input/output error > > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: > > > Input/output error > > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: > > > Input/output error > > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: > > > Input/output error > > > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output > > > error > > > > > > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64 > > > > > > Functional hosts that have yet to be upgraded are using: > > > > > > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64 > > > > > > > > > My libvirtd.conf looks like this: > > > > > > listen_tcp=0 > > > listen_tls=1 > > > tcp_port="16509" > > > auth_tcp="none" > > > mdns_adv = 0 > > > key_file="/etc/pki/libvirt/private/serverkey.pem" > > > cert_file="/etc/pki/libvirt/servercert.pem" > > > ca_file="/etc/pki/CA/cacert.pem" > > > tls_port="16514" > > > auth_tls=“none" > > > > > > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource] > > > (main:null) (logid:) uefi properties file not found due to: Unable to find > > > file uefi.properties. > > > 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null) > > > (logid:) No existing libvirtd connection found. Opening a new one > > > 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource] > > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to > > > BridgeVifDriver. > > > 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource] > > > (main:null) (logid:) iscsi session clean up is disabled > > > 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:) > > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : > > > workers = 5 : host = 192.168.30.59 : port = 8250 > > > 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:) > > > Connecting to 192.168.30.59:8250 > > > 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf > > > file found: /etc/cloudstack/agent/agent.properties > > > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:) > > > SSL: Handshake done > > > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:) > > > Connected to 192.168.30.59:8250 > > > 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt > > > 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection] > > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: > > > invalid > > > connection pointer in virConnectGetVersion > > > 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > > > 18796842-a137-475d-9799-9874240e3c0c, using it. > > > 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool > > > 18796842-a137-475d-9799-9874240e3c0c from libvirt > > > 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper] > > > (Agent-Handler-1:null) (logid:) Default Builder inited. > > > 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > > (logid:) Proccess agent startup answer, agent id = 0 > > > 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > > (logid:) Set agent id 0 > > > 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > > (logid:) Startup Response Received: agent id = 0 > > > 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent] > > > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill > > > 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Agent started > > > 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Implementation Version is 4.15.0.0 > > > 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties > > > 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Defaulting to using properties file for storage > > > 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Defaulting to the constant time backoff algorithm > > > 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:) > > > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > > > 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null) > > > (logid:) Using default Java settings for IPv6 preference for agent > > > connection > > > 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id > > > is 0 > > > 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource] > > > (main:null) (logid:) uefi properties file not found due to: Unable to find > > > file uefi.properties. > > > 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null) > > > (logid:) No existing libvirtd connection found. Opening a new one > > > 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource] > > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to > > > BridgeVifDriver. > > > 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource] > > > (main:null) (logid:) iscsi session clean up is disabled > > > 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:) > > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : > > > workers = 5 : host = 192.168.30.59 : port = 8250 > > > 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:) > > > Connecting to 192.168.30.59:8250 > > > 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf > > > file found: /etc/cloudstack/agent/agent.properties > > > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:) > > > SSL: Handshake done > > > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:) > > > Connected to 192.168.30.59:8250 > > > 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool > > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt > > > 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection] > > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: > > > invalid > > > connection pointer in virConnectGetVersion > > > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool > > > 18796842-a137-475d-9799-9874240e3c0c, using it. > > > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor] > > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool > > > 18796842-a137-475d-9799-9874240e3c0c from libvirt > > > 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper] > > > (Agent-Handler-1:null) (logid:) Default Builder inited. > > > 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > > (logid:) Proccess agent startup answer, agent id = 0 > > > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] > > > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill > > > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > > (logid:) Set agent id 0 > > > 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null) > > > (logid:) Startup Response Received: agent id = 0 > > > > > > Any ideas? I’m working toward upgrading the entire cluster along with > > > Cloudstack itself. > > > > > > -jeremy > > > > > > > > > > > >
signature.asc
Description: PGP signature
