GitHub user nlindblo closed a discussion: Agent in "connecting" state
I noticed that one of my Cloudstack hosts was in "Connecting" state, I have
tried to restart the agent service a couple of time and I am seeing this in
sequence recurring in the log. Any suggestion on what I should be
troubleshooting ? I can't see anything obvious in the log
2025-01-23 03:44:03,531 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Process agent startup answer, agent id = 0
2025-01-23 03:44:03,532 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Set agent id 0
2025-01-23 03:44:03,568 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Startup Response Received: agent id = 0
2025-01-23 03:45:03,651 INFO [cloud.agent.Agent] (AgentShutdownThread:null)
(logid:) Stopping the agent: Reason = sig.kill
2025-01-23 03:45:15,314 INFO [cloud.agent.AgentShell] (main:null) (logid:)
Agent started
2025-01-23 03:45:15,318 INFO [cloud.agent.AgentShell] (main:null) (logid:)
Implementation Version is 4.19.0.1
2025-01-23 03:45:15,320 INFO [cloud.agent.AgentShell] (main:null) (logid:)
agent.properties found at /etc/cloudstack/agent/agent.properties
2025-01-23 03:45:15,367 INFO [cloud.agent.AgentShell] (main:null) (logid:)
Defaulting to using properties file for storage
2025-01-23 03:45:15,368 INFO [cloud.agent.AgentShell] (main:null) (logid:)
Defaulting to the constant time backoff algorithm
2025-01-23 03:45:15,405 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j
configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2025-01-23 03:45:15,405 INFO [cloud.agent.AgentShell] (main:null) (logid:)
Using default Java settings for IPv6 preference for agent connection
2025-01-23 03:45:15,482 INFO [cloud.agent.Agent] (main:null) (logid:) id is 0
2025-01-23 03:45:15,494 ERROR [kvm.resource.LibvirtComputingResource]
(main:null) (logid:) uefi properties file not found due to: Unable to find file
uefi.properties.
2025-01-23 03:45:15,558 INFO [kvm.resource.LibvirtConnection] (main:null)
(logid:) No existing libvirtd connection found. Opening a new one
2025-01-23 03:45:15,764 INFO [kvm.resource.LibvirtComputingResource]
(main:null) (logid:) IO uring driver for Qemu: disabled
2025-01-23 03:45:15,846 INFO [kvm.storage.KVMStoragePoolManager] (main:null)
(logid:) adding storage adaptor for
com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor
2025-01-23 03:45:15,848 INFO [kvm.storage.KVMStoragePoolManager] (main:null)
(logid:) adding storage adaptor for
com.cloud.hypervisor.kvm.storage.StorPoolStorageAdaptor
2025-01-23 03:45:15,848 WARN [kvm.storage.KVMStoragePoolManager] (main:null)
(logid:) Duplicate StorageAdaptor type PowerFlex, not loading
com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
2025-01-23 03:45:15,849 INFO [kvm.storage.KVMStoragePoolManager] (main:null)
(logid:) adding storage adaptor for
com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor
2025-01-23 03:45:15,849 INFO [kvm.storage.KVMStoragePoolManager] (main:null)
(logid:) adding storage adaptor for
com.cloud.hypervisor.kvm.storage.FiberChannelAdapter
2025-01-23 03:45:15,855 INFO [kvm.storage.MultipathSCSIAdapterBase]
(main:null) (logid:) Loaded FiberChannelAdapter for StorageLayer
2025-01-23 03:45:15,856 INFO [kvm.resource.LibvirtComputingResource]
(main:null) (logid:) No libvirt.vif.driver specified. Defaults to
BridgeVifDriver.
2025-01-23 03:45:16,007 INFO [cloud.serializer.GsonHelper] (main:null)
(logid:) Default Builder inited.
2025-01-23 03:45:16,008 INFO [kvm.resource.LibvirtComputingResource]
(main:null) (logid:) iscsi session clean up is disabled
2025-01-23 03:45:16,010 INFO [kvm.resource.LibvirtComputingResource]
(main:null) (logid:) The [vm.memballoon.stats.period] property is set to '0',
this prevents memory statistics from being displayed correctly. Adjust
(increase) the value of this parameter to correct this.
2025-01-23 03:45:16,148 INFO [cloud.agent.Agent] (main:null) (logid:) Agent
[id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 :
host = 192.168.10.221 : port = 8250
2025-01-23 03:45:16,159 INFO [utils.nio.NioClient] (main:null) (logid:)
Connecting to 192.168.10.221:8250
2025-01-23 03:45:16,164 INFO [utils.nio.Link] (main:null) (logid:) Conf file
found: /etc/cloudstack/agent/agent.properties
2025-01-23 03:45:16,688 INFO [utils.nio.NioClient] (main:null) (logid:) SSL:
Handshake done
2025-01-23 03:45:16,690 INFO [utils.nio.NioClient] (main:null) (logid:)
Connected to 192.168.10.221:8250
2025-01-23 03:45:16,757 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null)
(logid:) Fetching CPU speed from command "lscpu".
2025-01-23 03:45:16,792 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null)
(logid:) Command [lscpu | grep -i 'Model name' | head -n 1 | egrep -o
'[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] resulted in the value [3200]
for CPU speed.
2025-01-23 03:45:16,840 INFO [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) (logid:) Host uses control group [tmpfs].
2025-01-23 03:45:16,841 INFO [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) (logid:) Setting host CPU max capacity to 0, as it uses
cgroup v1.
2025-01-23 03:45:17,004 ERROR [kvm.resource.LibvirtConnection]
(Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
connection pointer in virConnectGetVersion
2025-01-23 03:45:17,060 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Attempting to create storage pool
330e58d9-1f0d-48e1-82ac-e7aae6414e62 (Filesystem) in libvirt
2025-01-23 03:45:17,066 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Found existing defined storage pool
330e58d9-1f0d-48e1-82ac-e7aae6414e62, using it.
2025-01-23 03:45:17,067 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Trying to fetch storage pool
330e58d9-1f0d-48e1-82ac-e7aae6414e62 from libvirt
2025-01-23 03:45:17,131 ERROR [kvm.resource.LibvirtComputingResource]
(Thread-4:null) (logid:) LibvirtException was thrown in event loop:
org.libvirt.LibvirtException: An error occurred, but the cause is unknown
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Library.processEvent(Unknown Source)
at org.libvirt.Library.runEventLoop(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.lambda$setupLibvirtEventListener$0(LibvirtComputingResource.java:3699)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-01-23 03:45:17,141 ERROR [kvm.resource.LibvirtComputingResource]
(Agent-Handler-1:null) (logid:) Failed to get libvirt connection for domain
event lifecycle
org.libvirt.LibvirtException: internal error: could not initialize domain event
timer
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainEventRegister(Unknown Source)
at org.libvirt.Connect.domainEventRegister(Unknown Source)
at org.libvirt.Connect.addLifecycleListener(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.setupLibvirtEventListener(LibvirtComputingResource.java:3712)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createLocalStoragePool(LibvirtComputingResource.java:3692)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.initialize(LibvirtComputingResource.java:3636)
at com.cloud.agent.Agent.sendStartup(Agent.java:460)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1125)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-01-23 03:45:17,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Process agent startup answer, agent id = 0
2025-01-23 03:45:17,274 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Set agent id 0
2025-01-23 03:45:17,301 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Startup Response Received: agent id = 0
GitHub link: https://github.com/apache/cloudstack/discussions/10248
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscr...@cloudstack.apache.org
