GitHub user Shivamkashyap08 created a discussion: Host in Alert state after
CloudStack Upgrade to 4.20.2.0
Hi CloudStack Community,
We recently upgraded our CloudStack environment from version 4.19.2.0 to
4.20.2.0. The upgrade process completed successfully; however, we are
encountering an issue post-upgrade.
Environment Details:
Current Cloudstack Version - 4.19.2.0
Need to Upgrade to CloudStack Version: 4.20.2.0
Hypervisor: Citrix XenServer
Zone Types: EC Zone & VPC Zone(both)
Issue Description:
After the upgrade:
Hosts in the VPC Zone are functioning correctly and appear in the UP state in
the CloudStack UI. However, hosts in the EC Zone have gone into an Alert state
and are not transitioning back to UP.
Observations:
Upon analyzing the logs, we observed failures related to the callHostPlugin.
After logging into the hypervisor, we found that the task host.call_plugin is
stuck in a pending state.
Request:
Could you please help us identify the root cause of this issue and suggest
possible steps to resolve it?
I have attached the relevant error logs and hypervisor task list output for
reference.
2026-03-28 03:21:35,277 INFO [c.c.a.m.ClusteredAgentManagerImpl]
(DirectAgentCronJob-4:[ctx-8fed6763]) (logid:e1cdc087) PingMap for agent: 70098
(null) will not be updated because agent is no longer in the PingMap
2026-03-28 03:21:35,280 DEBUG [c.c.h.x.r.XenServerConnectionPool]
(DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) XmlRpcException for method:
host.call_plugin due to org.apache.xmlrpc.XmlRpcException: Failed to create
input stream: Read timed out
2026-03-28 03:21:35,280 DEBUG [c.c.h.x.r.XenServer650Resource]
(DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) callHostPlugin failed for cmd:
can_bridge_firewall with args instance: VM, host_uuid:
2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream:
Read timed out
2026-03-28 03:21:35,280 WARN [c.c.h.x.r.w.x.CitrixSetupCommandWrapper]
(DirectAgent-46:[ctx-ea6d6da5]) (logid:59e687ae) Unable to setup
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd:
can_bridge_firewall with args instance: VM, host_uuid:
2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream:
Read timed out
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:380)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.canBridgeFirewall(CitrixResourceBase.java:497)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:67)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:44)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1778)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-03-28 03:21:35,280 DEBUG [c.c.a.m.D.Task] (DirectAgent-46:[ctx-ea6d6da5])
(logid:59e687ae) Seq 70097-597289900580012033: Response Received:
2026-03-28 03:21:35,281 DEBUG [c.c.a.t.Request] (DirectAgent-46:[ctx-ea6d6da5])
(logid:59e687ae) Seq 70097-597289900580012033: Processing: { Ans: , MgmtId:
165385370412082, via: 70097, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.SetupAnswer":{"_reconnect":"true","result":"false","details":"callHostPlugin
failed for cmd: can_bridge_firewall with args instance: VM, host_uuid:
2ec20f60-ea68-4ec8-89bb-3379c14395f5, due to Failed to create input stream:
Read timed out","wait":"0","bypassHostMaintenance":"false"}}] }
Cmd - xe task-list
uuid ( RO) : 77840f1d-70fb-2a31-c941-ed93c75e68be
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
uuid ( RO) : 2ac624c1-9584-d56e-cae7-1988fba8c0b2
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
uuid ( RO) : 6a9ce2ab-3353-4bc3-b23c-b17cf4eafb28
name-label ( RO): host.call_plugin
name-description ( RO):
status ( RO): pending
progress ( RO): 0.000
Please note - I have rolled back the upgrade, and it is running on CloudStack
version 4.19.2.0.
Thanks in advance for your support.
Thanks
Shivam Kashyap
GitHub link: https://github.com/apache/cloudstack/discussions/13017
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
