vishesh92 opened a new issue, #14391:
URL: https://github.com/apache/cloudstack/issues/14391
### problem
If a host's network is unreachable and the systemVM is stopped and removed,
when the host comes back up again, I see the below errors in the log and the
running systemVMs are not shown in the UI.
```
2026-10-13 08:27:29,001 ERROR [c.c.h.d.HostDaoImpl]
(AgentConnectTaskPool-93:[ctx-e2dc4381]) (logid:f95ffe51) DB Exception on:
HikariProxyPreparedStatement@140904250 wrapping
com.mysql.cj.jdbc.ServerPreparedStatement[233]: INSERT INTO host
(host.disconnected, host.name, host.status, host.type, host.private_ip_address,
host.private_mac_address, host.private_netmask, host.public_netmask,
host.public_ip_address, host.public_mac_address, host.storage_ip_address,
host.cluster_id, host.storage_netmask, host.storage_mac_address,
host.storage_ip_address_2, host.storage_netmask_2, host.storage_mac_address_2,
host.hypervisor_type, host.proxy_port, host.resource, host.fs_type,
host.available, host.setup, host.resource_state, host.hypervisor_version,
host.arch, host.update_count, host.uuid, host.storage_access_groups,
host.data_center_id, host.pod_id, host.cpu_sockets, host.cpus, host.url,
host.speed, host.ram, host.parent, host.guid, host.capabilities,
host.total_size, host.last_ping, host.m
gmt_server_id, host.last_mgmt_server_id, host.dom0_memory, host.version,
host.created) VALUES (NULL, x'732d31342d564d', 'Creating',
'SecondaryStorageVM', x'31302e302e33362e3637',
x'31653a30313a37303a30303a30303a3162', x'3235352e3235352e3234302e30',
x'3235352e3235352e3234302e30', x'31302e302e35322e3631',
x'31653a30313a36613a30303a30303a3031', x'31302e302e33362e3637', NULL,
x'3235352e3235352e3234302e30', x'31653a30313a37303a30303a30303a3162', NULL,
NULL, NULL, NULL, NULL, NULL, NULL, 1, 0, 'Creating', NULL, 'x86_64', 0,
x'30656439363763342d383733362d343638652d626130322d393337303762393030646364',
NULL, 1, 1, NULL, NULL, x'4e6f49716e', NULL, 0, NULL,
x'732d31342d564d2d5072656d69756d5365636f6e6461727953746f726167655265736f75726365',
NULL, NULL, 1749882860, 32988771385657, NULL, 0, x'342e32322e312e31',
'2026-10-13 08:27:29') java.sql.SQLIntegrityConstraintViolationException:
Duplicate entry 's-14-VM-PremiumSecondaryStorageResource' for key 'host.guid'
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:118)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
at
com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
at
com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1639)
at com.cloud.host.dao.HostDaoImpl.persist(HostDaoImpl.java:1005)
at com.cloud.host.dao.HostDaoImpl.persist(HostDaoImpl.java:85)
at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy112.persist(Unknown Source)
at
com.cloud.resource.ResourceManagerImpl.persistNewHost(ResourceManagerImpl.java:3316)
at
com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:3286)
at
com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:3633)
at jdk.internal.reflect.GeneratedMethodAccessor257.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at
jdk.proxy3/jdk.proxy3.$Proxy217.createHostVOForConnectedAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1438)
at
com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1538)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-10-13 08:27:29,002 DEBUG [c.c.u.d.T.Transaction]
(AgentConnectTaskPool-93:[ctx-e2dc4381]) (logid:f95ffe51) Rolling back the
transaction: Time = 2 Name = AgentConnectTaskPool-93; called by
-TransactionLegacy.rollback:905-TransactionLegacy.removeUpTo:848-TransactionLegacy.close:672-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:175-ExposeInvocationInterceptor.invoke:97-ReflectiveMethodInvocation.proceed:186-JdkDynamicAopProxy.invoke:215-$Proxy112.persist:-1-ResourceManagerImpl.persistNewHost:3316-ResourceManagerImpl.createHostVO:3286-ResourceManagerImpl.createHostVOForConnectedAgent:3633
2026-10-13 08:27:29,002 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-93:[ctx-e2dc4381]) (logid:f95ffe51) Failed to handle host
connection: javax.persistence.EntityExistsException: Entity already exists
at
com.cloud.utils.db.GenericDaoBase.handleEntityExistsException(GenericDaoBase.java:892)
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1678)
at com.cloud.host.dao.HostDaoImpl.persist(HostDaoImpl.java:1005)
at com.cloud.host.dao.HostDaoImpl.persist(HostDaoImpl.java:85)
at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy112.persist(Unknown Source)
at
com.cloud.resource.ResourceManagerImpl.persistNewHost(ResourceManagerImpl.java:3316)
at
com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:3286)
at
com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:3633)
at jdk.internal.reflect.GeneratedMethodAccessor257.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at
jdk.proxy3/jdk.proxy3.$Proxy217.createHostVOForConnectedAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1438)
at
com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1538)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate
entry 's-14-VM-PremiumSecondaryStorageResource' for key 'host.guid'
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:118)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at
com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
at
com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
at
com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1639)
... 37 more
```
### versions
most likely all
### The steps to reproduce the bug
1. Block all traffic to the Management Servers
```sh
MS_IPS="10.0.0.10 10.0.0.11"
modprobe br_netfilter
sysctl -w net.bridge.bridge-nf-call-iptables=1
for ip in $MS_IPS; do
iptables -I OUTPUT -d "$ip" -j DROP # cloudstack-agent -> MS
iptables -I INPUT -s "$ip" -j DROP # MS -> host
iptables -I FORWARD -d "$ip" -j DROP # system VMs -> MS (bridged)
iptables -I FORWARD -s "$ip" -j DROP # MS -> system VMs
done
```
2. Force Stop the systemVM running on the host via the UI and then destroy
it from the UI. Now the systemVM is marked as removed in the management server
but the VM is still running. New systemVMs should start coming up after this.
3. Flush the iptables `iptables -F` on the host to restore connectivity with
the management server
4. After this, you will see the above error in the logs.
### What to do about it?
Remove the unique constraint or recover the older (removed) row and update
that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
