Hi, all:
I am upgrading my ACS clusters from XenServer 6.5 to XenServer 7.0. I am on
ACS version 4.9.3.0. On this ACS instance, I have another fully functioning
XenServer 7.0 cluster already.
This time, after I upgraded the pool master, it remains in “Alert” state, while
all the slave hosts eventually are in “Up” state. Attempts to reconnect the
host (via UI or API) or restart management service have no effects.
Looking at catalina.out log, there is an error executing following command on
the host: xe sm-list | grep "resigning of duplicates", what exactly does this
command do and how to fix it?
Note: I did a manual upgrade of the pool master (from XenServer 7.0 ISO
image), in order to keep the existing partition table and cluster
configurations, and following are the error logs from catalina.out file:
Yiping
INFO [c.c.h.x.r.CitrixResourceBase] (AgentTaskPool-4:ctx-7e09325c) XenServer
Version is 7.0.0 for host 10.0.1.18
INFO [c.c.h.x.r.CitrixResourceBase] (AgentTaskPool-4:ctx-7e09325c) Private
Network is mgmt for host 10.0.1.18
INFO [c.c.h.x.r.CitrixResourceBase] (AgentTaskPool-4:ctx-7e09325c) Guest
Network is mgmt for host 10.0.1.18
INFO [c.c.h.x.r.CitrixResourceBase] (AgentTaskPool-4:ctx-7e09325c) Public
Network is mgmt for host 10.0.1.18
ERROR [c.c.u.s.SshHelper] (AgentTaskPool-11:ctx-3ef0dede) SSH execution of
command xe sm-list | grep "resigning of duplicates" has an error
status code in return. Result output:
INFO [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-11:ctx-3ef0dede) Host:
xxxxxxxx connected with hypervisor type: XenServer. Checking CIDR...
INFO [c.c.a.m.DirectAgentAttache] (AgentTaskPool-11:ctx-3ef0dede)
StartupAnswer received 71 Interval = 60
WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-11:ctx-3ef0dede)
defaulting to xenserver650 resource for product brand: XenServer with product
version: 7.0.0
INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-c04388fd) Host
10.0.1.18 OpaqueRef:3a71d366-1db2-b082-93e0-73a70dd9d409: Host 10.0.1.18 is
already setup.
INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-c04388fd) Host
10.0.1.18 OpaqueRef:3a71d366-1db2-b082-93e0-73a70dd9d409: Host 10.0.1.18 is
already setup.
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-c04388fd)
callHostPlugin failed for cmd: setIptables with args due to The requested
plugin could not be found.
WARN [c.c.h.x.r.w.x.CitrixSetupCommandWrapper] (DirectAgent-219:ctx-c04388fd)
Unable to setup
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd:
setIptables with args due to The requested plugin could not be found.
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:340)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.setIptables(CitrixResourceBase.java:4555)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:63)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:45)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1693)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-11:ctx-3ef0dede) Unable to
setup agent 71 due to callHostPlugin failed for cmd: setIptables with args due
to The requested plugin could not be found.
INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-11:ctx-3ef0dede) Could not
find exception: com.cloud.exception.ConnectionException in error code list for
exceptions
WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-11:ctx-3ef0dede) Monitor
XcpServerDiscoverer says there is an error in the connect process for 71 due to
Reinitialize agent after setup.
INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-11:ctx-3ef0dede) Host 71 is
disconnecting with event AgentDisconnected
WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-11:ctx-3ef0dede) Unable to
connect due to
com.cloud.exception.ConnectionException: Reinitialize agent after setup.
at
com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:627)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:567)
at
com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1521)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1909)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2042)
at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy160.createHostAndAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.runInContext(AgentManagerImpl.java:1138)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
