Hi, Dag:
Thanks for remind me the backup partition! Yesterday I was in a bit of panic
mode when I sent that message because there are over 100 busy VM instances for
QE are running on this cluster and this week is our release week!
I have restored the master from this backup partition (Using XenServer 7.0
installation ISO) and now both Xen pool and ACS are happy. I'll leave this
cluster alone till after the release frenzy is over!
But seriously, I still would like to know what went wrong this time. I have
built two brand new XenServer 7.0 clusters and upgraded another two clusters
from 6.5 SP1 to 7.0 so far for three separate ACS instances (all running
version 4.9.3.0) without encounter this issue before.
In the log file, I saw following WARN message:
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-c04388fd)
callHostPlugin failed for cmd: setIptables with args due to The requested
plugin could not be found.
My questions are:
* Which plugin is requested by setIpTables cmd? What is its name and
expected full path?
* Is this plugin part of ACS or XenServer distribution?
* Where is it missing from, on Xen host or on management server (this
is unlikely since this ACS instance is managing six other XS 7.0 hosts already)?
Yiping
On 3/27/18, 12:56 AM, "Dag Sonstebo" <[email protected]> wrote:
Hi Yiping,
If I remember correctly a full ISO upgrade of a XenServer actually backs up
the existing version (OS partition), then installs a brand new XS version on
top before copying settings across from the backup. As a result you are
effectively looking at more or less a new install – and anything CloudStack
related may not have been copied across.
So – as Kristian said take a look at the docs and work out which files are
missing. If this still fails you may need to promote another poolmaster, eject
the broken host, rebuild and re-add it.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 27/03/2018, 08:36, "Kristian Liivak" <[email protected]> wrote:
Hi
Its really good question. I runned similar issue.
But did you fallow xen upgrade instarations from end of
http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.11/hypervisor/xenserver.html
And in my memory some paths where to copy files are changed and not
updated in documentation.
Lugupidamisega / Regards
Kristian Liivak
WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: [email protected]
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee
[email protected]
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
----- Original Message -----
From: "Yiping Zhang" <[email protected]>
To: "users" <[email protected]>
Sent: Monday, March 26, 2018 11:47:24 PM
Subject: upgraded XenServer host stays in Alert state
Hi, all:
I am upgrading my ACS clusters from XenServer 6.5 to XenServer 7.0. I
am on ACS version 4.9.3.0. On this ACS instance, I have another fully
functioning XenServer 7.0 cluster already.
This time, after I upgraded the pool master, it remains in “Alert”
state, while all the slave hosts eventually are in “Up” state. Attempts to
reconnect the host (via UI or API) or restart management service have no
effects.
Looking at catalina.out log, there is an error executing following
command on the host: xe sm-list | grep "resigning of duplicates", what exactly
does this command do and how to fix it?
Note: I did a manual upgrade of the pool master (from XenServer 7.0
ISO image), in order to keep the existing partition table and cluster
configurations, and following are the error logs from catalina.out file:
Yiping
INFO [c.c.h.x.r.CitrixResourceBase] (AgentTaskPool-4:ctx-7e09325c)
XenServer Version is 7.0.0 for host 10.0.1.18
INFO [c.c.h.x.r.CitrixResourceBase] (AgentTaskPool-4:ctx-7e09325c)
Private Network is mgmt for host 10.0.1.18
INFO [c.c.h.x.r.CitrixResourceBase] (AgentTaskPool-4:ctx-7e09325c)
Guest Network is mgmt for host 10.0.1.18
INFO [c.c.h.x.r.CitrixResourceBase] (AgentTaskPool-4:ctx-7e09325c)
Public Network is mgmt for host 10.0.1.18
ERROR [c.c.u.s.SshHelper] (AgentTaskPool-11:ctx-3ef0dede) SSH execution
of command xe sm-list | grep "resigning of duplicates" has an error
status code in return. Result output:
INFO [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-11:ctx-3ef0dede)
Host: xxxxxxxx connected with hypervisor type: XenServer. Checking CIDR...
INFO [c.c.a.m.DirectAgentAttache] (AgentTaskPool-11:ctx-3ef0dede)
StartupAnswer received 71 Interval = 60
WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-11:ctx-3ef0dede)
defaulting to xenserver650 resource for product brand: XenServer with product
version: 7.0.0
INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-c04388fd)
Host 10.0.1.18 OpaqueRef:3a71d366-1db2-b082-93e0-73a70dd9d409: Host 10.0.1.18
is already setup.
INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-c04388fd)
Host 10.0.1.18 OpaqueRef:3a71d366-1db2-b082-93e0-73a70dd9d409: Host 10.0.1.18
is already setup.
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-c04388fd)
callHostPlugin failed for cmd: setIptables with args due to The requested
plugin could not be found.
WARN [c.c.h.x.r.w.x.CitrixSetupCommandWrapper]
(DirectAgent-219:ctx-c04388fd) Unable to setup
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed
for cmd: setIptables with args due to The requested plugin could not be found.
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:340)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.setIptables(CitrixResourceBase.java:4555)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:63)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixSetupCommandWrapper.execute(CitrixSetupCommandWrapper.java:45)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1693)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-11:ctx-3ef0dede)
Unable to setup agent 71 due to callHostPlugin failed for cmd: setIptables with
args due to The requested plugin could not be found.
INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-11:ctx-3ef0dede)
Could not find exception: com.cloud.exception.ConnectionException in error code
list for exceptions
WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-11:ctx-3ef0dede)
Monitor XcpServerDiscoverer says there is an error in the connect process for
71 due to Reinitialize agent after setup.
INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-11:ctx-3ef0dede) Host
71 is disconnecting with event AgentDisconnected
WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-11:ctx-3ef0dede)
Unable to connect due to
com.cloud.exception.ConnectionException: Reinitialize agent after setup.
at
com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:627)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:567)
at
com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1521)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1909)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2042)
at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy160.createHostAndAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.runInContext(AgentManagerImpl.java:1138)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
