Hi Koushik, We've managed to remove the error now regarding host id 27708.... this was related to a very old secondary storage vm which was in a status of disconnected but had not been removed. It had long since been deleted.
However, we are still seeing the error, it is very similar to what is described here: https://issues.apache.org/jira/browse/CLOUDSTACK-3715 Still seeing this: ERROR [o.a.c.s.m.XenServerStorageMotionStrategy] (Work-Job-Executor-6:ctx-8e19de38 job-346708/job-346709 ctx-65bccb97) Error while migrating vm VM[User|i-2-26631-VM] to host Host[-181-Routing] com.cloud.exception.OperationTimedoutException: Commands 964614745187418300 to Host 181 timed out after 1200 The migration never happens, its as though the call to the xen host is never made, or never received by xen. What ports are used to make this call? I assume either 22 or 443? Cheers! On Wed, Sep 30, 2015 at 12:50 PM, Koushik Das <[email protected]> wrote: > As per the logs the host id 27708 is not in the right state and operation > is failing. Check in DB for the status of host id 27708. > Also grep for job-343376 and job-343377 in all the MS logs to trace in > which MS the job originated and where it is getting executed. > > -----Original Message----- > From: cs user [mailto:[email protected]] > Sent: Wednesday, 30 September 2015 16:01 > To: [email protected] > Subject: Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk > > Hi Koushik, > > We see errors such as : > > com.cloud.utils.exception.CloudRuntimeException: Failed to send command, > due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource > [Host:27708] is unreachable: Host 27708: Host with specified id is not in > the right state: Disconnected ERROR > [o.a.c.s.m.XenServerStorageMotionStrategy] > (Work-Job-Executor-159:ctx-6e40204e job-343376/job-343377 ctx-818f1a51) > copy failed ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-159:ctx-6e40204e > job-343376/job-343377 ctx-818f1a51) Invocation exception, caused by: > com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm > VM[User|i-2-26383-VM] along with its volumes. > INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-159:ctx-6e40204e > job-343376/job-343377 ctx-818f1a51) Rethrow exception > com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm > VM[User|i-2-26383-VM] along with its volumes. > com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm > VM[User|i-2-26383-VM] along with its volumes. > com.cloud.utils.exception.CloudRuntimeException: Failed to send command, > due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource > [Host:27708] is unreachable: Host 27708: Host with specified id is not in > the right state: Disconnected > > However, when I look at the hosts in cloudstack, all hosts are connected > and green, none are reported by cloudstack as being disconnected. So I'm > not really quite sure what is going wrong. As I say, when only one > cloudstack management server is running, everything works perfectly fine. > > Cheers > > On Wed, Sep 30, 2015 at 10:27 AM, Koushik Das <[email protected]> > wrote: > > > Check logs on all 4 MS in the cluster around the time the operation > > was done. If you see any error/exceptions that can point to the problem. > > > > -----Original Message----- > > From: cs user [mailto:[email protected]] > > Sent: Tuesday, 29 September 2015 20:53 > > To: [email protected] > > Subject: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk > > > > Hi Folks, > > > > We are seeing some problems when attempting live migrations in a > > clustered cloudstack environment. We have 4 management servers, and > > when we attempt a migration of an instance (local storage) from one > > host to another, the migration times out. > > > > However, if we only have one management server running, the migration > > works fine and we can move the instance around between hosts in the > > same pod without any problems at all. > > > > Just wanted to know if anyone else had come across this issue? > > > > Cheers > > >
