I do not believe that should happen as the VM is primarily tied to the storage. But just to avoid taking any chances, we could do this: 1. Put the failed host in to maintenance. 2. Set host_id and last_host_id to NULL in vm_instance table for that VM. 3. Make sure the XS pool doesn't have any stale data for this VM. Look for the VM and associated VDI in the pool one more time. I noticed there is a space "name-label= i-574-14584-VM" after the equal operator, remove that space and try again or use XenCenter.
If #3 doesn't raise any flags, go ahead and try starting the VM again. There is also another alternative, you could register the root disk of the VM as a template and launch a new VM from that template. Regards, Somesh -----Original Message----- From: Jaime Orlando Rojas Sanchez [mailto:[email protected]] Sent: Thursday, August 20, 2015 1:00 PM To: <[email protected]> Subject: Re: VM stuck in a failing Host Yes we could , but I'm not sure if maybe I lost the VM because it is tied to that host. Cordialmente, Jaime Rojas > On 20/08/2015, at 11:47 a.m., Somesh Naidu <[email protected]> wrote: > > Can you not remove the failed host from CCP and XS cluster? > > Regards, > Somesh > > > -----Original Message----- > From: Jaime Orlando Rojas Sanchez [mailto:[email protected]] > Sent: Thursday, August 20, 2015 12:38 PM > To: [email protected] > Subject: RE: VM stuck in a failing Host > > Hello, > > state/status of the failed host in CS? = Disconnected > > We run the command in the 2 remaining host with no results. > > [root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM > [root@dc1fdtptgcx04 /]# > > [root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM > [root@dc1fdtptgcx02 /]# > > > > Regards / Cordialmente, > > Jaime O. Rojas S. > Technology Manager > [email protected] > Mobile: +57 301-3382382 > Office: +57-1-8766767 x215 > > -----Mensaje original----- > De: Somesh Naidu [mailto:[email protected]] > Enviado el: jueves, 20 de agosto de 2015 11:00 a. m. > Para: [email protected] > Asunto: RE: VM stuck in a failing Host > > Quick question, what is the state/status of the failed host in CS? Also, look > up the particular VM on XS pool (xe vm-list name-label= i-574-14584-VM), what > does it say? > > Regards, > Somesh > > > -----Original Message----- > From: Jaime Orlando Rojas Sanchez [mailto:[email protected]] > Sent: Thursday, August 20, 2015 11:37 AM > To: [email protected] > Subject: RE: VM stuck in a failing Host > > Following the logs when I click 'run' in ACS after did the following in the DB > > > - Change the state to 'stopped' > > - Change host ID to a working host > > - Change last host ID to a working host > > - Check VR is up and running on a working host > > > -bash-4.1# tail -f management-server.log | grep 14584 > 2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-21:null) submit async job-45973 = [ > d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973, > userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, > instanceId: 14584, cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, > cmdOriginator: null, cmdInfo: > {"response":"json","id":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","sessionkey":"hwnxmM0He9EXs2craugKg3XyWL4\u003d","cmdEventType":"VM.STOP","ctxUserId":"2","httpmethod":"GET","_":"1440067422009","ctxAccountId":"2","ctxStartEventId":"38650949","forced":"true"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2015-08-20 05:46:33,404 DEBUG [agent.transport.Request] > (Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536 ]) Seq > 792-1073807425: Sending { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1, > Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/574/57764/28586b35-cb45-4565-bd9b-7aa46a2898da","volume":{"uuid":"a15d0923-0a25-408f-9d10-fd5d47b3fef9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_3000GB_ADV_SATA1","id":211,"poolType":"PreSetup","host":"localhost","path":"/3PAR_3000GB_ADV_SATA1","port":0}},"name":"ROOT-14584","size":107374182400,"path":"c7a8eebc-7750-455c-804f-64c0d66cb4f4","volumeId":57764,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":57764,"hypervisorType":"XenServer"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://172.16.4.65/vol/secondary_clpr","_role":"Image"}},"vmName":"i-574-14584-VM","name":"srvrasautos2_ROOT-14584_20141007233517","hypervisorType":"XenServer","id":11831}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/574/599","uuid":"70d21214-33d0-49e0-8b45-c7702b0fe579","id":599,"format":"RAW","accountId":574,"hvm":true,"displayText":"templateras","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://172.16.4.65/vol/secondary_clpr","_role":"Image"}},"name":"248e2097b-4af7-38f7-a851-029ef11f52cc","hypervisorType":"XenServer"}},"executeInSequence":true,"wait":10800}}] > } > 2015-08-20 05:49:03,212 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-1:null) submit async job-45975 = [ > 8ac23585-989d-4e3d-bcb9-3d3602842b8f ], details: AsyncJobVO {id:45975, > userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, > instanceId: 14584, cmd: org.apache.cloudstack.api.command.user.vm.StartVMCmd, > cmdOriginator: null, cmdInfo: > {"response":"json","id":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","sessionkey":"hwnxmM0He9EXs2craugKg3XyWL4\u003d","cmdEventType":"VM.START","ctxUserId":"2","httpmethod":"GET","_":"1440067641673","ctxAccountId":"2","ctxStartEventId":"38651246"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 139549854171544, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2015-08-20 05:49:05,821 DEBUG [cloud.network.NetworkManagerImpl] > (Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking > VirtualRouter to prepare for > Nic[194602-14584-5d82a92d-b828-45bc-882a-b5ce17401812-172.16.100.244] > 2015-08-20 05:49:08,947 DEBUG [cloud.network.NetworkManagerImpl] > (Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking > VirtualRouter to prepare for Nic[194621-14584-null-172.16.180.35] > 2015-08-20 05:49:10,181 DEBUG [cloud.storage.VolumeManagerImpl] > (Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) No > need to recreate the volume: Vol[13270|vm=14584|DATADISK], since it already > has a pool assigned: 208, adding disk to VM > 2015-08-20 05:49:10,184 DEBUG [cloud.storage.VolumeManagerImpl] > (Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) No > need to recreate the volume: Vol[57764|vm=14584|ROOT], since it already has a > pool assigned: 211, adding disk to VM > 2015-08-20 05:49:10,271 DEBUG [agent.transport.Request] > (Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Seq > 595-838074847: Sending { Cmd , MgmtId: 139549854171544, via: 595, Ver: v1, > Flags: 100111, > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":14584,"name":"i-574-14584-VM","bootloader":"PyGrub","type":"User","cpus":2,"minSpeed":525,"maxSpeed":2100,"minRam":4294967296,"maxRam":4294967296,"arch":"x86_64","os":"CentOS > 6.0 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"63729fa8c6c9ecae","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:true;apic:true;pae:true;nx:false","Message.ReservedCapacityFreed.Flag":"true","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"71095933-5ce2-4786-8527-1dbe11876004","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_2000GB_ADV_SAS","id":208,"poolType":"PreSetup","host":"localhost","path":"/3PAR_2000GB_ADV_SAS","port":0}},"name":"Datos","size":214748364800,"path":"69451be5-bd65-41d4-b465-08933d393498","volumeId":13270,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":13270,"hypervisorType":"XenServer"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a15d0923-0a25-408f-9d10-fd5d47b3fef9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_3000GB_ADV_SATA1","id":211,"poolType":"PreSetup","host":"localhost","path":"/3PAR_3000GB_ADV_SATA1","port":0}},"name":"ROOT-14584","size":107374182400,"path":"c7a8eebc-7750-455c-804f-64c0d66cb4f4","volumeId":57764,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":57764,"hypervisorType":"XenServer"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"040aaa12-4338-4181-bb60-9dc07aa804e8","ip":"172.16.100.244","netmask":"255.255.252.0","gateway":"172.16.100.1","mac":"02:00:3b:b1:00:16","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://3042","isolationUri":"vlan://3042","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"},{"deviceId":1,"networkRateMbps":200,"defaultNic":false,"uuid":"3ef8c0d3-83c0-4569-990a-577ecd21f707","ip":"172.16.180.35","netmask":"255.255.255.240","gateway":"172.16.180.33","mac":"06:07:8c:00:0f:0d","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://182","isolationUri":"vlan://182","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"}]},"hostIp":"172.16.1.11","executeInSequence":true,"wait":0}}] > } > 2015-08-20 05:49:10,273 DEBUG [agent.transport.Request] > (Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Seq > 595-838074847: Executing: { Cmd , MgmtId: 139549854171544, via: 595, Ver: > v1, Flags: 100111, > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":14584,"name":"i-574-14584-VM","bootloader":"PyGrub","type":"User","cpus":2,"minSpeed":525,"maxSpeed":2100,"minRam":4294967296,"maxRam":4294967296,"arch":"x86_64","os":"CentOS > 6.0 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"63729fa8c6c9ecae","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:true;apic:true;pae:true;nx:false","Message.ReservedCapacityFreed.Flag":"true","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"71095933-5ce2-4786-8527-1dbe11876004","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_2000GB_ADV_SAS","id":208,"poolType":"PreSetup","host":"localhost","path":"/3PAR_2000GB_ADV_SAS","port":0}},"name":"Datos","size":214748364800,"path":"69451be5-bd65-41d4-b465-08933d393498","volumeId":13270,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":13270,"hypervisorType":"XenServer"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a15d0923-0a25-408f-9d10-fd5d47b3fef9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_3000GB_ADV_SATA1","id":211,"poolType":"PreSetup","host":"localhost","path":"/3PAR_3000GB_ADV_SATA1","port":0}},"name":"ROOT-14584","size":107374182400,"path":"c7a8eebc-7750-455c-804f-64c0d66cb4f4","volumeId":57764,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":57764,"hypervisorType":"XenServer"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"040aaa12-4338-4181-bb60-9dc07aa804e8","ip":"172.16.100.244","netmask":"255.255.252.0","gateway":"172.16.100.1","mac":"02:00:3b:b1:00:16","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://3042","isolationUri":"vlan://3042","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"},{"deviceId":1,"networkRateMbps":200,"defaultNic":false,"uuid":"3ef8c0d3-83c0-4569-990a-577ecd21f707","ip":"172.16.180.35","netmask":"255.255.255.240","gateway":"172.16.180.33","mac":"06:07:8c:00:0f:0d","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://182","isolationUri":"vlan://182","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"}]},"hostIp":"172.16.1.11","executeInSequence":true,"wait":0}}] > } > 2015-08-20 05:49:10,402 DEBUG [xen.resource.CitrixResourceBase] > (DirectAgent-434:null) VM i-574-14584-VM is runing on host > 53109eef-2f53-4f0e-a763-68817d573bd9 > 2015-08-20 05:49:10,403 DEBUG [xen.resource.CitrixResourceBase] > (DirectAgent-434:null) The VM is in stopped state, detected problem during > startup : i-574-14584-VM > 2015-08-20 05:49:10,404 DEBUG [agent.transport.Request] > (DirectAgent-434:null) Seq 595-838074847: Processing: { Ans: , MgmtId: > 139549854171544, via: 595, Ver: v1, Flags: 110, > [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":14584,"name":"i-574-14584-VM","bootloader":"PyGrub","type":"User","cpus":2,"minSpeed":525,"maxSpeed":2100,"minRam":4294967296,"maxRam":4294967296,"arch":"x86_64","os":"CentOS > 6.0 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"63729fa8c6c9ecae","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:true;apic:true;pae:true;nx:false","Message.ReservedCapacityFreed.Flag":"true","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"71095933-5ce2-4786-8527-1dbe11876004","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_2000GB_ADV_SAS","id":208,"poolType":"PreSetup","host":"localhost","path":"/3PAR_2000GB_ADV_SAS","port":0}},"name":"Datos","size":214748364800,"path":"69451be5-bd65-41d4-b465-08933d393498","volumeId":13270,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":13270,"hypervisorType":"XenServer"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a15d0923-0a25-408f-9d10-fd5d47b3fef9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_3000GB_ADV_SATA1","id":211,"poolType":"PreSetup","host":"localhost","path":"/3PAR_3000GB_ADV_SATA1","port":0}},"name":"ROOT-14584","size":107374182400,"path":"c7a8eebc-7750-455c-804f-64c0d66cb4f4","volumeId":57764,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":57764,"hypervisorType":"XenServer"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"040aaa12-4338-4181-bb60-9dc07aa804e8","ip":"172.16.100.244","netmask":"255.255.252.0","gateway":"172.16.100.1","mac":"02:00:3b:b1:00:16","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://3042","isolationUri":"vlan://3042","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"},{"deviceId":1,"networkRateMbps":200,"defaultNic":false,"uuid":"3ef8c0d3-83c0-4569-990a-577ecd21f707","ip":"172.16.180.35","netmask":"255.255.255.240","gateway":"172.16.180.33","mac":"06:07:8c:00:0f:0d","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://182","isolationUri":"vlan://182","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"}]},"host_guid":"53109eef-2f53-4f0e-a763-68817d573bd9","result":true,"details":"VM > i-574-14584-VM is runing on host > 53109eef-2f53-4f0e-a763-68817d573bd9","wait":0}}] } > 2015-08-20 05:49:19,614 WARN [xen.resource.CitrixResourceBase] > (DirectAgent-41:null) Detecting a new state but couldn't find a old state so > adding it to the changes: i-574-14584-VM > 2015-08-20 05:49:19,615 DEBUG [agent.transport.Request] (DirectAgent-41:null) > Seq 566-1609629709: Processing: { Ans: , MgmtId: 139549854171544, via: 566, > Ver: v1, Flags: 10, > [{"com.cloud.agent.api.ClusterSyncAnswer":{"_clusterId":5,"_newStates":{"i-574-14584-VM":{"t":"53109eef-2f53-4f0e-a763-68817d573bd9","u":"Running","v":"viridian:true;acpi:true;apic:true;pae:true;nx:false"}},"_isExecuted":false,"result":true,"wait":0}}] > } > 2015-08-20 05:49:19,627 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (DirectAgent-41:null) VM i-574-14584-VM: cs state = Running and realState = > Running > 2015-08-20 05:49:19,627 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (DirectAgent-41:null) VM i-574-14584-VM: cs state = Running and realState = > Running > > Regards / Cordialmente, > > Jaime O. Rojas S. > Technology Manager > [email protected]<mailto:[email protected]> > Mobile: +57 301-3382382 > Office: +57-1-8766767 x215 > > De: Jaime Orlando Rojas Sanchez > Enviado el: jueves, 20 de agosto de 2015 9:54 a. m. > Para: '[email protected]' > Asunto: VM stuck in a failing Host > > Hello, > > We have a 4.2.1 ACS, running on XenServer 6.2.0, we have a zone with a pool > of 3 host, yesterday 1 host crash and OS get corrupted. I think we lost that > host and have to reinstall it, but the issue is that we had a couple of VM > and VR running on that host. The failing host was the master of the pool, so > once it fails all the pool was disconnected, we change the master role and > recover pool management from Xencenter and ACS, once we did it a VM moved to > the remaining host, all VR and 1 VM kept stuck in failing host. > > In DB we see the VR and VM running, even if the host was marked as down and > maintenance. We changed the VR state to 'stopped' and change de "last host > ID" and "Host ID" to a working host. Once we did it we were able to destroy > the VR and recreate them with successful results, they came up on working > host. If we change only the state, the VR couldn't be destroyed. Here we > workaround with the 70% of the outage, BUT one VM remain stuck to the host, > we change the state, the last host ID, but once we press start, it "runs" on > the failing host and the VM appears as running even if it doesn't. Any > suggestion to force the VM to start in a different host and remove it from > the failing host? This is a critical VM, we hope somebody else could give us > a hand. > > Regards / Cordialmente, > > Jaime O. Rojas S. > Technology Manager > [email protected]<mailto:[email protected]> > Mobile: +57 301-3382382 > Office: +57-1-8766767 x215 > > > Email asegurado por Check Point > > Email asegurado por Check Point
