Hi Dan, Network is definitely not a problem. iptables is disabled on MS and both hosts. I am able to ssh from MS to both hosts. I am also able to connect to port 80 from the MS.
I need to head out now and will try what you suggested in a couple hours. thanks, Carlos On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland <daan.hoogl...@gmail.com>wrote: > Carlos, > > can the ms reach the xs hosts? > if so try this, Stop the ms, set the state for the vm_instances for > cpvm and vr from expunging to stopped. and restart. > if not you need to troubleshoot your network connectivity. It seems > this would not be the issue as you only stopped and started it. Did > you > > Daan > > On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui <car...@reategui.com> > wrote: > > Hi Dan, > > > > Unfortunately not. Still trying to solve this. See my last email to > Ahmad > > in the recover cs thread about next steps I am trying to figure out. > > > > The CPVM and VR are not running. They are currently in expunging state. > > > > If I try to start an instance the MS is unable to communicate the start > vm > > commands to my XS hosts and then starts writing to the logs at an > amazingly > > high rate. You can see one of my logs here: > > http://reategui.com/cloudstack/management-server.log.new > > > > thank you > > Carlos > > > > > > > > > > > > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland <daan.hoogl...@gmail.com> > > wrote: > >> > >> H Carlos, > >> > >> Did you solve this problem yet? > >> > >> You can try removing the cpvm and vr at the hypervisor and change > >> their state to 'Stopped' in the db. > >> > >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui <car...@reategui.com> > >> wrote: > >> > In the last 15 minutes management log file it grew 500MB.. > >> > > >> > Any ideas what I should look at to figure out what is happening? > >> > > >> > > >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui > >> > <create...@gmail.com>wrote: > >> > > >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on > ubuntu + > >> >> XS > >> >> 6.0.2) that I shutdown and cannot bring back to life. > >> >> > >> >> I tried to destroy the CPVM and the VR and all I am seeing in the > logs > >> >> is > >> >> the following and they are filling up fast (management log is almost > >> >> 1GB). > >> >> > >> >> > >> >> 2013-10-29 11:51:08,291 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq > >> >> 1-201919186: > >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq > >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, > Flags: > >> >> 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq > >> >> 1-201919186: > >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq > >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, > Flags: > >> >> 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq > >> >> 1-201919186: > >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq > >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, > Flags: > >> >> 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq > >> >> 1-201919186: > >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq > >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, > Flags: > >> >> 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq > >> >> 1-201919186: > >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> 2013-10-29 11:51:08,299 DEBUG [agent.manager.ClusteredAgentAttache] > >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq > >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, > Flags: > >> >> 100111, > >> >> > >> >> > [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] > >> >> } to 233845174730255 > >> >> > >> >> > >> >> > >> >> > > > > >
