Sorry Carlos, family and $dayjob kept me from looking at this anymore. Glad you got it working.
Daan On Wed, Nov 6, 2013 at 8:08 PM, Carlos Reategui <car...@reategui.com> wrote: > Since I didn't hear back I bit the bullet and decided to make the DB > changes and I am back up and running. Woohoo. I'll post the specifics in > the other thread I started about the mshost id. > > > On Tue, Nov 5, 2013 at 11:36 AM, Carlos Reategui <car...@reategui.com>wrote: > >> Hi Daan, >> >> The mshost thing I mentioned previously has been bugging me and as I look >> through the db I think I have a problem here. I am guessing I have a >> botched 4.1.0 to 4.1.1 upgrade that I did not notice when I shut things >> down. >> >> The mshost table is showing 2 management servers that are supposedly on >> the same machine listening on the same port which is not really possible. >> So really only one of the 2 is up (I believe that would be the 4.1.1 >> version). The logs do seem to indicate something along those lines where >> the 2 management servers are failing to communicate with each other. >> >> If I look at the host table I see my 2 hosts but they say they are on >> version 4.1.0 and are pointing to management server id of the 4.1.0 line in >> the mshost table. >> >> I am wondering if this is the problem and why their is no communication to >> my hosts from the management server. >> >> Anyone know the schema well enough to help me update some stuff and >> recover my installation? Would an upgrade to 4.2 possibly cleanup my DB? >> >> thanks, >> Carlos >> >> >> >> >> On Tue, Nov 5, 2013 at 10:59 AM, Carlos Reategui <car...@reategui.com>wrote: >> >>> Hi Daan, >>> >>> I have been looking through my logs and can't figure out why the ssvm >>> does not want to start. >>> >>> Here is a link to my latest logs: >>> http://reategui.com/cloudstack/management-server-131105.log >>> >>> Maybe someone else can spot something. >>> >>> There is nothing on either the XenServer hosts logs. >>> >>> thanks, >>> Carlos >>> >>> >>> On Tue, Nov 5, 2013 at 12:55 AM, Daan Hoogland >>> <daan.hoogl...@gmail.com>wrote: >>> >>>> Carlos, >>>> >>>> It makes no sense trying to start the vms if not the ssvm is started >>>> first. >>>> >>>> I would say stop the system, clear the logs and start again. Then >>>> analyse the logs to see what is wrong with the secondary storage vm. >>>> >>>> On Tue, Nov 5, 2013 at 2:08 AM, Carlos Reategui <car...@reategui.com> >>>> wrote: >>>> > Hi Daan, >>>> > There is no trace of CS communicating with either of my XS hosts in the >>>> > SMLog file or the xensource.log files when I try to start one of my >>>> stopped >>>> > instances. >>>> > >>>> > thanks, >>>> > Carlos >>>> > >>>> > >>>> > >>>> > On Sat, Nov 2, 2013 at 4:50 AM, Daan Hoogland <daan.hoogl...@gmail.com >>>> >wrote: >>>> > >>>> >> H Carlos, >>>> >> >>>> >> I wouldn't know about the mshost and mshost_peer but they seem >>>> >> alright. The expunging ssvm are a real problem. I think Ahmed >>>> >> mentioned that as well. You are having problems getting your ssvm in >>>> >> the air. All other things are consequential. This is the first thing >>>> >> to look at. Check the logs and the SMLog on the hypervisor to see what >>>> >> happens when cs tries to start a ssvm. By your db you can see that it >>>> >> tried to start it several times. Those are the expunging vm_instance >>>> >> entries. >>>> >> >>>> >> regards, >>>> >> Daan >>>> >> >>>> >> On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <car...@reategui.com> >>>> >> wrote: >>>> >> > Hi Daan :) >>>> >> > >>>> >> > Primary and Secondary storage are on the MS. Primary is on a >>>> different >>>> >> > subnet and it is currently mounted (and visible as an NFS SR from >>>> >> > XenCenter) and working fine from the xen machines. I have also >>>> been able >>>> >> > to mount Secondary without a problem. >>>> >> > >>>> >> > Here are my current system VMs: >>>> >> > mysql> select >>>> >> > id,name,instance_name,state,host_id,type,vm_type,desired_state from >>>> >> > vm_instance where type!='User'; >>>> >> > >>>> >> >>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+ >>>> >> > | id | name | instance_name | state | host_id | type >>>> >> | >>>> >> > vm_type | desired_state | >>>> >> > >>>> >> >>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+ >>>> >> > | 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy >>>> >> | >>>> >> > ConsoleProxy | NULL | >>>> >> > | 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter >>>> >> | >>>> >> > DomainRouter | NULL | >>>> >> > | 1 | s-1-VM | s-1-VM | Expunging | NULL | >>>> >> SecondaryStorageVm | >>>> >> > SecondaryStorageVm | NULL | >>>> >> > | 3 | s-3-VM | s-3-VM | Expunging | NULL | >>>> >> SecondaryStorageVm | >>>> >> > SecondaryStorageVm | NULL | >>>> >> > | 52 | s-52-VM | s-52-VM | Expunging | NULL | >>>> >> SecondaryStorageVm | >>>> >> > SecondaryStorageVm | NULL | >>>> >> > | 53 | s-53-VM | s-53-VM | Expunging | NULL | >>>> >> SecondaryStorageVm | >>>> >> > SecondaryStorageVm | NULL | >>>> >> > | 54 | s-54-VM | s-54-VM | Expunging | NULL | >>>> >> SecondaryStorageVm | >>>> >> > SecondaryStorageVm | NULL | >>>> >> > | 55 | s-55-VM | s-55-VM | Expunging | NULL | >>>> >> SecondaryStorageVm | >>>> >> > SecondaryStorageVm | NULL | >>>> >> > | 56 | s-56-VM | s-56-VM | Expunging | NULL | >>>> >> SecondaryStorageVm | >>>> >> > SecondaryStorageVm | NULL | >>>> >> > >>>> >> >>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+ >>>> >> > >>>> >> > >>>> >> > I have been looking through the DB to see if anything sticks out. I >>>> >> notice >>>> >> > that the mshost and mshost_peer table have 2 entries but they are >>>> the >>>> >> same >>>> >> > server. Is that ok? Looks like one is from the install of 4.1.0 >>>> an the >>>> >> > other from after I upgraded to 4.1.1: >>>> >> > >>>> >> > mysql> select * from mshost; >>>> >> > >>>> >> >>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+ >>>> >> > | id | msid | runid | name | state | >>>> version | >>>> >> > service_ip | service_port | last_update | removed | >>>> alert_count >>>> >> | >>>> >> > >>>> >> >>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+ >>>> >> > | 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0 >>>> | >>>> >> > 172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL | >>>> 0 >>>> >> | >>>> >> > | 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1 >>>> | >>>> >> > 172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL | >>>> 0 >>>> >> | >>>> >> > >>>> >> >>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+ >>>> >> > 2 rows in set (0.00 sec) >>>> >> > >>>> >> > mysql> select * from mshost_peer; >>>> >> > >>>> >> >>>> +----+--------------+-------------+---------------+------------+---------------------+ >>>> >> > | id | owner_mshost | peer_mshost | peer_runid | peer_state | >>>> >> > last_update | >>>> >> > >>>> >> >>>> +----+--------------+-------------+---------------+------------+---------------------+ >>>> >> > | 17 | 1 | 1 | 1375383759777 | Up | >>>> >> 2013-08-01 >>>> >> > 19:02:49 | >>>> >> > | 34 | 2 | 2 | 1383331508626 | Up | >>>> >> 2013-11-01 >>>> >> > 18:45:19 | >>>> >> > >>>> >> >>>> +----+--------------+-------------+---------------+------------+---------------------+ >>>> >> > >>>> >> > >>>> >> > thanks, >>>> >> > Carlos >>>> >> > >>>> >> > >>>> >> > >>>> >> > On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland < >>>> daan.hoogl...@gmail.com >>>> >> >wrote: >>>> >> > >>>> >> >> h Carlos, >>>> >> >> >>>> >> >> the states of virtual machines, especially the entries for the >>>> ssvm. >>>> >> >> if it is not Stopped. set it to stopped and check if your storage, >>>> >> >> both primary and secondary, are reachable botjh from the ms and the >>>> >> >> hosts. >>>> >> >> >>>> >> >> Daan (double a!) >>>> >> >> >>>> >> >> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui < >>>> create...@gmail.com> >>>> >> >> wrote: >>>> >> >> > Hi Dan, >>>> >> >> > The host value is correct and I am able to connect on ports >>>> 22,80 and >>>> >> >> 8080 >>>> >> >> > from the Xen hosts to that IP (same subnet). Port 8787 is not >>>> >> listening >>>> >> >> on >>>> >> >> > my MS. >>>> >> >> > >>>> >> >> > What can I look for in the DB? >>>> >> >> > >>>> >> >> > thanks >>>> >> >> > Carlos >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland < >>>> >> daan.hoogl...@gmail.com> >>>> >> >> > wrote: >>>> >> >> >> >>>> >> >> >> H Carlox, >>>> >> >> >> >>>> >> >> >> It still seems to me you have a communication problem albeit >>>> not at >>>> >> >> >> the network level. ( I agree this doesn't seem likely) But even >>>> so >>>> >> the >>>> >> >> >> ms can't reach the xen hosts. The next thing to look at would be >>>> >> >> >> configuration. I presume you did check the ip addresses for the >>>> >> hosts. >>>> >> >> >> check the value 'host' in the global vars and try to telnet it >>>> on >>>> >> port >>>> >> >> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the >>>> >> >> >> problem is hidden somewhere in the mysql database. >>>> >> >> >> >>>> >> >> >> regards, >>>> >> >> >> Daan >>>> >> >> >> >>>> >> >> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui < >>>> car...@reategui.com >>>> >> > >>>> >> >> >> wrote: >>>> >> >> >> > Hi Dan, >>>> >> >> >> > Finally getting around to trying this. I am not able to >>>> start any >>>> >> vm >>>> >> >> >> > (mine >>>> >> >> >> > or system). >>>> >> >> >> > >>>> >> >> >> > I am in the process of trying Ahmad's suggestion of removing >>>> one of >>>> >> >> the >>>> >> >> >> > hosts, reinstalling it and then adding it back in to see if >>>> that >>>> >> >> helps. >>>> >> >> >> > However I am not able to put it in maintenance mode. I am >>>> getting >>>> >> the >>>> >> >> >> > following when I try: >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet] >>>> >> >> >> > (catalina-exec-5:null) >>>> >> >> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >>>> >> >>>> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952 >>>> >> >> >> > 2013-10-31 15:19:59,181 DEBUG >>>> [cloud.async.AsyncJobManagerImpl] >>>> >> >> >> > (catalina-exec-5:null) submit async job-167, details: >>>> AsyncJobVO >>>> >> >> {id:16 >>>> >> >> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: >>>> Host, >>>> >> >> >> > instanceId: 1, cmd: >>>> >> org.apache.cloudstack.api.command.admin.host.Prep >>>> >> >> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo: >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >>>> >> >>>> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >>>> >> >>>> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"}, >>>> >> >> >> > cmdVersion: 0, callbackTy >>>> >> >> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0, >>>> >> resultCode: >>>> >> >> >> > 0, >>>> >> >> >> > result: null, initMsid: 233845174730253, completeMsid: null, >>>> >> >> >> > lastUpdated: null, lastPolled: null, created: null} >>>> >> >> >> > 2013-10-31 15:19:59,185 DEBUG >>>> [cloud.async.AsyncJobManagerImpl] >>>> >> >> >> > (Job-Executor-1:job-167) Executing >>>> >> >> org.apache.cloudstack.api.command.ad >>>> >> >> >> > min.host.PrepareForMaintenanceCmd for job-167 >>>> >> >> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet] >>>> >> >> >> > (catalina-exec-5:null) >>>> >> >> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >>>> >> >>>> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952 >>>> >> >> >> > 2013-10-31 15:19:59,207 DEBUG >>>> [cloud.cluster.ClusterManagerImpl] >>>> >> >> >> > (Job-Executor-1:job-167) Propagating agent change request >>>> >> event:AdminA >>>> >> >> >> > skMaintenace to agent:1 >>>> >> >> >> > 2013-10-31 15:19:59,208 DEBUG >>>> [cloud.cluster.ClusterManagerImpl] >>>> >> >> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 >>>> >> [{"Propa >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >>>> >> >>>> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}] >>>> >> >> >> > 2013-10-31 15:19:59,213 INFO >>>> >> >> [cloud.cluster.ClusterServiceServletImpl] >>>> >> >> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service >>>> >> >> >> > url: http://172.30.45.2:9090/clusterservice, request >>>> timeout: 300 >>>> >> >> >> > seconds >>>> >> >> >> > 2013-10-31 15:19:59,214 DEBUG >>>> [cloud.cluster.ClusterManagerImpl] >>>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> >>>> >> 233845174730255 >>>> >> >> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json: >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >>>> >> >>>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{ >>>> >> >> >> > },"wait":0}}] >>>> >> >> >> > 2013-10-31 15:19:59,343 DEBUG >>>> [cloud.cluster.ClusterManagerImpl] >>>> >> >> >> > (Cluster-Worker-7:null) Dispatch ->1, json: >>>> >> [{"PropagateResourceEventC >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >>>> >> >>>> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}] >>>> >> >> >> > 2013-10-31 15:19:59,347 DEBUG >>>> [cloud.cluster.ClusterManagerImpl] >>>> >> >> >> > (Cluster-Worker-7:null) Intercepting command to propagate >>>> event >>>> >> AdminA >>>> >> >> >> > skMaintenace for host 1 >>>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG >>>> >> >> [cloud.cluster.ClusterServiceServletImpl] >>>> >> >> >> > (Cluster-Worker-1:null) POST >>>> http://172.30.45.2:9090/clusterser >>>> >> >> >> > vice response :true, responding time: 80 ms >>>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG >>>> [cloud.cluster.ClusterManagerImpl] >>>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> >>>> >> 233845174730255 >>>> >> >> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, >>>> >> json: >>>> >> >> >> > >>>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai >>>> >> >> >> > ntenace","contextMap":{},"wait":0}}] >>>> >> >> >> > 2013-10-31 15:19:59,356 DEBUG >>>> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq >>>> 1-102760483 >>>> >> >> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: >>>> 100111, >>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255 >>>> >> >> >> > 2013-10-31 15:19:59,360 DEBUG >>>> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq >>>> 1-102 >>>> >> >> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> > 100111, >>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255 >>>> >> >> >> > 2013-10-31 15:19:59,365 DEBUG >>>> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding >>>> Seq 1-10 >>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> > 100111, >>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255 >>>> >> >> >> > 2013-10-31 15:19:59,369 DEBUG >>>> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding >>>> Seq 1-10 >>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> > 100111, >>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255 >>>> >> >> >> > 2013-10-31 15:19:59,372 DEBUG >>>> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding >>>> Seq 1-10 >>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> > 100111, >>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255 >>>> >> >> >> > 2013-10-31 15:19:59,376 DEBUG >>>> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding >>>> Seq 1-10 >>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> > 100111, >>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255 >>>> >> >> >> > >>>> >> >> >> > These MaintainCommand lines are repeated endlessly and very >>>> fast >>>> >> (1GB >>>> >> >> of >>>> >> >> >> > logs in about 20 mins) so I currently have the MS stopped. >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui < >>>> >> car...@reategui.com >>>> >> >> > >>>> >> >> >> > wrote: >>>> >> >> >> >> >>>> >> >> >> >> Hi Dan, >>>> >> >> >> >> Network is definitely not a problem. iptables is disabled >>>> on MS >>>> >> and >>>> >> >> >> >> both >>>> >> >> >> >> hosts. I am able to ssh from MS to both hosts. I am also >>>> able to >>>> >> >> >> >> connect >>>> >> >> >> >> to port 80 from the MS. >>>> >> >> >> >> >>>> >> >> >> >> I need to head out now and will try what you suggested in a >>>> couple >>>> >> >> >> >> hours. >>>> >> >> >> >> >>>> >> >> >> >> thanks, >>>> >> >> >> >> Carlos >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland >>>> >> >> >> >> <daan.hoogl...@gmail.com> >>>> >> >> >> >> wrote: >>>> >> >> >> >>> >>>> >> >> >> >>> Carlos, >>>> >> >> >> >>> >>>> >> >> >> >>> can the ms reach the xs hosts? >>>> >> >> >> >>> if so try this, Stop the ms, set the state for the >>>> vm_instances >>>> >> for >>>> >> >> >> >>> cpvm and vr from expunging to stopped. and restart. >>>> >> >> >> >>> if not you need to troubleshoot your network connectivity. >>>> It >>>> >> seems >>>> >> >> >> >>> this would not be the issue as you only stopped and started >>>> it. >>>> >> Did >>>> >> >> >> >>> you >>>> >> >> >> >>> >>>> >> >> >> >>> Daan >>>> >> >> >> >>> >>>> >> >> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui >>>> >> >> >> >>> <car...@reategui.com> >>>> >> >> >> >>> wrote: >>>> >> >> >> >>> > Hi Dan, >>>> >> >> >> >>> > >>>> >> >> >> >>> > Unfortunately not. Still trying to solve this. See my >>>> last >>>> >> email >>>> >> >> >> >>> > to >>>> >> >> >> >>> > Ahmad >>>> >> >> >> >>> > in the recover cs thread about next steps I am trying to >>>> figure >>>> >> >> out. >>>> >> >> >> >>> > >>>> >> >> >> >>> > The CPVM and VR are not running. They are currently in >>>> >> expunging >>>> >> >> >> >>> > state. >>>> >> >> >> >>> > >>>> >> >> >> >>> > If I try to start an instance the MS is unable to >>>> communicate >>>> >> the >>>> >> >> >> >>> > start >>>> >> >> >> >>> > vm >>>> >> >> >> >>> > commands to my XS hosts and then starts writing to the >>>> logs at >>>> >> an >>>> >> >> >> >>> > amazingly >>>> >> >> >> >>> > high rate. You can see one of my logs here: >>>> >> >> >> >>> > http://reategui.com/cloudstack/management-server.log.new >>>> >> >> >> >>> > >>>> >> >> >> >>> > thank you >>>> >> >> >> >>> > Carlos >>>> >> >> >> >>> > >>>> >> >> >> >>> > >>>> >> >> >> >>> > >>>> >> >> >> >>> > >>>> >> >> >> >>> > >>>> >> >> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland >>>> >> >> >> >>> > <daan.hoogl...@gmail.com> >>>> >> >> >> >>> > wrote: >>>> >> >> >> >>> >> >>>> >> >> >> >>> >> H Carlos, >>>> >> >> >> >>> >> >>>> >> >> >> >>> >> Did you solve this problem yet? >>>> >> >> >> >>> >> >>>> >> >> >> >>> >> You can try removing the cpvm and vr at the hypervisor >>>> and >>>> >> change >>>> >> >> >> >>> >> their state to 'Stopped' in the db. >>>> >> >> >> >>> >> >>>> >> >> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui >>>> >> >> >> >>> >> <car...@reategui.com> >>>> >> >> >> >>> >> wrote: >>>> >> >> >> >>> >> > In the last 15 minutes management log file it grew >>>> 500MB.. >>>> >> >> >> >>> >> > >>>> >> >> >> >>> >> > Any ideas what I should look at to figure out what is >>>> >> >> happening? >>>> >> >> >> >>> >> > >>>> >> >> >> >>> >> > >>>> >> >> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui >>>> >> >> >> >>> >> > <create...@gmail.com>wrote: >>>> >> >> >> >>> >> > >>>> >> >> >> >>> >> >> I am trying to recover my CloudStack installation (CS >>>> >> 4.1.1 on >>>> >> >> >> >>> >> >> ubuntu + >>>> >> >> >> >>> >> >> XS >>>> >> >> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life. >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am >>>> seeing >>>> >> in >>>> >> >> >> >>> >> >> the >>>> >> >> >> >>> >> >> logs >>>> >> >> >> >>> >> >> is >>>> >> >> >> >>> >> >> the following and they are filling up fast >>>> (management log >>>> >> is >>>> >> >> >> >>> >> >> almost >>>> >> >> >> >>> >> >> 1GB). >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 1-201919186: >>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: >>>> 2, >>>> >> Ver: >>>> >> >> v1, >>>> >> >> >> >>> >> >> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 1-201919186: >>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: >>>> 2, >>>> >> Ver: >>>> >> >> v1, >>>> >> >> >> >>> >> >> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 1-201919186: >>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: >>>> 2, >>>> >> Ver: >>>> >> >> v1, >>>> >> >> >> >>> >> >> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 1-201919186: >>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: >>>> Forwarding >>>> >> >> Seq >>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: >>>> 2, >>>> >> Ver: >>>> >> >> v1, >>>> >> >> >> >>> >> >> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 1-201919186: >>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, >>>> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG >>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache] >>>> >> >> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: >>>> Forwarding >>>> >> Seq >>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: >>>> 2, >>>> >> Ver: >>>> >> >> v1, >>>> >> >> >> >>> >> >> Flags: >>>> >> >> >> >>> >> >> 100111, >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >>>> >> >>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}] >>>> >> >> >> >>> >> >> } to 233845174730255 >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> >> >> >>>> >> >> >> >>> > >>>> >> >> >> >>> > >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> >>>> >> >>>> >>> >>> >>
