Since I didn't hear back I bit the bullet and decided to make the DB
changes and I am back up and running.  Woohoo.  I'll post the specifics in
the other thread I started about the mshost id.


On Tue, Nov 5, 2013 at 11:36 AM, Carlos Reategui <car...@reategui.com>wrote:

> Hi Daan,
>
> The mshost thing I mentioned previously has been bugging me and as I look
> through the db I think I have a problem here.  I am guessing I have a
> botched 4.1.0 to 4.1.1 upgrade that I did not notice when I shut things
> down.
>
> The mshost table is showing 2 management servers that are supposedly on
> the same machine listening on the same port which is not really possible.
>  So really only one of the 2 is up (I believe that would be the 4.1.1
> version). The logs do seem to indicate something along those lines where
> the 2 management servers are failing to communicate with each other.
>
> If I look at the host table I see my 2 hosts but they say they are on
> version 4.1.0 and are pointing to management server id of the 4.1.0 line in
> the mshost table.
>
> I am wondering if this is the problem and why their is no communication to
> my hosts from the management server.
>
> Anyone know the schema well enough to help me update some stuff and
> recover my installation?  Would an upgrade to 4.2 possibly cleanup my DB?
>
> thanks,
> Carlos
>
>
>
>
> On Tue, Nov 5, 2013 at 10:59 AM, Carlos Reategui <car...@reategui.com>wrote:
>
>> Hi Daan,
>>
>> I have been looking through my logs and can't figure out why the ssvm
>> does not want to start.
>>
>> Here is a link to my latest logs:
>> http://reategui.com/cloudstack/management-server-131105.log
>>
>> Maybe someone else can spot something.
>>
>> There is nothing on either the XenServer hosts logs.
>>
>> thanks,
>> Carlos
>>
>>
>> On Tue, Nov 5, 2013 at 12:55 AM, Daan Hoogland 
>> <daan.hoogl...@gmail.com>wrote:
>>
>>> Carlos,
>>>
>>> It makes no sense trying to start the vms if not the ssvm is started
>>> first.
>>>
>>> I would say stop the system, clear the logs and start again. Then
>>> analyse the logs to see what is wrong with the secondary storage vm.
>>>
>>> On Tue, Nov 5, 2013 at 2:08 AM, Carlos Reategui <car...@reategui.com>
>>> wrote:
>>> > Hi Daan,
>>> > There is no trace of CS communicating with either of my XS hosts in the
>>> > SMLog file or the xensource.log files when I try to start one of my
>>> stopped
>>> > instances.
>>> >
>>> > thanks,
>>> > Carlos
>>> >
>>> >
>>> >
>>> > On Sat, Nov 2, 2013 at 4:50 AM, Daan Hoogland <daan.hoogl...@gmail.com
>>> >wrote:
>>> >
>>> >> H Carlos,
>>> >>
>>> >> I wouldn't know about the mshost and mshost_peer but they seem
>>> >> alright. The expunging ssvm are a real problem. I think Ahmed
>>> >> mentioned that as well. You are having problems getting your ssvm in
>>> >> the air. All other things are consequential. This is the first thing
>>> >> to look at. Check the logs and the SMLog on the hypervisor to see what
>>> >> happens when cs tries to start a ssvm. By your db you can see that it
>>> >> tried to start it several times. Those are the expunging vm_instance
>>> >> entries.
>>> >>
>>> >> regards,
>>> >> Daan
>>> >>
>>> >> On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <car...@reategui.com>
>>> >> wrote:
>>> >> > Hi Daan  :)
>>> >> >
>>> >> > Primary and Secondary storage are on the MS.  Primary is on a
>>> different
>>> >> > subnet and it is currently mounted (and visible as an NFS SR from
>>> >> > XenCenter) and working fine from the xen machines.  I have also
>>> been able
>>> >> > to mount Secondary without a problem.
>>> >> >
>>> >> > Here are my current system VMs:
>>> >> > mysql> select
>>> >> > id,name,instance_name,state,host_id,type,vm_type,desired_state from
>>> >> > vm_instance where type!='User';
>>> >> >
>>> >>
>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>> >> > | id | name    | instance_name | state     | host_id | type
>>> >>   |
>>> >> > vm_type            | desired_state |
>>> >> >
>>> >>
>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>> >> > |  2 | v-2-VM  | v-2-VM        | Stopped   |       1 | ConsoleProxy
>>> >>   |
>>> >> > ConsoleProxy       | NULL          |
>>> >> > |  5 | r-5-VM  | r-5-VM        | Stopped   |       2 | DomainRouter
>>> >>   |
>>> >> > DomainRouter       | NULL          |
>>> >> > |  1 | s-1-VM  | s-1-VM        | Expunging |    NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL          |
>>> >> > |  3 | s-3-VM  | s-3-VM        | Expunging |    NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL          |
>>> >> > | 52 | s-52-VM | s-52-VM       | Expunging |    NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL          |
>>> >> > | 53 | s-53-VM | s-53-VM       | Expunging |    NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL          |
>>> >> > | 54 | s-54-VM | s-54-VM       | Expunging |    NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL          |
>>> >> > | 55 | s-55-VM | s-55-VM       | Expunging |    NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL          |
>>> >> > | 56 | s-56-VM | s-56-VM       | Expunging |    NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL          |
>>> >> >
>>> >>
>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>> >> >
>>> >> >
>>> >> > I have been looking through the DB to see if anything sticks out.  I
>>> >> notice
>>> >> > that the mshost and mshost_peer table have 2 entries but they are
>>> the
>>> >> same
>>> >> > server.  Is that ok?  Looks like one is from the install of 4.1.0
>>> an the
>>> >> > other from after I upgraded to 4.1.1:
>>> >> >
>>> >> > mysql> select * from mshost;
>>> >> >
>>> >>
>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>> >> > | id | msid            | runid         | name       | state |
>>> version |
>>> >> > service_ip  | service_port | last_update         | removed |
>>> alert_count
>>> >> |
>>> >> >
>>> >>
>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>> >> > |  1 | 233845174730255 | 1375383759777 | isbustor01 | Up    | 4.1.0
>>>   |
>>> >> > 172.30.45.2 |         9090 | 2013-10-24 21:13:10 | NULL    |
>>>     0
>>> >> |
>>> >> > |  2 | 233845174730253 | 1383331508626 | isbustor01 | Up    | 4.1.1
>>>   |
>>> >> > 172.30.45.2 |         9090 | 2013-11-01 20:00:04 | NULL    |
>>>     0
>>> >> |
>>> >> >
>>> >>
>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>> >> > 2 rows in set (0.00 sec)
>>> >> >
>>> >> > mysql> select * from mshost_peer;
>>> >> >
>>> >>
>>> +----+--------------+-------------+---------------+------------+---------------------+
>>> >> > | id | owner_mshost | peer_mshost | peer_runid    | peer_state |
>>> >> > last_update         |
>>> >> >
>>> >>
>>> +----+--------------+-------------+---------------+------------+---------------------+
>>> >> > | 17 |            1 |           1 | 1375383759777 | Up         |
>>> >> 2013-08-01
>>> >> > 19:02:49 |
>>> >> > | 34 |            2 |           2 | 1383331508626 | Up         |
>>> >> 2013-11-01
>>> >> > 18:45:19 |
>>> >> >
>>> >>
>>> +----+--------------+-------------+---------------+------------+---------------------+
>>> >> >
>>> >> >
>>> >> > thanks,
>>> >> > Carlos
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <
>>> daan.hoogl...@gmail.com
>>> >> >wrote:
>>> >> >
>>> >> >> h Carlos,
>>> >> >>
>>> >> >> the states of virtual machines, especially the entries for the
>>> ssvm.
>>> >> >> if it is not Stopped. set it to stopped and check if your storage,
>>> >> >> both primary and secondary, are reachable botjh from the ms and the
>>> >> >> hosts.
>>> >> >>
>>> >> >> Daan (double a!)
>>> >> >>
>>> >> >> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <
>>> create...@gmail.com>
>>> >> >> wrote:
>>> >> >> > Hi Dan,
>>> >> >> > The host value is correct and I am able to connect on ports
>>> 22,80 and
>>> >> >> 8080
>>> >> >> > from the Xen hosts to that IP (same subnet).  Port 8787 is not
>>> >> listening
>>> >> >> on
>>> >> >> > my MS.
>>> >> >> >
>>> >> >> > What can I look for in the DB?
>>> >> >> >
>>> >> >> > thanks
>>> >> >> > Carlos
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <
>>> >> daan.hoogl...@gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> H Carlox,
>>> >> >> >>
>>> >> >> >> It still seems to me you have a communication problem albeit
>>> not at
>>> >> >> >> the network level. ( I agree this doesn't seem likely) But even
>>> so
>>> >> the
>>> >> >> >> ms can't reach the xen hosts. The next thing to look at would be
>>> >> >> >> configuration. I presume you did check the ip addresses for the
>>> >> hosts.
>>> >> >> >> check the value 'host' in the global vars and try to telnet it
>>> on
>>> >> port
>>> >> >> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
>>> >> >> >> problem is hidden somewhere in the mysql database.
>>> >> >> >>
>>> >> >> >> regards,
>>> >> >> >> Daan
>>> >> >> >>
>>> >> >> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <
>>> car...@reategui.com
>>> >> >
>>> >> >> >> wrote:
>>> >> >> >> > Hi Dan,
>>> >> >> >> > Finally getting around to trying this.  I am not able to
>>> start any
>>> >> vm
>>> >> >> >> > (mine
>>> >> >> >> > or system).
>>> >> >> >> >
>>> >> >> >> > I am in the process of trying Ahmad's suggestion of removing
>>> one of
>>> >> >> the
>>> >> >> >> > hosts, reinstalling it and then adding it back in to see if
>>> that
>>> >> >> helps.
>>> >> >> >> > However I am not able to put it in maintenance mode.  I am
>>> getting
>>> >> the
>>> >> >> >> > following when I try:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
>>> >> >> >> > (catalina-exec-5:null)
>>> >> >> >> > ===START===  172.30.40.135 -- GET  command=prepareHostForMa
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>>> >> >> >> > 2013-10-31 15:19:59,181 DEBUG
>>> [cloud.async.AsyncJobManagerImpl]
>>> >> >> >> > (catalina-exec-5:null) submit async job-167, details:
>>> AsyncJobVO
>>> >> >> {id:16
>>> >> >> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType:
>>> Host,
>>> >> >> >> > instanceId: 1, cmd:
>>> >> org.apache.cloudstack.api.command.admin.host.Prep
>>> >> >> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
>>> >> >> >> > cmdVersion: 0, callbackTy
>>> >> >> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0,
>>> >> resultCode:
>>> >> >> >> > 0,
>>> >> >> >> > result: null, initMsid: 233845174730253, completeMsid: null,
>>> >> >> >> > lastUpdated: null, lastPolled: null, created: null}
>>> >> >> >> > 2013-10-31 15:19:59,185 DEBUG
>>> [cloud.async.AsyncJobManagerImpl]
>>> >> >> >> > (Job-Executor-1:job-167) Executing
>>> >> >> org.apache.cloudstack.api.command.ad
>>> >> >> >> > min.host.PrepareForMaintenanceCmd for job-167
>>> >> >> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
>>> >> >> >> > (catalina-exec-5:null)
>>> >> >> >> > ===END===  172.30.40.135 -- GET  command=prepareHostForMain
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>>> >> >> >> > 2013-10-31 15:19:59,207 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Job-Executor-1:job-167) Propagating agent change request
>>> >> event:AdminA
>>> >> >> >> > skMaintenace to agent:1
>>> >> >> >> > 2013-10-31 15:19:59,208 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1
>>> >> [{"Propa
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>>> >> >> >> > 2013-10-31 15:19:59,213 INFO
>>> >> >>  [cloud.cluster.ClusterServiceServletImpl]
>>> >> >> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
>>> >> >> >> > url: http://172.30.45.2:9090/clusterservice, request
>>> timeout: 300
>>> >> >> >> > seconds
>>> >> >> >> > 2013-10-31 15:19:59,214 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>>> >> 233845174730255
>>> >> >> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
>>> >> >> >> > },"wait":0}}]
>>> >> >> >> > 2013-10-31 15:19:59,343 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Cluster-Worker-7:null) Dispatch ->1, json:
>>> >> [{"PropagateResourceEventC
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>>> >> >> >> > 2013-10-31 15:19:59,347 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Cluster-Worker-7:null) Intercepting command to propagate
>>> event
>>> >> AdminA
>>> >> >> >> > skMaintenace for host 1
>>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
>>> >> >> [cloud.cluster.ClusterServiceServletImpl]
>>> >> >> >> > (Cluster-Worker-1:null) POST
>>> http://172.30.45.2:9090/clusterser
>>> >> >> >> > vice response :true, responding time: 80 ms
>>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>>> >> 233845174730255
>>> >> >> >> >  completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0,
>>> >> json:
>>> >> >> >> >
>>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
>>> >> >> >> > ntenace","contextMap":{},"wait":0}}]
>>> >> >> >> > 2013-10-31 15:19:59,356 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq
>>> 1-102760483
>>> >> >> >> > :  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>>> 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,360 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq
>>> 1-102
>>> >> >> >> > 760483:  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,365 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding
>>> Seq 1-10
>>> >> >> >> > 2760483:  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,369 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding
>>> Seq 1-10
>>> >> >> >> > 2760483:  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,372 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding
>>> Seq 1-10
>>> >> >> >> > 2760483:  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,376 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding
>>> Seq 1-10
>>> >> >> >> > 2760483:  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> >
>>> >> >> >> > These MaintainCommand lines are repeated endlessly and very
>>> fast
>>> >> (1GB
>>> >> >> of
>>> >> >> >> > logs in about 20 mins) so I currently have the MS stopped.
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <
>>> >> car...@reategui.com
>>> >> >> >
>>> >> >> >> > wrote:
>>> >> >> >> >>
>>> >> >> >> >> Hi Dan,
>>> >> >> >> >> Network is definitely not a problem.  iptables is disabled
>>> on MS
>>> >> and
>>> >> >> >> >> both
>>> >> >> >> >> hosts.  I am able to ssh from MS to both hosts.  I am also
>>> able to
>>> >> >> >> >> connect
>>> >> >> >> >> to port 80 from the MS.
>>> >> >> >> >>
>>> >> >> >> >> I need to head out now and will try what you suggested in a
>>> couple
>>> >> >> >> >> hours.
>>> >> >> >> >>
>>> >> >> >> >> thanks,
>>> >> >> >> >> Carlos
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
>>> >> >> >> >> <daan.hoogl...@gmail.com>
>>> >> >> >> >> wrote:
>>> >> >> >> >>>
>>> >> >> >> >>> Carlos,
>>> >> >> >> >>>
>>> >> >> >> >>> can the ms reach the xs hosts?
>>> >> >> >> >>> if so try this, Stop the ms, set the state for the
>>> vm_instances
>>> >> for
>>> >> >> >> >>> cpvm and vr from expunging to stopped. and restart.
>>> >> >> >> >>> if not you need to troubleshoot your network connectivity.
>>> It
>>> >> seems
>>> >> >> >> >>> this would not be the issue as you only stopped and started
>>> it.
>>> >> Did
>>> >> >> >> >>> you
>>> >> >> >> >>>
>>> >> >> >> >>> Daan
>>> >> >> >> >>>
>>> >> >> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
>>> >> >> >> >>> <car...@reategui.com>
>>> >> >> >> >>> wrote:
>>> >> >> >> >>> > Hi Dan,
>>> >> >> >> >>> >
>>> >> >> >> >>> > Unfortunately not.  Still trying to solve this.  See my
>>> last
>>> >> email
>>> >> >> >> >>> > to
>>> >> >> >> >>> > Ahmad
>>> >> >> >> >>> > in the recover cs thread about next steps I am trying to
>>> figure
>>> >> >> out.
>>> >> >> >> >>> >
>>> >> >> >> >>> > The CPVM and VR are not running.  They are currently in
>>> >> expunging
>>> >> >> >> >>> > state.
>>> >> >> >> >>> >
>>> >> >> >> >>> > If I try to start an instance the MS is unable to
>>> communicate
>>> >> the
>>> >> >> >> >>> > start
>>> >> >> >> >>> > vm
>>> >> >> >> >>> > commands to my XS hosts and then starts writing to the
>>> logs at
>>> >> an
>>> >> >> >> >>> > amazingly
>>> >> >> >> >>> > high rate.  You can see one of my logs here:
>>> >> >> >> >>> > http://reategui.com/cloudstack/management-server.log.new
>>> >> >> >> >>> >
>>> >> >> >> >>> > thank you
>>> >> >> >> >>> > Carlos
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
>>> >> >> >> >>> > <daan.hoogl...@gmail.com>
>>> >> >> >> >>> > wrote:
>>> >> >> >> >>> >>
>>> >> >> >> >>> >> H Carlos,
>>> >> >> >> >>> >>
>>> >> >> >> >>> >> Did you solve this problem yet?
>>> >> >> >> >>> >>
>>> >> >> >> >>> >> You can try removing the cpvm and vr at the hypervisor
>>> and
>>> >> change
>>> >> >> >> >>> >> their state to 'Stopped' in the db.
>>> >> >> >> >>> >>
>>> >> >> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
>>> >> >> >> >>> >> <car...@reategui.com>
>>> >> >> >> >>> >> wrote:
>>> >> >> >> >>> >> > In the last 15 minutes management log file it grew
>>> 500MB..
>>> >> >> >> >>> >> >
>>> >> >> >> >>> >> > Any ideas what I should look at to figure out what is
>>> >> >> happening?
>>> >> >> >> >>> >> >
>>> >> >> >> >>> >> >
>>> >> >> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>>> >> >> >> >>> >> > <create...@gmail.com>wrote:
>>> >> >> >> >>> >> >
>>> >> >> >> >>> >> >> I am trying to recover my CloudStack installation (CS
>>> >> 4.1.1 on
>>> >> >> >> >>> >> >> ubuntu +
>>> >> >> >> >>> >> >> XS
>>> >> >> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am
>>> seeing
>>> >> in
>>> >> >> >> >>> >> >> the
>>> >> >> >> >>> >> >> logs
>>> >> >> >> >>> >> >> is
>>> >> >> >> >>> >> >> the following and they are filling up fast
>>> (management log
>>> >> is
>>> >> >> >> >>> >> >> almost
>>> >> >> >> >>> >> >> 1GB).
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >>  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613:  { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >>  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613:  { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >>  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613:  { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >>  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613:
>>> Forwarding
>>> >> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613:  { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >>  { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613:  { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >>
>>>
>>
>>
>

Reply via email to