Hi Dan,
Finally getting around to trying this. I am not able to start any vm (mine
or system).
I am in the process of trying Ahmad's suggestion of removing one of the
hosts, reinstalling it and then adding it back in to see if that helps.
However I am not able to put it in maintenance mode. I am getting the
following when I try:
2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet] (catalina-exec-5:null)
===START=== 172.30.40.135 -- GET command=prepareHostForMa
intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-5:null) submit async job-167, details: AsyncJobVO {id:16
7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
instanceId: 1, cmd: org.apache.cloudstack.api.command.admin.host.Prep
areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
{"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
cmdVersion: 0, callbackTy
pe: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0,
result: null, initMsid: 233845174730253, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-1:job-167) Executing org.apache.cloudstack.api.command.ad
min.host.PrepareForMaintenanceCmd for job-167
2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet] (catalina-exec-5:null)
===END=== 172.30.40.135 -- GET command=prepareHostForMain
tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
(Job-Executor-1:job-167) Propagating agent change request event:AdminA
skMaintenace to agent:1
2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
(Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 [{"Propa
gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
2013-10-31 15:19:59,213 INFO [cloud.cluster.ClusterServiceServletImpl]
(Cluster-Worker-1:null) Setup cluster service servlet. service
url: http://172.30.45.2:9090/clusterservice, request timeout: 300 seconds
2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
(Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
[{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
},"wait":0}}]
2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
(Cluster-Worker-7:null) Dispatch ->1, json: [{"PropagateResourceEventC
ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
(Cluster-Worker-7:null) Intercepting command to propagate event AdminA
skMaintenace for host 1
2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterServiceServletImpl]
(Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
vice response :true, responding time: 80 ms
2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
(Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
[{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
ntenace","contextMap":{},"wait":0}}]
2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
(Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
These MaintainCommand lines are repeated endlessly and very fast (1GB of
logs in about 20 mins) so I currently have the MS stopped.
On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <car...@reategui.com>wrote:
> Hi Dan,
> Network is definitely not a problem. iptables is disabled on MS and both
> hosts. I am able to ssh from MS to both hosts. I am also able to connect
> to port 80 from the MS.
>
> I need to head out now and will try what you suggested in a couple hours.
>
> thanks,
> Carlos
>
>
> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland <daan.hoogl...@gmail.com>wrote:
>
>> Carlos,
>>
>> can the ms reach the xs hosts?
>> if so try this, Stop the ms, set the state for the vm_instances for
>> cpvm and vr from expunging to stopped. and restart.
>> if not you need to troubleshoot your network connectivity. It seems
>> this would not be the issue as you only stopped and started it. Did
>> you
>>
>> Daan
>>
>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui <car...@reategui.com>
>> wrote:
>> > Hi Dan,
>> >
>> > Unfortunately not. Still trying to solve this. See my last email to
>> Ahmad
>> > in the recover cs thread about next steps I am trying to figure out.
>> >
>> > The CPVM and VR are not running. They are currently in expunging state.
>> >
>> > If I try to start an instance the MS is unable to communicate the start
>> vm
>> > commands to my XS hosts and then starts writing to the logs at an
>> amazingly
>> > high rate. You can see one of my logs here:
>> > http://reategui.com/cloudstack/management-server.log.new
>> >
>> > thank you
>> > Carlos
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland <daan.hoogl...@gmail.com
>> >
>> > wrote:
>> >>
>> >> H Carlos,
>> >>
>> >> Did you solve this problem yet?
>> >>
>> >> You can try removing the cpvm and vr at the hypervisor and change
>> >> their state to 'Stopped' in the db.
>> >>
>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui <car...@reategui.com>
>> >> wrote:
>> >> > In the last 15 minutes management log file it grew 500MB..
>> >> >
>> >> > Any ideas what I should look at to figure out what is happening?
>> >> >
>> >> >
>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>> >> > <create...@gmail.com>wrote:
>> >> >
>> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
>> ubuntu +
>> >> >> XS
>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>> >> >>
>> >> >> I tried to destroy the CPVM and the VR and all I am seeing in the
>> logs
>> >> >> is
>> >> >> the following and they are filling up fast (management log is almost
>> >> >> 1GB).
>> >> >>
>> >> >>
>> >> >> 2013-10-29 11:51:08,291 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,299 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >
>> >
>>
>
>
