Can you check that the cloudstack agent is running on the host and the agent
logs (usual logs directory)
Also worth checking that libvirt has started ok. Do you have some NUMA
constraints or anything which requires particular RAM configuration?
paul.an...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
-----Original Message-----
From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
Sent: 19 March 2019 14:49
To: users@cloudstack.apache.org
Subject: Re: Disaster after maintenance
Can you try migrating a VM to the server that you changed the RAM amount?
Also:
What is the hypervisor version?
KVM
QEMU Version : 2.0.0
Release : 1.el7.6
Host status in ACS?
1st server: Unsecure
2nd server: Disconnected
Did you try to force a VM to start/deploy in this server where you changed the
RAM?
Host status became disconnected. I don't know how to make it "connected"
again
On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:
> Can you try migrating a VM to the server that you changed the RAM amount?
>
> Also:
> What is the hypervisor version?
> Host status in ACS?
> Did you try to force a VM to start/deploy in this server where you
> changed the RAM?
>
>
> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov
> <j.zolotar...@gmail.com
> >
> wrote:
>
> > We have Cloudstack 4.11.2 setup running fine for few months (>4) The
> > setup is very simple: 2 hosts We decided to do a maintenance to
> > increase RAM on both servers
> >
> > For this we put first server to maintenance. All VMS moved to second
> > host after a while.
> >
> > Then first server was shutdown, RAM increased, server turned ON.
> > Now nothing starts on first server.
> >
> >
> > Tried to delete network, but this fails as well
> >
> > Please help !
> >
> > Here is extract from log:
> > ======================================
> > 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to
> > launch secondary storage VM
> > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to
> > launch console proxy
> > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===
> 192.168.5.140
> > -- GET
> >
> >
> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> =json&_=1552998473154
> > 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) CIDRs
> > from which account
> > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> allowed
> > to perform API calls: 0.0.0.0/0,::/0
> > 2019-03-19 12:27:53,208 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add
> > job-5081 into job monitoring
> > 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) submit
> async
> > job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId: 2,
> > instanceType: Network, instanceId: 204, cmd:
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> interface
> >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> EventType":"NETWORK.DELETE","_":"1552998473154"},
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: null}
> > 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be)
> > ===END===
> > 192.168.5.140 -- GET
> >
> >
> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> =json&_=1552998473154
> > 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> > Executing AsyncJobVO {id:5081, userId: 2, accountId: 2,
> > instanceType: Network,
> > instanceId: 204, cmd:
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> interface
> >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> EventType":"NETWORK.DELETE","_":"1552998473154"},
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: null}
> > 2019-03-19 12:27:53,257 WARN [o.a.c.e.o.NetworkOrchestrator]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979)
> > (logid:16897ea6) Can't delete the network, not all user vms are
> > expunged. Vm VM[User|i-2-11-VM] is in Stopped state
> > 2019-03-19 12:27:53,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Complete
> async
> > job-5081, jobStatus: FAILED, resultCode: 530, result:
> >
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> [],"errorcode":530,"errortext":"Failed
> > to delete network"}
> > 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Publish
> > async
> > job-5081 complete on message bus
> > 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Wake up
> > jobs related to job-5081
> > 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Update
> > db status for job-5081
> > 2019-03-19 12:27:53,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Wake up
> > jobs joined with job-5081 and disjoin all subjobs created from job-
> > 5081
> > 2019-03-19 12:27:53,267 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Done
> executing
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for
> > job-5081
> > 2019-03-19 12:27:53,267 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Remove
> job-5081
> > from job monitoring
> > 2019-03-19 12:27:56,230 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-28:ctx-e6c5bc85) (logid:7fe68f75) ===START===
> 192.168.5.140
> > -- GET
> >
> >
> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> &response=json&_=1552998476202
> > 2019-03-19 12:27:56,238 DEBUG [c.c.a.ApiServer]
> > (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) (logid:7fe68f75) CIDRs
> > from which account
> > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> allowed
> > to perform API calls: 0.0.0.0/0,::/0
> > 2019-03-19 12:27:56,260 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) (logid:7fe68f75)
> > ===END===
> > 192.168.5.140 -- GET
> >
> >
> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> &response=json&_=1552998476202
> > 2019-03-19 12:28:00,946 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) Begin
> > cleanup expired async-jobs
> > 2019-03-19 12:28:00,951 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) End cleanup
> expired
> > async-jobs
> > 2019-03-19 12:28:01,142 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-ad6bbe7e) (logid:04e4c72b) Found 0
> > routers to update status.
> >
>
>
> --
> Rafael Weingärtner
>
