+1 on what Paul said - set that to false, restart mgmt and try network
restart again.
On Tue, Mar 19, 2019, 16:48 Boris Stoyanov <boris.stoya...@shapeblue.com>
wrote:
> Hi, you shouldn’t be worried about data as long as you use a separate
> shared storage.
>
> What does the cloudstack console say about your host? Is it up? If it’s up
> then you should be able to deploy a VM on it.
>
> If not, you’ll need to investigate the reason cloudstack-agent is not
> connected, is the agent running? If not just start it and see if it
> connects.
>
> If it’s running but still not connected please share the agent.log.
>
> If nothing helps at all you can remove the host and add it again, but
> let’s try to troubleshoot first.
>
> Bobby.
>
>
> boris.stoya...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
> > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> wrote:
> >
> > Guys, please help with it.
> > What can be done here?
> > There is too much valuable data.
> >
> > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> > wrote:
> >
> >> Tried that just now and got error:
> >> Resource [DataCenter:1] is unreachable: Can't find all necessary running
> >> routers!
> >>
> >> In the log I see:
> >> =============================================
> >>
> >> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> >> Restarting network 204...
> >> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> >> Performing rolling restart of routers of network Ntwk[204|Guest|6]
> >> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> >> Asking VirtualRouter to implemenet Ntwk[204|Guest|6]
> >> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Unexpected
> >> exception while executing
> >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> >> com.cloud.exception.ResourceUnavailableException: Resource
> [DataCenter:1]
> >> is unreachable: Can't find all necessary running routers!
> >> at
> >>
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> >> at
> >>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> >> at
> >>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> >> at
> >>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> >> at
> >>
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> >> at java.lang.reflect.Method.invoke(Unknown Source)
> >> at
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >> at
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> at
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> at
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >> at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >> at
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >> at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> >> at
> >>
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >> at
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> >> Source)
> >> at java.util.concurrent.FutureTask.run(Unknown Source)
> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> >> Source)
> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >> Source)
> >> at java.lang.Thread.run(Unknown Source)
> >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete
> async
> >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> >>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> >> [DataCenter:1] is un
> >>
> >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> andrija.pa...@shapeblue.com>
> >> wrote:
> >>
> >>>
> >>> Your network can't be deleted due to "Can't delete the network, not all
> >>> user vms are expunged. Vm
> >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
> >>>
> >>> You should be able to just start the user VM - but if you have actually
> >>> delete the VR itself, then just do Network restart with "cleanup" and
> it
> >>> will recreate a new VR, after which you should be able to start the VM.
> >>>
> >>> Andrija
> >>>
> >>> andrija.pa...@shapeblue.com
> >>> www.shapeblue.com
> >>> Amadeus House, Floral Street, London WC2E 9DPUK
> >>> @shapeblue
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> >>> Sent: 19 March 2019 15:10
> >>> To: users@cloudstack.apache.org
> >>> Subject: Re: Disaster after maintenance
> >>>
> >>> I mean I cannot delete network: In the management server log I see
> >>>
> >>> ==========================================================
> >>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
> >>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
> >>> instanceId: 204, cmd:
> >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> cmdInfo:
> >>>
> >>>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> >>>
> >>>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> >>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >>> result: null, initMsid: 264216221068220, completeMsid: null,
> lastUpdated:
> >>> null, lastPolled: null, created: null}
> >>> 2019-03-19 14:06:36,351 WARN [o.a.c.e.o.NetworkOrchestrator]
> >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa)
> (logid:c734edfc)
> >>> Can't delete the network, not all user vms are expunged. Vm
> >>> VM[User|i-2-11-VM] is in Stopped state
> >>> 2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete
> async
> >>> job-5090, jobStatus: FAILED, resultCode: 530, result:
> >>>
> >>>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> >>> to delete network"}
> >>> ==========================================================
> >>>
> >>>
> >>> I deleted a router, expecting it to be recreated on deleting network.
> But
> >>> I am unable to delete network because of above error
> >>>
> >>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov <
> >>> j.zolotar...@gmail.com>
> >>> wrote:
> >>>
> >>>> I've managed to make libvirtd running
> >>>> Now cloudstack console shows both hosts - running
> >>>>
> >>>> But now as I have removed network, VMs are unable to start.
> >>>>
> >>>> How can I recreate the network now?
> >>>>
> >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
> >>>> <kudryavtsev...@bw-sw.com>
> >>>> wrote:
> >>>>
> >>>>> Jevgeniy, it may be a documentation bug. Take s look:
> >>>>> https://github.com/apache/cloudstack-documentation/pull/27/files
> >>>>>
> >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >:
> >>>>>
> >>>>>> That's it - libvirtd failed to start on second host.
> >>>>>> Tried restarting, but it does not start.
> >>>>>>
> >>>>>>
> >>>>>>>> Do you have some NUMA constraints or anything which requires
> >>>>> particular
> >>>>>> RAM configuration?
> >>>>>> No
> >>>>>>
> >>>>>> libvirtd.service - Virtualization daemon
> >>>>>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service;
> >>>>>> enabled; vendor preset: enabled)
> >>>>>> Active: failed (Result: start-limit) since Tue 2019-03-19
> >>>>>> 13:03:07
> >>>>> GMT;
> >>>>>> 12s ago
> >>>>>> Docs: man:libvirtd(8)
> >>>>>> https://libvirt.org
> >>>>>> Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
> >>>>>> (code=exited,
> >>>>>> status=1/FAILURE)
> >>>>>> Main PID: 892 (code=exited, status=1/FAILURE)
> >>>>>> Tasks: 19 (limit: 32768)
> >>>>>> CGroup: /system.slice/libvirtd.service
> >>>>>> ├─11338 /usr/sbin/libvirtd -d -l
> >>>>>> ├─11909 /usr/sbin/dnsmasq
> >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >>>>>> └─11910 /usr/sbin/dnsmasq
> >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >>>>>>
> >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to
> >>>>>> start Virtualization daemon.
> >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
> >>>>>> libvirtd.service entered failed state.
> >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> >>>>> libvirtd.service
> >>>>>> failed.
> >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> >>>>> libvirtd.service
> >>>>>> holdoff time over, scheduling restart.
> >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped
> >>>>>> Virtualization daemon.
> >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start
> >>>>>> request repeated too quickly for libvirtd.service Mar 19 13:03:07
> >>>>>> mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
> >>>>>> Virtualization daemon.
> >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
> >>>>>> libvirtd.service entered failed state.
> >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> >>>>> libvirtd.service
> >>>>>> failed.
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Mar 19, 2019 at 3:04 PM Paul Angus
> >>>>>> <paul.an...@shapeblue.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Can you check that the cloudstack agent is running on the host
> >>>>>>> and the agent logs (usual logs directory) Also worth checking
> >>>>>>> that libvirt has started ok. Do you have some
> >>>>> NUMA
> >>>>>>> constraints or anything which requires particular RAM
> >>> configuration?
> >>>>>>>
> >>>>>>> paul.an...@shapeblue.com
> >>>>>>> www.shapeblue.com
> >>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> >>>>>>> Sent: 19 March 2019 14:49
> >>>>>>> To: users@cloudstack.apache.org
> >>>>>>> Subject: Re: Disaster after maintenance
> >>>>>>>
> >>>>>>> Can you try migrating a VM to the server that you changed the RAM
> >>>>> amount?
> >>>>>>>
> >>>>>>> Also:
> >>>>>>> What is the hypervisor version?
> >>>>>>> KVM
> >>>>>>> QEMU Version : 2.0.0
> >>>>>>> Release : 1.el7.6
> >>>>>>>
> >>>>>>>
> >>>>>>> Host status in ACS?
> >>>>>>> 1st server: Unsecure
> >>>>>>> 2nd server: Disconnected
> >>>>>>>
> >>>>>>> Did you try to force a VM to start/deploy in this server where you
> >>>>>> changed
> >>>>>>> the RAM?
> >>>>>>> Host status became disconnected. I don't know how to make it
> >>>>> "connected"
> >>>>>>> again
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
> >>>>>>> rafaelweingart...@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Can you try migrating a VM to the server that you changed the RAM
> >>>>>> amount?
> >>>>>>>>
> >>>>>>>> Also:
> >>>>>>>> What is the hypervisor version?
> >>>>>>>> Host status in ACS?
> >>>>>>>> Did you try to force a VM to start/deploy in this server where
> >>> you
> >>>>>>>> changed the RAM?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov
> >>>>>>>> <j.zolotar...@gmail.com
> >>>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> We have Cloudstack 4.11.2 setup running fine for few months
> >>> (>4)
> >>>>> The
> >>>>>>>>> setup is very simple: 2 hosts We decided to do a maintenance to
> >>>>>>>>> increase RAM on both servers
> >>>>>>>>>
> >>>>>>>>> For this we put first server to maintenance. All VMS moved to
> >>>>> second
> >>>>>>>>> host after a while.
> >>>>>>>>>
> >>>>>>>>> Then first server was shutdown, RAM increased, server turned
> >>> ON.
> >>>>>>>>> Now nothing starts on first server.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Tried to delete network, but this fails as well
> >>>>>>>>>
> >>>>>>>>> Please help !
> >>>>>>>>>
> >>>>>>>>> Here is extract from log:
> >>>>>>>>> ======================================
> >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG
> >>>>> [o.a.c.s.SecondaryStorageManagerImpl]
> >>>>>>>>> (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to
> >>>>>>>>> launch secondary storage VM
> >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>>>>>>>> (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready
> >>> to
> >>>>>>>>> launch console proxy
> >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===
> >>>>>>>> 192.168.5.140
> >>>>>>>>> -- GET
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> >>>>>>>> =json&_=1552998473154
> >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
> >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be)
> >>>>> CIDRs
> >>>>>>>>> from which account
> >>>>>>>>> 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> >>>>>>>> allowed
> >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> >>>>>>>>> 2019-03-19 12:27:53,208 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add
> >>>>>>>>> job-5081 into job monitoring
> >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be)
> >>>>> submit
> >>>>>>>> async
> >>>>>>>>> job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId:
> >>> 2,
> >>>>>>>>> instanceType: Network, instanceId: 204, cmd:
> >>>>>>>>>
> >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> >>>>>>> cmdInfo:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> >>>>>>>>
> >>>>>
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> >>>>>>>> interface
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> >>>>>>>>
> >>>>>
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> >>>>>>>> EventType":"NETWORK.DELETE","_":"1552998473154"},
> >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> >>> resultCode:
> >>>>> 0,
> >>>>>>>>> result: null, initMsid: 264216221068220, completeMsid: null,
> >>>>>>> lastUpdated:
> >>>>>>>>> null, lastPolled: null, created: null}
> >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
> >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be)
> >>>>>>>>> ===END===
> >>>>>>>>> 192.168.5.140 -- GET
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> >>>>>>>> =json&_=1552998473154
> >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> >>>>>>>>> Executing AsyncJobVO {id:5081, userId: 2, accountId: 2,
> >>>>>>>>> instanceType: Network,
> >>>>>>>>> instanceId: 204, cmd:
> >>>>>>>>>
> >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> >>>>>>> cmdInfo:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> >>>>>>>>
> >>>>>
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> >>>>>>>> interface
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> >>>>>>>>
> >>>>>
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> >>>>>>>> EventType":"NETWORK.DELETE","_":"1552998473154"},
> >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> >>> resultCode:
> >>>>> 0,
> >>>>>>>>> result: null, initMsid: 264216221068220, completeMsid: null,
> >>>>>>> lastUpdated:
> >>>>>>>>> null, lastPolled: null, created: null}
> >>>>>>>>> 2019-03-19 12:27:53,257 WARN [o.a.c.e.o.NetworkOrchestrator]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979)
> >>>>>>>>> (logid:16897ea6) Can't delete the network, not all user vms are
> >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] is in Stopped state
> >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> >>>>> Complete
> >>>>>>>> async
> >>>>>>>>> job-5081, jobStatus: FAILED, resultCode: 530, result:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> >>>>>>>> [],"errorcode":530,"errortext":"Failed
> >>>>>>>>> to delete network"}
> >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> >>>>> Publish
> >>>>>>>>> async
> >>>>>>>>> job-5081 complete on message bus
> >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> >>> Wake
> >>>>> up
> >>>>>>>>> jobs related to job-5081
> >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> >>> Update
> >>>>>>>>> db status for job-5081
> >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> >>> Wake
> >>>>> up
> >>>>>>>>> jobs joined with job-5081 and disjoin all subjobs created from
> >>>>> job-
> >>>>>>>>> 5081
> >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> >>> Done
> >>>>>>>> executing
> >>>>>>>>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd
> >>>>> for
> >>>>>>>>> job-5081
> >>>>>>>>> 2019-03-19 12:27:53,267 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> >>> Remove
> >>>>>>>> job-5081
> >>>>>>>>> from job monitoring
> >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG [c.c.a.ApiServlet]
> >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85) (logid:7fe68f75) ===START===
> >>>>>>>> 192.168.5.140
> >>>>>>>>> -- GET
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> >>>>>>>> &response=json&_=1552998476202
> >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG [c.c.a.ApiServer]
> >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) (logid:7fe68f75)
> >>> CIDRs
> >>>>>>>>> from which account
> >>>>>>>>> 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> >>>>>>>> allowed
> >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG [c.c.a.ApiServlet]
> >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) (logid:7fe68f75)
> >>>>>>>>> ===END===
> >>>>>>>>> 192.168.5.140 -- GET
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> >>>>>>>> &response=json&_=1552998476202
> >>>>>>>>> 2019-03-19 12:28:00,946 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) Begin
> >>>>>>>>> cleanup expired async-jobs
> >>>>>>>>> 2019-03-19 12:28:00,951 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) End
> >>>>> cleanup
> >>>>>>>> expired
> >>>>>>>>> async-jobs
> >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG
> >>>>>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >>>>>>>>> (RouterStatusMonitor-1:ctx-ad6bbe7e) (logid:04e4c72b) Found 0
> >>>>>>>>> routers to update status.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Rafael Weingärtner
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
