select * from networks where removed is null;
select * from vm_instance where id=87;
select id,name from vm_instance where name like 'r%' and removed is null;

Basically since the network offering is not redundant this error is only thrown 
when there is no router associated with your network. Usually management server 
restart tries to implement network again. Please restart management server, 
save and share management server log.




On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote:

    >>>>Check network_offering table for  value in column
    redundant_router_service  for the network offering you use.
    in table network_offering_table all records have redundant_router_service =
    0
    
    Can you also run the following:
    >>>select name, state, removed  from host where name like 'r%'
    returns zero rows - nothing
    
    >>>select * from domain_router;
    # id, element_id, public_mac_address, public_ip_address, public_netmask,
    guest_netmask, guest_ip_address, is_redundant_router, priority,
    redundant_state, stop_pending, role, template_version, scripts_version,
    vpc_id, update_state
    '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
    '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL
    '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
    'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
    '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
    'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
    '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
    '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
    'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
    '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
    'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
    '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
    'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED'
    '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
    'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
    '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
    'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
    '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
    '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
    'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 15:06:49 UTC
    2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL
    
    
    >>>select * from router_network_ref;
    # id, router_id, network_id, guest_type
    '1', '4', '204', 'Shared'
    '2', '49', '204', 'Shared'
    '3', '73', '204', 'Shared'
    '4', '75', '204', 'Shared'
    '5', '76', '204', 'Shared'
    '6', '77', '204', 'Shared'
    '7', '80', '204', 'Shared'
    '8', '85', '204', 'Shared'
    '9', '86', '204', 'Shared'
    '10', '87', '204', 'Shared'
    
    
    On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy <serg...@hotmail.com>
    wrote:
    
    > Check network_offering table for  value in column
    > redundant_router_service  for the network offering you use.
    > Can you also run the following:
    > select name, state, removed  from host where name like 'r%'
    > select * from domain_router;
    > select * from router_network_ref;
    >
    > Cloudstack is supposed to recreate you VR. If it is not happening there is
    > something fundamentally wrong. I would advise to destroy your VR again.
    > Stop you management server. Rotate management server log and start it
    > again. If your VR doesn't start in few min, post your complete  management
    > server log  and agent log again.
    >
    >
    >
    >
    > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote:
    >
    >     >>>Network offering needs to be change to non-redundant
    >     How do I do that?
    >
    >     On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy <serg...@hotmail.com
    > >
    >     wrote:
    >
    >     > Network offering needs to be change to non-redundant. Most likely
    > old bug
    >     > is resurfaced.
    >     > https://issues.apache.org/jira/browse/CLOUDSTACK-9024
    >     >
    >     >
    >     >
    >     > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com>
    > wrote:
    >     >
    >     >     Hello
    >     >
    >     >     I did exactly like you suggested.
    >     >
    >     >     After UPDATE on db, I can see the router in cloudstack console.
    >     >     But attempt to restart network fails.
    >     >     I get error:
    >     >     Resource [DataCenter:1] is unreachable: Can't find all necessary
    >     > running
    >     >     routers!
    >     >
    >     >     I rechecked agents on both servers. They look running OK.
    >     >
    >     >     On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic <
    >     > andrija.pa...@gmail.com>
    >     >     wrote:
    >     >
    >     >     > Ok, so:
    >     >     >
    >     >     > 1.BACKUP YOUR DB - in case there is issues, you will want to
    > restore
    >     >     > 2. I tried to reproduce your case on 4.11.2 (though didn't do
    > any
    >     >     > maintenance etc.) - and could not - I have artificially marked
    >     > existing VR
    >     >     > as removed in DB - and tried to restart network and it worked
    > just
    >     > fine.
    >     >     >
    >     >     > Let's try to "undelete" the VR for that network - though I
    > can't be
    >     > sure if
    >     >     > this will work or not.
    >     >     >
    >     >     > Find VR based on the network UUID:
    >     >     >       SELECT * FROM vm_instance WHERE id IN (SELECT
    > instance_id from
    >     > nics
    >     >     > where network_id IN (SELECT id FROM networks WHERE
    >     >     > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
    >     >     > Obviously, replace this UUID with your network UUID.
    >     >     > SQL above will return all VMs (user VMs, VRs, etc, delete or
    > not...)
    >     > that
    >     >     > have NIC inside the network with that UUID - look for the
    > r-XX-VM
    >     > being a
    >     >     > router.
    >     >     >
    >     >     > Now that you have the name of the VR (i.e. r-4-VM) - you want
    > to set
    >     > 2
    >     >     > fields to some other value than what it might be already, as
    > below:
    >     >     >
    >     >     > Need to set field "state" to Stopped:
    >     >     >       UPDATE cloud.vm_instance SET state='Stopped' WHERE
    >     > name="r-4-VM";
    >     >     >  (what state is set currently ???)
    >     >     >
    >     >     > Need to set "removed" to NULL value:
    >     >     >       UPDATE cloud.vm_instance SET removed = NULL WHERE
    >     > NAME="r-4-VM"; (is
    >     >     > there a removed date already set ???)
    >     >     >
    >     >     > Obviously make sure that this VR is not running on any
    > hypervisor
    >     > and if
    >     >     > running, virsh destroy it...
    >     >     >
    >     >     > AFTER above has been done (VR is considered to exist but
    > stopped,
    >     > from
    >     >     > CloudStack point of view) - try to RESTART the network (don't
    > bother
    >     >     > deleteing it, since there are VMs in that network)
    >     >     >
    >     >     > I'm not 100% positive this will fix your issue, but doesn't
    > hurt to
    >     > try
    >     >     >
    >     >     > If above doesn't work - I would still take a look into agents
    > and if
    >     > they
    >     >     > are still connected - optionally, restart agents on both hosts
    > once
    >     > more
    >     >     > and confirm they are connected and up.
    >     >     >
    >     >     > Let us know how it goes (and please backup DB once more before
    > any
    >     > actions
    >     >     > !)
    >     >     >
    >     >     > Andrija
    >     >     >
    >     >     >
    >     >     > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov <
    >     > j.zolotar...@gmail.com>
    >     >     > wrote:
    >     >     >
    >     >     > > Yes.  Just a single network.
    >     >     > >
    >     >     > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, <
    > andrija.pa...@gmail.com
    >     > >
    >     >     > wrote:
    >     >     > >
    >     >     > > > Just one more clarification - this is Isolate single
    > network (not
    >     >     > Shared
    >     >     > > > Network, not VPC) ?
    >     >     > > >
    >     >     > > >
    >     >     > > >
    >     >     > > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov <
    >     > j.zolotar...@gmail.com
    >     >     > >
    >     >     > > > wrote:
    >     >     > > >
    >     >     > > > > Name defaultGuestNetwork
    >     >     > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
    >     >     > > > > Zone PBT zone 1
    >     >     > > > > Description defaultGuestNetwork
    >     >     > > > > Type Shared
    >     >     > > > > State Setup
    >     >     > > > > VPC ID N/A
    >     >     > > > > Persistent No
    >     >     > > > >
    >     >     > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic <
    >     >     > andrija.pa...@gmail.com
    >     >     > > >
    >     >     > > > > wrote:
    >     >     > > > >
    >     >     > > > > > Share the network id and name as seen from GUI...
    >     >     > > > > >
    >     >     > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov <
    >     >     > > j.zolotar...@gmail.com
    >     >     > > > >
    >     >     > > > > > wrote:
    >     >     > > > > >
    >     >     > > > > > > >>> 1. Confirm please rolling restart is set to 
false
    >     > please -
    >     >     > > double
    >     >     > > > > > check
    >     >     > > > > > > Double checked - It is set to false
    >     >     > > > > > >
    >     >     > > > > > > >>>>2. If so - do you know the name of VR which you
    >     > deleted ? Is
    >     >     > it
    >     >     > > > > last
    >     >     > > > > > > one
    >     >     > > > > > > >>> ever created - if so we can find it easily...
    >     >     > > > > > > I dont know the name
    >     >     > > > > > > Is there a way to fetch it from DB?
    >     >     > > > > > >
    >     >     > > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <
    >     >     > > > andrija.pa...@gmail.com
    >     >     > > > > >
    >     >     > > > > > > wrote:
    >     >     > > > > > >
    >     >     > > > > > > > Right...still complaining on missing running
    > routers.
    >     >     > > > > > > >
    >     >     > > > > > > > 1. Confirm please rolling restart is set to false
    > please
    >     > -
    >     >     > double
    >     >     > > > > check
    >     >     > > > > > > > 2. If so - do you know the name of VR which you
    > deleted
    >     > ? Is it
    >     >     > > > last
    >     >     > > > > > one
    >     >     > > > > > > > ever created - if so we can find it easily...
    >     >     > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <
    >     >     > > > > j.zolotar...@gmail.com
    >     >     > > > > > >
    >     >     > > > > > > > wrote:
    >     >     > > > > > > >
    >     >     > > > > > > > > Here is management server log
    >     >     > > > > > > > >
    >     >     > > >
    >     > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
    >     >     > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <
    >     >     > > > > > andrija.pa...@gmail.com
    >     >     > > > > > > >
    >     >     > > > > > > > > wrote:
    >     >     > > > > > > > >
    >     >     > > > > > > > > > I would also try to "undelete" VR in DB, but
    > let's
    >     > keep
    >     >     > this
    >     >     > > as
    >     >     > > > > > last
    >     >     > > > > > > > > step.
    >     >     > > > > > > > > >
    >     >     > > > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic <
    >     >     > > > > andrija.pa...@gmail.com
    >     >     > > > > > >
    >     >     > > > > > > > > wrote:
    >     >     > > > > > > > > >
    >     >     > > > > > > > > > > Disclaimer: from mobile device.
    >     >     > > > > > > > > > >
    >     >     > > > > > > > > > > I don't see any special failure in agent 
log,
    >     > except some
    >     >     > > > long
    >     >     > > > > > > > running
    >     >     > > > > > > > > > > migration, timeout for graceful VM shutdown
    > etc
    >     > (and
    >     >     > agent
    >     >     > > > > > restart)
    >     >     > > > > > > > > > >
    >     >     > > > > > > > > > > You did not send mgmt log after last network
    >     > restart
    >     >     > > failure
    >     >     > > > ?
    >     >     > > > > > > > > > >
    >     >     > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni
    > Zolotarjov <
    >     >     > > > > > > > j.zolotar...@gmail.com
    >     >     > > > > > > > > >
    >     >     > > > > > > > > > > wrote:
    >     >     > > > > > > > > > >
    >     >     > > > > > > > > > >> Under
    >     >     > > > > > > > > > >> Infrastructure / Hosts:
    >     >     > > > > > > > > > >> both hosts are enabled and green (Unsecure
    > though)
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > > >> agent.log -
    >     >     > > > > > > > > > >>
    >     >     > > > > > >
    >     >     >
    > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > > >> I set network.rolling.restart to false now.
    >     > Restarted
    >     >     > > > > management
    >     >     > > > > > > > > server
    >     >     > > > > > > > > > -
    >     >     > > > > > > > > > >> same problem - cannot restart not delete
    > network
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris
    > Stoyanov <
    >     >     > > > > > > > > > >> boris.stoya...@shapeblue.com>
    >     >     > > > > > > > > > >> wrote:
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > > >> > Hi, you shouldn’t be worried about data
    > as long
    >     > as you
    >     >     > > > use a
    >     >     > > > > > > > > separate
    >     >     > > > > > > > > > >> > shared storage.
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> > What does the cloudstack console say
    > about your
    >     > host?
    >     >     > Is
    >     >     > > > it
    >     >     > > > > > up?
    >     >     > > > > > > If
    >     >     > > > > > > > > > it’s
    >     >     > > > > > > > > > >> up
    >     >     > > > > > > > > > >> > then you should be able to deploy a VM on
    > it.
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> > If not, you’ll need to investigate the
    > reason
    >     >     > > > > cloudstack-agent
    >     >     > > > > > > is
    >     >     > > > > > > > > not
    >     >     > > > > > > > > > >> > connected, is the agent running? If not
    > just
    >     > start it
    >     >     > > and
    >     >     > > > > see
    >     >     > > > > > if
    >     >     > > > > > > > it
    >     >     > > > > > > > > > >> > connects.
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> > If it’s running but still not connected
    > please
    >     > share
    >     >     > the
    >     >     > > > > > > > agent.log.
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> > If nothing helps at all you can remove
    > the host
    >     > and
    >     >     > add
    >     >     > > it
    >     >     > > > > > > again,
    >     >     > > > > > > > > but
    >     >     > > > > > > > > > >> > let’s try to troubleshoot first.
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> > Bobby.
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> > boris.stoya...@shapeblue.com
    >     >     > > > > > > > > > >> > www.shapeblue.com
    >     >     > > > > > > > > > >> > Amadeus House, Floral Street, London
    > WC2E 9DPUK
    >     >     > > > > > > > > > >> > @shapeblue
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni
    > Zolotarjov <
    >     >     > > > > > > > > > j.zolotar...@gmail.com>
    >     >     > > > > > > > > > >> > wrote:
    >     >     > > > > > > > > > >> > >
    >     >     > > > > > > > > > >> > > Guys, please help with it.
    >     >     > > > > > > > > > >> > > What can be done here?
    >     >     > > > > > > > > > >> > > There is too much valuable data.
    >     >     > > > > > > > > > >> > >
    >     >     > > > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni
    >     > Zolotarjov <
    >     >     > > > > > > > > > >> > j.zolotar...@gmail.com>
    >     >     > > > > > > > > > >> > > wrote:
    >     >     > > > > > > > > > >> > >
    >     >     > > > > > > > > > >> > >> Tried that just now and got error:
    >     >     > > > > > > > > > >> > >> Resource [DataCenter:1] is 
unreachable:
    >     > Can't find
    >     >     > > all
    >     >     > > > > > > > necessary
    >     >     > > > > > > > > > >> running
    >     >     > > > > > > > > > >> > >> routers!
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> > >> In the log I see:
    >     >     > > > > > > > > > >> > >>
    > =============================================
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> > >> 2019-03-19 14:20:39,644 DEBUG
    >     >     > > > > > [o.a.c.e.o.NetworkOrchestrator]
    >     >     > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb
    > job-5093
    >     >     > > ctx-9be30648)
    >     >     > > > > > > > > > >> (logid:265a6099)
    >     >     > > > > > > > > > >> > >> Restarting network 204...
    >     >     > > > > > > > > > >> > >> 2019-03-19 14:20:39,645 DEBUG
    >     >     > > > > > [o.a.c.e.o.NetworkOrchestrator]
    >     >     > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb
    > job-5093
    >     >     > > ctx-9be30648)
    >     >     > > > > > > > > > >> (logid:265a6099)
    >     >     > > > > > > > > > >> > >> Performing rolling restart of routers
    > of
    >     > network
    >     >     > > > > > > > > Ntwk[204|Guest|6]
    >     >     > > > > > > > > > >> > >> 2019-03-19 14:20:39,649 DEBUG
    >     >     > > > > > [o.a.c.e.o.NetworkOrchestrator]
    >     >     > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb
    > job-5093
    >     >     > > ctx-9be30648)
    >     >     > > > > > > > > > >> (logid:265a6099)
    >     >     > > > > > > > > > >> > >> Asking VirtualRouter to implemenet
    >     >     > Ntwk[204|Guest|6]
    >     >     > > > > > > > > > >> > >> 2019-03-19 14:20:39,658 ERROR
    >     >     > > > > [c.c.a.ApiAsyncJobDispatcher]
    >     >     > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb
    > job-5093)
    >     >     > > > > (logid:265a6099)
    >     >     > > > > > > > > > >> Unexpected
    >     >     > > > > > > > > > >> > >> exception while executing
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > >
    >     >     >
    > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
    >     >     > > > > > > > > > >> > >>
    >     > com.cloud.exception.ResourceUnavailableException:
    >     >     > > > > Resource
    >     >     > > > > > > > > > >> > [DataCenter:1]
    >     >     > > > > > > > > > >> > >> is unreachable: Can't find all
    > necessary
    >     > running
    >     >     > > > routers!
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
    >     >     > > > > > > > > > Method)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
    >     >     > > > > > > > > > Source)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > >
    > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
    >     >     > > > > > > > > > >> > Source)
    >     >     > > > > > > > > > >> > >>        at
    >     > java.lang.reflect.Method.invoke(Unknown
    >     >     > > > Source)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown
    >     >     > > > > > > > Source)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >>
    >     >     > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > >
    >     > java.util.concurrent.Executors$RunnableAdapter.call(Unknown
    >     >     > > > > > > > > > >> > >> Source)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > java.util.concurrent.FutureTask.run(Unknown
    >     >     > > > > > Source)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > >
    >     > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
    >     >     > > > > > > > > > >> > >> Source)
    >     >     > > > > > > > > > >> > >>        at
    >     >     > > > > > > > > >
    >     > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
    >     >     > > > > > > > > > >> > >> Source)
    >     >     > > > > > > > > > >> > >>        at java.lang.Thread.run(Unknown
    >     > Source)
    >     >     > > > > > > > > > >> > >> 2019-03-19 14:20:39,659 DEBUG
    >     >     > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb
    > job-5093)
    >     >     > > > > (logid:265a6099)
    >     >     > > > > > > > > > Complete
    >     >     > > > > > > > > > >> > async
    >     >     > > > > > > > > > >> > >> job-5093, jobStatus: FAILED,
    > resultCode: 530,
    >     >     > result:
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
    >     >     > > > > > > > > > >> > >> [DataCenter:1] is un
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM 
Andrija
    >     > Panic <
    >     >     > > > > > > > > > >> > andrija.pa...@shapeblue.com>
    >     >     > > > > > > > > > >> > >> wrote:
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> > >>> ​​
    >     >     > > > > > > > > > >> > >>> Your network can't be deleted due to
    > "Can't
    >     > delete
    >     >     > > the
    >     >     > > > > > > > network,
    >     >     > > > > > > > > > not
    >     >     > > > > > > > > > >> all
    >     >     > > > > > > > > > >> > >>> user vms are expunged. Vm
    >     >     > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped
    > state" -
    >     > which is
    >     >     > > > fine.
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>> You should be able to just start the
    > user
    >     > VM - but
    >     >     > > if
    >     >     > > > > you
    >     >     > > > > > > have
    >     >     > > > > > > > > > >> actually
    >     >     > > > > > > > > > >> > >>> delete the VR itself, then just do
    > Network
    >     > restart
    >     >     > > > with
    >     >     > > > > > > > > "cleanup"
    >     >     > > > > > > > > > >> and
    >     >     > > > > > > > > > >> > it
    >     >     > > > > > > > > > >> > >>> will recreate a new VR, after which
    > you
    >     > should be
    >     >     > > able
    >     >     > > > > to
    >     >     > > > > > > > start
    >     >     > > > > > > > > > the
    >     >     > > > > > > > > > >> VM.
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>> Andrija
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>> andrija.pa...@shapeblue.com
    >     >     > > > > > > > > > >> > >>> www.shapeblue.com
    >     >     > > > > > > > > > >> > >>> Amadeus House, Floral Street, London
    > WC2E
    >     > 9DPUK
    >     >     > > > > > > > > > >> > >>> @shapeblue
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>> -----Original Message-----
    >     >     > > > > > > > > > >> > >>> From: Jevgeni Zolotarjov <
    >     > j.zolotar...@gmail.com>
    >     >     > > > > > > > > > >> > >>> Sent: 19 March 2019 15:10
    >     >     > > > > > > > > > >> > >>> To: users@cloudstack.apache.org
    >     >     > > > > > > > > > >> > >>> Subject: Re: Disaster after
    > maintenance
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>> I mean I cannot delete network: In 
the
    >     > management
    >     >     > > > server
    >     >     > > > > > > log I
    >     >     > > > > > > > > see
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > >
    > ==========================================================
    >     >     > > > > > > > > > >> > >>> 019-03-19 14:06:36,316 DEBUG
    >     >     > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc
    > job-5090)
    >     >     > > > > > (logid:c734edfc)
    >     >     > > > > > > > > > >> Executing
    >     >     > > > > > > > > > >> > >>> AsyncJobVO {id:5090, userId: 2,
    > accountId:
    >     > 2,
    >     >     > > > > > instanceType:
    >     >     > > > > > > > > > Network,
    >     >     > > > > > > > > > >> > >>> instanceId: 204, cmd:
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > >
    >     >     >
    > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
    >     >     > > > > > > > > > >> > cmdInfo:
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
    >     >     > > > > > > > > > >> > >>> cmdVersion: 0, status: IN_PROGRESS,
    >     > processStatus:
    >     >     > > 0,
    >     >     > > > > > > > > resultCode:
    >     >     > > > > > > > > > 0,
    >     >     > > > > > > > > > >> > >>> result: null, initMsid:
    > 264216221068220,
    >     >     > > completeMsid:
    >     >     > > > > > null,
    >     >     > > > > > > > > > >> > lastUpdated:
    >     >     > > > > > > > > > >> > >>> null, lastPolled: null, created: 
null}
    >     >     > > > > > > > > > >> > >>> 2019-03-19 14:06:36,351 WARN
    >     >     > > > > > > [o.a.c.e.o.NetworkOrchestrator]
    >     >     > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc
    > job-5090
    >     >     > > > ctx-134954fa)
    >     >     > > > > > > > > > >> > (logid:c734edfc)
    >     >     > > > > > > > > > >> > >>> Can't delete the network, not all
    > user vms
    >     > are
    >     >     > > > expunged.
    >     >     > > > > > Vm
    >     >     > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped 
state
    >     >     > > > > > > > > > >> > >>> 2019-03-19 14:06:36,356 DEBUG
    >     >     > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc
    > job-5090)
    >     >     > > > > > (logid:c734edfc)
    >     >     > > > > > > > > > Complete
    >     >     > > > > > > > > > >> > async
    >     >     > > > > > > > > > >> > >>> job-5090, jobStatus: FAILED,
    > resultCode:
    >     > 530,
    >     >     > > result:
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >
    > 
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
    >     >     > > > > > > > > > >> > >>> to delete network"}
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > >
    > ==========================================================
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>> I deleted a router, expecting it to 
be
    >     > recreated
    >     >     > on
    >     >     > > > > > deleting
    >     >     > > > > > > > > > >> network.
    >     >     > > > > > > > > > >> > But
    >     >     > > > > > > > > > >> > >>> I am unable to delete network because
    > of
    >     > above
    >     >     > error
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>> On Tue, Mar 19, 2019 at 3:58 PM
    > Jevgeni
    >     >     > Zolotarjov <
    >     >     > > > > > > > > > >> > >>> j.zolotar...@gmail.com>
    >     >     > > > > > > > > > >> > >>> wrote:
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>>> I've managed to make libvirtd 
running
    >     >     > > > > > > > > > >> > >>>> Now cloudstack console shows both
    > hosts -
    >     > running
    >     >     > > > > > > > > > >> > >>>>
    >     >     > > > > > > > > > >> > >>>> But now as I have removed network,
    > VMs are
    >     > unable
    >     >     > > to
    >     >     > > > > > start.
    >     >     > > > > > > > > > >> > >>>>
    >     >     > > > > > > > > > >> > >>>> How can I recreate the network now?
    >     >     > > > > > > > > > >> > >>>>
    >     >     > > > > > > > > > >> > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan
    >     > Kudryavtsev
    >     >     > > > > > > > > > >> > >>>> <kudryavtsev...@bw-sw.com>
    >     >     > > > > > > > > > >> > >>>> wrote:
    >     >     > > > > > > > > > >> > >>>>
    >     >     > > > > > > > > > >> > >>>>> Jevgeniy, it may be a documentation
    > bug.
    >     > Take s
    >     >     > > > look:
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > >
    >     >     > > >
    > https://github.com/apache/cloudstack-documentation/pull/27/files
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni
    >     > Zolotarjov <
    >     >     > > > > > > > > > >> j.zolotar...@gmail.com
    >     >     > > > > > > > > > >> > >:
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> > >>>>>> That's it - libvirtd failed to
    > start on
    >     > second
    >     >     > > > host.
    >     >     > > > > > > > > > >> > >>>>>> Tried restarting, but it does not
    > start.
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>> Do you have some NUMA
    > constraints or
    >     > anything
    >     >     > > > which
    >     >     > > > > > > > > requires
    >     >     > > > > > > > > > >> > >>>>> particular
    >     >     > > > > > > > > > >> > >>>>>> RAM configuration?
    >     >     > > > > > > > > > >> > >>>>>> No
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > > > > > > > > > >> > >>>>>> libvirtd.service - Virtualization
    > daemon
    >     >     > > > > > > > > > >> > >>>>>>   Loaded: loaded
    >     >     > > > > > > (/usr/lib/systemd/system/libvirtd.service;
    >     >     > > > > > > > > > >> > >>>>>> enabled; vendor preset: enabled)
    >     >     > > > > > > > > > >> > >>>>>>   Active: failed (Result:
    > start-limit)
    >     > since
    >     >     > Tue
    >     >     > > > > > > 2019-03-19
    >     >     > > > > > > > > > >> > >>>>>> 13:03:07
    >     >     > > > > > > > > > >> > >>>>> GMT;
    >     >     > > > > > > > > > >> > >>>>>> 12s ago
    >     >     > > > > > > > > > >> > >>>>>>     Docs: man:libvirtd(8)
    >     >     > > > > > > > > > >> > >>>>>>           https://libvirt.org
    >     >     > > > > > > > > > >> > >>>>>>  Process: 892
    >     > ExecStart=/usr/sbin/libvirtd
    >     >     > > > > > $LIBVIRTD_ARGS
    >     >     > > > > > > > > > >> > >>>>>> (code=exited,
    >     >     > > > > > > > > > >> > >>>>>> status=1/FAILURE)
    >     >     > > > > > > > > > >> > >>>>>> Main PID: 892 (code=exited,
    >     > status=1/FAILURE)
    >     >     > > > > > > > > > >> > >>>>>>    Tasks: 19 (limit: 32768)
    >     >     > > > > > > > > > >> > >>>>>>   CGroup:
    > /system.slice/libvirtd.service
    >     >     > > > > > > > > > >> > >>>>>>           ├─11338
    > /usr/sbin/libvirtd -d
    >     > -l
    >     >     > > > > > > > > > >> > >>>>>>           ├─11909 
/usr/sbin/dnsmasq
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > --conf-file=/var/lib/libvirt/dnsmasq/default.conf
    >     >     > > > > > > > > > --leasefile-ro
    >     >     > > > > > > > > > >> > >>>>>>
    >     > --dhcp-script=/usr/libexec/libvirt_leaseshelper
    >     >     > > > > > > > > > >> > >>>>>>           └─11910 
/usr/sbin/dnsmasq
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > --conf-file=/var/lib/libvirt/dnsmasq/default.conf
    >     >     > > > > > > > > > --leasefile-ro
    >     >     > > > > > > > > > >> > >>>>>>
    >     > --dhcp-script=/usr/libexec/libvirt_leaseshelper
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
    >     > mtl1-apphst04.mt.pbt.com.mt
    >     >     > > > > > systemd[1]:
    >     >     > > > > > > > > Failed
    >     >     > > > > > > > > > >> to
    >     >     > > > > > > > > > >> > >>>>>> start Virtualization daemon.
    >     >     > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
    >     > mtl1-apphst04.mt.pbt.com.mt
    >     >     > > > > > systemd[1]:
    >     >     > > > > > > > Unit
    >     >     > > > > > > > > > >> > >>>>>> libvirtd.service entered failed
    > state.
    >     >     > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
    >     > mtl1-apphst04.mt.pbt.com.mt
    >     >     > > > > > systemd[1]:
    >     >     > > > > > > > > > >> > >>>>> libvirtd.service
    >     >     > > > > > > > > > >> > >>>>>> failed.
    >     >     > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
    >     > mtl1-apphst04.mt.pbt.com.mt
    >     >     > > > > > systemd[1]:
    >     >     > > > > > > > > > >> > >>>>> libvirtd.service
    >     >     > > > > > > > > > >> > >>>>>> holdoff time over, scheduling
    > restart.
    >     >     > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
    >     > mtl1-apphst04.mt.pbt.com.mt
    >     >     > > > > > systemd[1]:
    >     >     > > > > > > > > > Stopped
    >     >     > > > > > > > > > >> > >>>>>> Virtualization daemon.
    >     >     > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
    >     > mtl1-apphst04.mt.pbt.com.mt
    >     >     > > > > > systemd[1]:
    >     >     > > > > > > > > start
    >     >     > > > > > > > > > >> > >>>>>> request repeated too quickly for
    >     >     > libvirtd.service
    >     >     > > > Mar
    >     >     > > > > > 19
    >     >     > > > > > > > > > 13:03:07
    >     >     > > > > > > > > > >> > >>>>>> mtl1-apphst04.mt.pbt.com.mt
    > systemd[1]:
    >     > Failed
    >     >     > > to
    >     >     > > > > > start
    >     >     > > > > > > > > > >> > >>>>>> Virtualization daemon.
    >     >     > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
    >     > mtl1-apphst04.mt.pbt.com.mt
    >     >     > > > > > systemd[1]:
    >     >     > > > > > > > Unit
    >     >     > > > > > > > > > >> > >>>>>> libvirtd.service entered failed
    > state.
    >     >     > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
    >     > mtl1-apphst04.mt.pbt.com.mt
    >     >     > > > > > systemd[1]:
    >     >     > > > > > > > > > >> > >>>>> libvirtd.service
    >     >     > > > > > > > > > >> > >>>>>> failed.
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > > > > > > > > > >> > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM
    > Paul
    >     > Angus
    >     >     > > > > > > > > > >> > >>>>>> <paul.an...@shapeblue.com>
    >     >     > > > > > > > > > >> > >>>>>> wrote:
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > > > > > > > > > >> > >>>>>>> Can you check that the cloudstack
    > agent
    >     > is
    >     >     > > running
    >     >     > > > > on
    >     >     > > > > > > the
    >     >     > > > > > > > > host
    >     >     > > > > > > > > > >> > >>>>>>> and the agent logs (usual logs
    >     > directory) Also
    >     >     > > > worth
    >     >     > > > > > > > > checking
    >     >     > > > > > > > > > >> > >>>>>>> that libvirt has started ok.  Do
    > you
    >     > have some
    >     >     > > > > > > > > > >> > >>>>> NUMA
    >     >     > > > > > > > > > >> > >>>>>>> constraints or anything which
    > requires
    >     >     > > particular
    >     >     > > > > RAM
    >     >     > > > > > > > > > >> > >>> configuration?
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>> paul.an...@shapeblue.com
    >     >     > > > > > > > > > >> > >>>>>>> www.shapeblue.com
    >     >     > > > > > > > > > >> > >>>>>>> Amadeus House, Floral Street,
    > London
    >     > WC2E
    >     >     > 9DPUK
    >     >     > > > > > > > @shapeblue
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>> -----Original Message-----
    >     >     > > > > > > > > > >> > >>>>>>> From: Jevgeni Zolotarjov <
    >     >     > > j.zolotar...@gmail.com>
    >     >     > > > > > > > > > >> > >>>>>>> Sent: 19 March 2019 14:49
    >     >     > > > > > > > > > >> > >>>>>>> To: users@cloudstack.apache.org
    >     >     > > > > > > > > > >> > >>>>>>> Subject: Re: Disaster after
    > maintenance
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>> Can you try migrating a VM to the
    >     > server that
    >     >     > > you
    >     >     > > > > > > changed
    >     >     > > > > > > > > the
    >     >     > > > > > > > > > >> RAM
    >     >     > > > > > > > > > >> > >>>>> amount?
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>> Also:
    >     >     > > > > > > > > > >> > >>>>>>> What is the hypervisor version?
    >     >     > > > > > > > > > >> > >>>>>>> KVM
    >     >     > > > > > > > > > >> > >>>>>>> QEMU Version     : 2.0.0
    >     >     > > > > > > > > > >> > >>>>>>> Release     : 1.el7.6
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>> Host status in ACS?
    >     >     > > > > > > > > > >> > >>>>>>> 1st server: Unsecure
    >     >     > > > > > > > > > >> > >>>>>>> 2nd server: Disconnected
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>> Did you try to force a VM to
    >     > start/deploy in
    >     >     > > this
    >     >     > > > > > server
    >     >     > > > > > > > > where
    >     >     > > > > > > > > > >> you
    >     >     > > > > > > > > > >> > >>>>>> changed
    >     >     > > > > > > > > > >> > >>>>>>> the RAM?
    >     >     > > > > > > > > > >> > >>>>>>> Host status became disconnected. 
I
    >     > don't know
    >     >     > > how
    >     >     > > > to
    >     >     > > > > > > make
    >     >     > > > > > > > it
    >     >     > > > > > > > > > >> > >>>>> "connected"
    >     >     > > > > > > > > > >> > >>>>>>> again
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM
    > Rafael
    >     >     > > > Weingärtner <
    >     >     > > > > > > > > > >> > >>>>>>> rafaelweingart...@gmail.com>
    > wrote:
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>> Can you try migrating a VM to 
the
    >     > server that
    >     >     > > you
    >     >     > > > > > > changed
    >     >     > > > > > > > > the
    >     >     > > > > > > > > > >> RAM
    >     >     > > > > > > > > > >> > >>>>>> amount?
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>> Also:
    >     >     > > > > > > > > > >> > >>>>>>>> What is the hypervisor version?
    >     >     > > > > > > > > > >> > >>>>>>>> Host status in ACS?
    >     >     > > > > > > > > > >> > >>>>>>>> Did you try to force a VM to
    >     > start/deploy in
    >     >     > > this
    >     >     > > > > > > server
    >     >     > > > > > > > > > where
    >     >     > > > > > > > > > >> > >>> you
    >     >     > > > > > > > > > >> > >>>>>>>> changed the RAM?
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM
    > Jevgeni
    >     >     > > > Zolotarjov
    >     >     > > > > > > > > > >> > >>>>>>>> <j.zolotar...@gmail.com
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>> wrote:
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>> We have Cloudstack 4.11.2 setup
    >     > running fine
    >     >     > > for
    >     >     > > > > few
    >     >     > > > > > > > > months
    >     >     > > > > > > > > > >> > >>> (>4)
    >     >     > > > > > > > > > >> > >>>>> The
    >     >     > > > > > > > > > >> > >>>>>>>>> setup is very simple: 2 hosts 
We
    >     > decided to
    >     >     > > do a
    >     >     > > > > > > > > maintenance
    >     >     > > > > > > > > > >> to
    >     >     > > > > > > > > > >> > >>>>>>>>> increase RAM on both servers
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>> For this we put first server to
    >     > maintenance.
    >     >     > > All
    >     >     > > > > VMS
    >     >     > > > > > > > moved
    >     >     > > > > > > > > > to
    >     >     > > > > > > > > > >> > >>>>> second
    >     >     > > > > > > > > > >> > >>>>>>>>> host after a while.
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>> Then first server was shutdown,
    > RAM
    >     >     > increased,
    >     >     > > > > > server
    >     >     > > > > > > > > turned
    >     >     > > > > > > > > > >> > >>> ON.
    >     >     > > > > > > > > > >> > >>>>>>>>> Now nothing starts on first
    > server.
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>> Tried to delete network, but
    > this
    >     > fails as
    >     >     > > well
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>> Please help !
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>> Here is extract from log:
    >     >     > > > > > > > > > >> > >>>>>>>>>
    > ======================================
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG
    >     >     > > > > > > > > > >> > >>>>>
    > [o.a.c.s.SecondaryStorageManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> (secstorage-1:ctx-16d6c797)
    >     > (logid:7e3160ce)
    >     >     > > > Zone
    >     >     > > > > 1
    >     >     > > > > > is
    >     >     > > > > > > > > ready
    >     >     > > > > > > > > > >> to
    >     >     > > > > > > > > > >> > >>>>>>>>> launch secondary storage VM
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG
    >     >     > > > > > > > > > [c.c.c.ConsoleProxyManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9)
    >     >     > (logid:0a8c8bf4)
    >     >     > > > > Zone
    >     >     > > > > > 1
    >     >     > > > > > > is
    >     >     > > > > > > > > > ready
    >     >     > > > > > > > > > >> > >>> to
    >     >     > > > > > > > > > >> > >>>>>>>>> launch console proxy
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG
    >     >     > > [c.c.a.ApiServlet]
    >     >     > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3)
    >     >     > > (logid:cd8e30be)
    >     >     > > > > > > > > ===START===
    >     >     > > > > > > > > > >> > >>>>>>>> 192.168.5.140
    >     >     > > > > > > > > > >> > >>>>>>>>> -- GET
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
    >     >     > > > > > > > > > >> > >>>>>>>> =json&_=1552998473154
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG
    >     >     > > [c.c.a.ApiServer]
    >     >     > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
    >     > ctx-0cc34dc6)
    >     >     > > > > > > > > > (logid:cd8e30be)
    >     >     > > > > > > > > > >> > >>>>> CIDRs
    >     >     > > > > > > > > > >> > >>>>>>>>> from which account
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]'
    >     >     > > > > > is
    >     >     > > > > > > > > > >> > >>>>>>>> allowed
    >     >     > > > > > > > > > >> > >>>>>>>>> to perform API calls:
    > 0.0.0.0/0,::/0
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,208 INFO
    >     >     > > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:f6751fa7)
    >     >     > > > > > > > > > >> Add
    >     >     > > > > > > > > > >> > >>>>>>>>> job-5081 into job monitoring
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
    >     > ctx-0cc34dc6)
    >     >     > > > > > > > > > (logid:cd8e30be)
    >     >     > > > > > > > > > >> > >>>>> submit
    >     >     > > > > > > > > > >> > >>>>>>>> async
    >     >     > > > > > > > > > >> > >>>>>>>>> job-5081, details: AsyncJobVO
    >     > {id:5081,
    >     >     > > userId:
    >     >     > > > 2,
    >     >     > > > > > > > > > accountId:
    >     >     > > > > > > > > > >> > >>> 2,
    >     >     > > > > > > > > > >> > >>>>>>>>> instanceType: Network,
    > instanceId:
    >     > 204, cmd:
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > >
    >     >     >
    > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
    >     >     > > > > > > > > > >> > >>>>>>> cmdInfo:
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
    >     >     > > > > > > > > > >> > >>>>>>>> interface
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > EventType":"NETWORK.DELETE","_":"1552998473154"},
    >     >     > > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status:
    > IN_PROGRESS,
    >     >     > > > processStatus:
    >     >     > > > > > 0,
    >     >     > > > > > > > > > >> > >>> resultCode:
    >     >     > > > > > > > > > >> > >>>>> 0,
    >     >     > > > > > > > > > >> > >>>>>>>>> result: null, initMsid:
    >     > 264216221068220,
    >     >     > > > > > completeMsid:
    >     >     > > > > > > > > null,
    >     >     > > > > > > > > > >> > >>>>>>> lastUpdated:
    >     >     > > > > > > > > > >> > >>>>>>>>> null, lastPolled: null,
    > created: null}
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG
    >     >     > > [c.c.a.ApiServlet]
    >     >     > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
    >     > ctx-0cc34dc6)
    >     >     > > > > > > > > > (logid:cd8e30be)
    >     >     > > > > > > > > > >> > >>>>>>>>> ===END===
    >     >     > > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
    >     >     > > > > > > > > > >> > >>>>>>>> =json&_=1552998473154
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:16897ea6)
    >     >     > > > > > > > > > >> > >>>>>>>>> Executing AsyncJobVO {id:5081,
    >     > userId: 2,
    >     >     > > > > accountId:
    >     >     > > > > > > 2,
    >     >     > > > > > > > > > >> > >>>>>>>>> instanceType: Network,
    >     >     > > > > > > > > > >> > >>>>>>>>> instanceId: 204, cmd:
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > >
    >     >     >
    > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
    >     >     > > > > > > > > > >> > >>>>>>> cmdInfo:
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
    >     >     > > > > > > > > > >> > >>>>>>>> interface
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > EventType":"NETWORK.DELETE","_":"1552998473154"},
    >     >     > > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status:
    > IN_PROGRESS,
    >     >     > > > processStatus:
    >     >     > > > > > 0,
    >     >     > > > > > > > > > >> > >>> resultCode:
    >     >     > > > > > > > > > >> > >>>>> 0,
    >     >     > > > > > > > > > >> > >>>>>>>>> result: null, initMsid:
    >     > 264216221068220,
    >     >     > > > > > completeMsid:
    >     >     > > > > > > > > null,
    >     >     > > > > > > > > > >> > >>>>>>> lastUpdated:
    >     >     > > > > > > > > > >> > >>>>>>>>> null, lastPolled: null,
    > created: null}
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,257 WARN
    >     >     > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081
    >     >     > > > > > > ctx-d5de7979)
    >     >     > > > > > > > > > >> > >>>>>>>>> (logid:16897ea6) Can't delete
    > the
    >     > network,
    >     >     > not
    >     >     > > > all
    >     >     > > > > > > user
    >     >     > > > > > > > > vms
    >     >     > > > > > > > > > >> are
    >     >     > > > > > > > > > >> > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM]
    > is in
    >     >     > Stopped
    >     >     > > > > state
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:16897ea6)
    >     >     > > > > > > > > > >> > >>>>> Complete
    >     >     > > > > > > > > > >> > >>>>>>>> async
    >     >     > > > > > > > > > >> > >>>>>>>>> job-5081, jobStatus: FAILED,
    >     > resultCode:
    >     >     > 530,
    >     >     > > > > > result:
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
    >     >     > > > > > > > > > >> > >>>>>>>>
    > [],"errorcode":530,"errortext":"Failed
    >     >     > > > > > > > > > >> > >>>>>>>>> to delete network"}
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:16897ea6)
    >     >     > > > > > > > > > >> > >>>>> Publish
    >     >     > > > > > > > > > >> > >>>>>>>>> async
    >     >     > > > > > > > > > >> > >>>>>>>>> job-5081 complete on message 
bus
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:16897ea6)
    >     >     > > > > > > > > > >> > >>> Wake
    >     >     > > > > > > > > > >> > >>>>> up
    >     >     > > > > > > > > > >> > >>>>>>>>> jobs related to job-5081
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:16897ea6)
    >     >     > > > > > > > > > >> > >>> Update
    >     >     > > > > > > > > > >> > >>>>>>>>> db status for job-5081
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:16897ea6)
    >     >     > > > > > > > > > >> > >>> Wake
    >     >     > > > > > > > > > >> > >>>>> up
    >     >     > > > > > > > > > >> > >>>>>>>>> jobs joined with job-5081 and
    > disjoin
    >     > all
    >     >     > > > subjobs
    >     >     > > > > > > > created
    >     >     > > > > > > > > > from
    >     >     > > > > > > > > > >> > >>>>> job-
    >     >     > > > > > > > > > >> > >>>>>>>>> 5081
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:16897ea6)
    >     >     > > > > > > > > > >> > >>> Done
    >     >     > > > > > > > > > >> > >>>>>>>> executing
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >>
    >     >     > > > >
    > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd
    >     >     > > > > > > > > > >> > >>>>> for
    >     >     > > > > > > > > > >> > >>>>>>>>> job-5081
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 INFO
    >     >     > > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor]
    >     >     > > > > > > > > > >> > >>>>>>>>> 
(API-Job-Executor-1:ctx-d4970c19
    >     > job-5081)
    >     >     > > > > > > > > (logid:16897ea6)
    >     >     > > > > > > > > > >> > >>> Remove
    >     >     > > > > > > > > > >> > >>>>>>>> job-5081
    >     >     > > > > > > > > > >> > >>>>>>>>> from job monitoring
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG
    >     >     > > [c.c.a.ApiServlet]
    >     >     > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85)
    >     >     > > (logid:7fe68f75)
    >     >     > > > > > > > > ===START===
    >     >     > > > > > > > > > >> > >>>>>>>> 192.168.5.140
    >     >     > > > > > > > > > >> > >>>>>>>>> -- GET
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
    >     >     > > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG
    >     >     > > [c.c.a.ApiServer]
    >     >     > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85
    >     > ctx-da1f4cbd)
    >     >     > > > > > > > > (logid:7fe68f75)
    >     >     > > > > > > > > > >> > >>> CIDRs
    >     >     > > > > > > > > > >> > >>>>>>>>> from which account
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]'
    >     >     > > > > > is
    >     >     > > > > > > > > > >> > >>>>>>>> allowed
    >     >     > > > > > > > > > >> > >>>>>>>>> to perform API calls:
    > 0.0.0.0/0,::/0
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG
    >     >     > > [c.c.a.ApiServlet]
    >     >     > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85
    >     > ctx-da1f4cbd)
    >     >     > > > > > > > > (logid:7fe68f75)
    >     >     > > > > > > > > > >> > >>>>>>>>> ===END===
    >     >     > > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > >
    >     >     > > > > >
    >     >     > >
    >     >
    > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
    >     >     > > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,946 INFO
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>>
    > (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
    >     >     > > > > > > (logid:a605267a)
    >     >     > > > > > > > > > Begin
    >     >     > > > > > > > > > >> > >>>>>>>>> cleanup expired async-jobs
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,951 INFO
    >     >     > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>>
    > (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
    >     >     > > > > > > (logid:a605267a)
    >     >     > > > > > > > > End
    >     >     > > > > > > > > > >> > >>>>> cleanup
    >     >     > > > > > > > > > >> > >>>>>>>> expired
    >     >     > > > > > > > > > >> > >>>>>>>>> async-jobs
    >     >     > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
    >     >     > > > > > > > > > >> > >>>>>>>>>
    > (RouterStatusMonitor-1:ctx-ad6bbe7e)
    >     >     > > > > > (logid:04e4c72b)
    >     >     > > > > > > > > Found
    >     >     > > > > > > > > > 0
    >     >     > > > > > > > > > >> > >>>>>>>>> routers to update status.
    >     >     > > > > > > > > > >> > >>>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>> --
    >     >     > > > > > > > > > >> > >>>>>>>> Rafael Weingärtner
    >     >     > > > > > > > > > >> > >>>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>>
    >     >     > > > > > > > > > >> > >>>>>>
    >     >     > > > > > > > > > >> > >>>>>
    >     >     > > > > > > > > > >> > >>>>
    >     >     > > > > > > > > > >> > >>>
    >     >     > > > > > > > > > >> > >>
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >> >
    >     >     > > > > > > > > > >>
    >     >     > > > > > > > > > >
    >     >     > > > > > > > > >
    >     >     > > > > > > > >
    >     >     > > > > > > >
    >     >     > > > > > >
    >     >     > > > > >
    >     >     > > > >
    >     >     > > >
    >     >     > >
    >     >     >
    >     >     >
    >     >     > --
    >     >     >
    >     >     > Andrija Panić
    >     >     >
    >     >
    >     >
    >     >
    >
    >
    >
    

Reply via email to