Jevgeni,
Can you also explain your infrastructure - you said you have two hosts only,
where does CloudStack management run?
Reason I'm asking is when checking your logs from yesterday the IP address
192.168.1.14 seems to be used for management, NFS and a KVM host? Is this the
case, do you co-host everything on the same server?
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 20/03/2019, 12:49, "Andrija Panic" <andrija.pa...@gmail.com> wrote:
Just to confirm, you are using Basic Zone in CloudStack, right ?
Can you confirm that router has been actually started/created on KVM side,
again, as requested please post logs (mgmt and agent - and note the time
around which you tried to start VR last time it partially succeeded) - we
can't guess what went wrong without logs.
I would push more effort solving this one, instead of reinstalling - you
might hit the issue again and then it's no good.
Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e.
not disabled and hosts connected AND both SSVM and CPVM are
connectedUP/green
Is your dashboard in GUI all green - no issues there ?
Can you connect your both KVM hosts can access/mount both Primary and
Secondary Storage
On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
wrote:
> After dozen of attempts, the Virtual Router could finally be recreated.
But
> its in eternal Starting status, and console prompts it required upgrade
and
> Version is UNKNOWN
>
> It does not resolve the problem, I cannot move further form this point.
> Any hints?
>
> Or I am condemned to do reinstall cloudstack from scratch?
>
> On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no
> > VMS - "no data to show"
> >
> > I dont have any network, other than this single default one.
> >
> > I've tried adding new network - Add guest network. But I am not able to
> do
> > so, cause in the wizard popup, it offers empty dropdown with Zones
> > selection. And this wizard doesnt not allow to go further without
> selecting
> > Zone
> >
> > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic <andrija.pa...@gmail.com>
> > wrote:
> >
> >> you need to delete/remove all VMs inside this network (tick the
> "Expunge"
> >> button during VM deletion - if you want to really delete the VMs) in
> order
> >> to be able to delete the network - OR simply attach this VM to another
> >> network, make this new network a DEFAULT one (NIC that is...), and then
> >> detach from old network - and then effectively your VM was "removed"
> from
> >> old network - after this you should be able to delete the old network.
I
> >> assume some DB incosistencies perhaps, being the reason you can not
> >> restart
> >> the network.
> >>
> >> Did you try restarting some other Network - or deploying a new network,
> >> spin a VM in it, then again try to restart this new network - does it
> >> work ?
> >>
> >> Andrija
> >>
> >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> >> wrote:
> >>
> >> > >>>Stop mgmt,
> >> > >>>Stop all agents
> >> > >>>Restart libvirtd (and check libvirt logs afterwards)
> >> > >>>Start agents
> >> > >>>Start mgmt.
> >> >
> >> > I did that numerous time. Nothing really suspicious
> >> > I can see that systems VMs are running - both in cloudstack console
> and
> >> > with virsh list -all
> >> >
> >> > It is apparently problem with network.
> >> > Is there a way to force recreation of defaultGuestNetwork? or force
> >> > recreation of Virtual Router.
> >> > I am unable to delete network, which is supposed to rebuild network
> with
> >> > its router. Thats the issue
> >> >
> >> > The issue with libvirtd was, that eventually at some point it was
> >> updated
> >> > during 4 months of running, and not rebooted. It still worked. We had
> to
> >> > add listen_tcp = 1 for libvirtd to start working again.
> >> >
> >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic <
> andrija.pa...@gmail.com>
> >> > wrote:
> >> >
> >> > > As Sergey suggested... but i would also verify no libvirt issues or
> >> > storage
> >> > > pool issues - so perhaps:
> >> > >
> >> > > Stop mgmt,
> >> > > Stop all agents
> >> > > Restart libvirtd (and check libvirt logs afterwards)
> >> > > Start agents
> >> > > Start mgmt.
> >> > >
> >> > > What was originally issue with libvirtd ?
> >> > > That sounds fishy to me...
> >> > >
> >> > > Andrija
> >> > >
> >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <serg...@hotmail.com>
> >> > wrote:
> >> > >
> >> > > > select * from networks where removed is null;
> >> > > > select * from vm_instance where id=87;
> >> > > > select id,name from vm_instance where name like 'r%' and removed
> is
> >> > null;
> >> > > >
> >> > > > Basically since the network offering is not redundant this error
> is
> >> > only
> >> > > > thrown when there is no router associated with your network.
> Usually
> >> > > > management server restart tries to implement network again.
Please
> >> > > restart
> >> > > > management server, save and share management server log.
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" <
> j.zolotar...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > >>>>Check network_offering table for value in column
> >> > > > redundant_router_service for the network offering you use.
> >> > > > in table network_offering_table all records have
> >> > > > redundant_router_service =
> >> > > > 0
> >> > > >
> >> > > > Can you also run the following:
> >> > > > >>>select name, state, removed from host where name like
'r%'
> >> > > > returns zero rows - nothing
> >> > > >
> >> > > > >>>select * from domain_router;
> >> > > > # id, element_id, public_mac_address, public_ip_address,
> >> > > > public_netmask,
> >> > > > guest_netmask, guest_ip_address, is_redundant_router,
> priority,
> >> > > > redundant_state, stop_pending, role, template_version,
> >> > > scripts_version,
> >> > > > vpc_id, update_state
> >> > > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> 15:37:28
> >> > UTC
> >> > > > 2018',
> >> > > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL
> >> > > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> 15:37:28
> >> > UTC
> >> > > > 2018',
> >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> >> > > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> 15:37:28
> >> > UTC
> >> > > > 2018',
> >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> >> > > > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
> >> > > > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> 15:37:28
> >> > UTC
> >> > > > 2018',
> >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> >> > > > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> 15:37:28
> >> > UTC
> >> > > > 2018',
> >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> >> > > > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22
> 07:52:17
> >> > UTC
> >> > > > 2018',
> >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED'
> >> > > > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22
> 07:52:17
> >> > UTC
> >> > > > 2018',
> >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> >> > > > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22
> 07:52:17
> >> > UTC
> >> > > > 2018',
> >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> >> > > > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
> >> > > > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
'UNKNOWN',
> >> '0',
> >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12
> >> 15:06:49
> >> > > UTC
> >> > > > 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL
> >> > > >
> >> > > >
> >> > > > >>>select * from router_network_ref;
> >> > > > # id, router_id, network_id, guest_type
> >> > > > '1', '4', '204', 'Shared'
> >> > > > '2', '49', '204', 'Shared'
> >> > > > '3', '73', '204', 'Shared'
> >> > > > '4', '75', '204', 'Shared'
> >> > > > '5', '76', '204', 'Shared'
> >> > > > '6', '77', '204', 'Shared'
> >> > > > '7', '80', '204', 'Shared'
> >> > > > '8', '85', '204', 'Shared'
> >> > > > '9', '86', '204', 'Shared'
> >> > > > '10', '87', '204', 'Shared'
> >> > > >
> >> > > >
> >> > > > On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy <
> >> > > serg...@hotmail.com
> >> > > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Check network_offering table for value in column
> >> > > > > redundant_router_service for the network offering you use.
> >> > > > > Can you also run the following:
> >> > > > > select name, state, removed from host where name like 'r%'
> >> > > > > select * from domain_router;
> >> > > > > select * from router_network_ref;
> >> > > > >
> >> > > > > Cloudstack is supposed to recreate you VR. If it is not
> >> happening
> >> > > > there is
> >> > > > > something fundamentally wrong. I would advise to destroy
> your
> >> VR
> >> > > > again.
> >> > > > > Stop you management server. Rotate management server log
and
> >> > start
> >> > > it
> >> > > > > again. If your VR doesn't start in few min, post your
> complete
> >> > > > management
> >> > > > > server log and agent log again.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" <
> >> > j.zolotar...@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > >>>Network offering needs to be change to non-redundant
> >> > > > > How do I do that?
> >> > > > >
> >> > > > > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy <
> >> > > > serg...@hotmail.com
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Network offering needs to be change to non-redundant.
> >> Most
> >> > > > likely
> >> > > > > old bug
> >> > > > > > is resurfaced.
> >> > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <
> >> > > > j.zolotar...@gmail.com>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > Hello
> >> > > > > >
> >> > > > > > I did exactly like you suggested.
> >> > > > > >
> >> > > > > > After UPDATE on db, I can see the router in
> >> cloudstack
> >> > > > console.
> >> > > > > > But attempt to restart network fails.
> >> > > > > > I get error:
> >> > > > > > Resource [DataCenter:1] is unreachable: Can't
find
> >> all
> >> > > > necessary
> >> > > > > > running
> >> > > > > > routers!
> >> > > > > >
> >> > > > > > I rechecked agents on both servers. They look
> >> running
> >> > OK.
> >> > > > > >
> >> > > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic <
> >> > > > > > andrija.pa...@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Ok, so:
> >> > > > > > >
> >> > > > > > > 1.BACKUP YOUR DB - in case there is issues, you
> >> will
> >> > > > want to
> >> > > > > restore
> >> > > > > > > 2. I tried to reproduce your case on 4.11.2
> >> (though
> >> > > > didn't do
> >> > > > > any
> >> > > > > > > maintenance etc.) - and could not - I have
> >> > artificially
> >> > > > marked
> >> > > > > > existing VR
> >> > > > > > > as removed in DB - and tried to restart network
> >> and
> >> > it
> >> > > > worked
> >> > > > > just
> >> > > > > > fine.
> >> > > > > > >
> >> > > > > > > Let's try to "undelete" the VR for that network
> -
> >> > > though
> >> > > > I
> >> > > > > can't be
> >> > > > > > sure if
> >> > > > > > > this will work or not.
> >> > > > > > >
> >> > > > > > > Find VR based on the network UUID:
> >> > > > > > > SELECT * FROM vm_instance WHERE id IN
> >> (SELECT
> >> > > > > instance_id from
> >> > > > > > nics
> >> > > > > > > where network_id IN (SELECT id FROM networks
> WHERE
> >> > > > > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> >> > > > > > > Obviously, replace this UUID with your network
> >> UUID.
> >> > > > > > > SQL above will return all VMs (user VMs, VRs,
> etc,
> >> > > > delete or
> >> > > > > not...)
> >> > > > > > that
> >> > > > > > > have NIC inside the network with that UUID -
> look
> >> for
> >> > > the
> >> > > > > r-XX-VM
> >> > > > > > being a
> >> > > > > > > router.
> >> > > > > > >
> >> > > > > > > Now that you have the name of the VR (i.e.
> >> r-4-VM) -
> >> > > you
> >> > > > want
> >> > > > > to set
> >> > > > > > 2
> >> > > > > > > fields to some other value than what it might
be
> >> > > > already, as
> >> > > > > below:
> >> > > > > > >
> >> > > > > > > Need to set field "state" to Stopped:
> >> > > > > > > UPDATE cloud.vm_instance SET
> state='Stopped'
> >> > > WHERE
> >> > > > > > name="r-4-VM";
> >> > > > > > > (what state is set currently ???)
> >> > > > > > >
> >> > > > > > > Need to set "removed" to NULL value:
> >> > > > > > > UPDATE cloud.vm_instance SET removed =
> NULL
> >> > WHERE
> >> > > > > > NAME="r-4-VM"; (is
> >> > > > > > > there a removed date already set ???)
> >> > > > > > >
> >> > > > > > > Obviously make sure that this VR is not running
> on
> >> > any
> >> > > > > hypervisor
> >> > > > > > and if
> >> > > > > > > running, virsh destroy it...
> >> > > > > > >
> >> > > > > > > AFTER above has been done (VR is considered to
> >> exist
> >> > > but
> >> > > > > stopped,
> >> > > > > > from
> >> > > > > > > CloudStack point of view) - try to RESTART the
> >> > network
> >> > > > (don't
> >> > > > > bother
> >> > > > > > > deleteing it, since there are VMs in that
> network)
> >> > > > > > >
> >> > > > > > > I'm not 100% positive this will fix your issue,
> >> but
> >> > > > doesn't
> >> > > > > hurt to
> >> > > > > > try
> >> > > > > > >
> >> > > > > > > If above doesn't work - I would still take a
> look
> >> > into
> >> > > > agents
> >> > > > > and if
> >> > > > > > they
> >> > > > > > > are still connected - optionally, restart
agents
> >> on
> >> > > both
> >> > > > hosts
> >> > > > > once
> >> > > > > > more
> >> > > > > > > and confirm they are connected and up.
> >> > > > > > >
> >> > > > > > > Let us know how it goes (and please backup DB
> once
> >> > more
> >> > > > before
> >> > > > > any
> >> > > > > > actions
> >> > > > > > > !)
> >> > > > > > >
> >> > > > > > > Andrija
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni
> Zolotarjov <
> >> > > > > > j.zolotar...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Yes. Just a single network.
> >> > > > > > > >
> >> > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, <
> >> > > > > andrija.pa...@gmail.com
> >> > > > > > >
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Just one more clarification - this is
> Isolate
> >> > > single
> >> > > > > network (not
> >> > > > > > > Shared
> >> > > > > > > > > Network, not VPC) ?
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni
> >> Zolotarjov <
> >> > > > > > j.zolotar...@gmail.com
> >> > > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Name defaultGuestNetwork
> >> > > > > > > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> >> > > > > > > > > > Zone PBT zone 1
> >> > > > > > > > > > Description defaultGuestNetwork
> >> > > > > > > > > > Type Shared
> >> > > > > > > > > > State Setup
> >> > > > > > > > > > VPC ID N/A
> >> > > > > > > > > > Persistent No
> >> > > > > > > > > >
> >> > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija
> >> Panic <
> >> > > > > > > andrija.pa...@gmail.com
> >> > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Share the network id and name as seen
> from
> >> > > GUI...
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni
> >> > Zolotarjov
> >> > > <
> >> > > > > > > > j.zolotar...@gmail.com
> >> > > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > >>> 1. Confirm please rolling restart
> is
> >> > set
> >> > > > to false
> >> > > > > > please -
> >> > > > > > > > double
> >> > > > > > > > > > > check
> >> > > > > > > > > > > > Double checked - It is set to false
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > >>>>2. If so - do you know the name
of
> >> VR
> >> > > > which you
> >> > > > > > deleted ? Is
> >> > > > > > > it
> >> > > > > > > > > > last
> >> > > > > > > > > > > > one
> >> > > > > > > > > > > > >>> ever created - if so we can find
> it
> >> > > > easily...
> >> > > > > > > > > > > > I dont know the name
> >> > > > > > > > > > > > Is there a way to fetch it from DB?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM
> Andrija
> >> > > Panic <
> >> > > > > > > > > andrija.pa...@gmail.com
> >> > > > > > > > > > >
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Right...still complaining on
missing
> >> > > running
> >> > > > > routers.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > 1. Confirm please rolling restart
is
> >> set
> >> > to
> >> > > > false
> >> > > > > please
> >> > > > > > -
> >> > > > > > > double
> >> > > > > > > > > > check
> >> > > > > > > > > > > > > 2. If so - do you know the name of
> VR
> >> > which
> >> > > > you
> >> > > > > deleted
> >> > > > > > ? Is it
> >> > > > > > > > > last
> >> > > > > > > > > > > one
> >> > > > > > > > > > > > > ever created - if so we can find it
> >> > > easily...
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni
> >> > > > Zolotarjov <
> >> > > > > > > > > > j.zolotar...@gmail.com
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Here is management server log
> >> > > > > > > > > > > > > >
> >> > > > > > > > >
> >> > > > > >
> >> > > >
> https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM
> >> Andrija
> >> > > > Panic <
> >> > > > > > > > > > > andrija.pa...@gmail.com
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > I would also try to "undelete"
> VR
> >> in
> >> > > DB,
> >> > > > but
> >> > > > > let's
> >> > > > > > keep
> >> > > > > > > this
> >> > > > > > > > as
> >> > > > > > > > > > > last
> >> > > > > > > > > > > > > > step.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:24
> >> Andrija
> >> > > > Panic <
> >> > > > > > > > > > andrija.pa...@gmail.com
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Disclaimer: from mobile
> device.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > I don't see any special
> failure
> >> in
> >> > > > agent log,
> >> > > > > > except some
> >> > > > > > > > > long
> >> > > > > > > > > > > > > running
> >> > > > > > > > > > > > > > > > migration, timeout for
> graceful
> >> VM
> >> > > > shutdown
> >> > > > > etc
> >> > > > > > (and
> >> > > > > > > agent
> >> > > > > > > > > > > restart)
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > You did not send mgmt log
> after
> >> > last
> >> > > > network
> >> > > > > > restart
> >> > > > > > > > failure
> >> > > > > > > > > ?
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 17:29
> >> Jevgeni
> >> > > > > Zolotarjov <
> >> > > > > > > > > > > > > j.zolotar...@gmail.com
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >> Under
> >> > > > > > > > > > > > > > > >> Infrastructure / Hosts:
> >> > > > > > > > > > > > > > > >> both hosts are enabled and
> >> green
> >> > > > (Unsecure
> >> > > > > though)
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > >> agent.log -
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > >
> >> > > > >
> >> > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > >> I set
network.rolling.restart
> >> to
> >> > > > false now.
> >> > > > > > Restarted
> >> > > > > > > > > > management
> >> > > > > > > > > > > > > > server
> >> > > > > > > > > > > > > > > -
> >> > > > > > > > > > > > > > > >> same problem - cannot
restart
> >> not
> >> > > > delete
> >> > > > > network
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48
> PM
> >> > > Boris
> >> > > > > Stoyanov <
> >> > > > > > > > > > > > > > > >> boris.stoya...@shapeblue.com
> >
> >> > > > > > > > > > > > > > > >> wrote:
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > >> > Hi, you shouldn’t be
> worried
> >> > about
> >> > > > data
> >> > > > > as long
> >> > > > > > as you
> >> > > > > > > > > use a
> >> > > > > > > > > > > > > > separate
> >> > > > > > > > > > > > > > > >> > shared storage.
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> > What does the cloudstack
> >> console
> >> > > say
> >> > > > > about your
> >> > > > > > host?
> >> > > > > > > Is
> >> > > > > > > > > it
> >> > > > > > > > > > > up?
> >> > > > > > > > > > > > If
> >> > > > > > > > > > > > > > > it’s
> >> > > > > > > > > > > > > > > >> up
> >> > > > > > > > > > > > > > > >> > then you should be able to
> >> > deploy
> >> > > a
> >> > > > VM on
> >> > > > > it.
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> > If not, you’ll need to
> >> > investigate
> >> > > > the
> >> > > > > reason
> >> > > > > > > > > > cloudstack-agent
> >> > > > > > > > > > > > is
> >> > > > > > > > > > > > > > not
> >> > > > > > > > > > > > > > > >> > connected, is the agent
> >> running?
> >> > > If
> >> > > > not
> >> > > > > just
> >> > > > > > start it
> >> > > > > > > > and
> >> > > > > > > > > > see
> >> > > > > > > > > > > if
> >> > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > >> > connects.
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> > If it’s running but still
> not
> >> > > > connected
> >> > > > > please
> >> > > > > > share
> >> > > > > > > the
> >> > > > > > > > > > > > > agent.log.
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> > If nothing helps at all
you
> >> can
> >> > > > remove
> >> > > > > the host
> >> > > > > > and
> >> > > > > > > add
> >> > > > > > > > it
> >> > > > > > > > > > > > again,
> >> > > > > > > > > > > > > > but
> >> > > > > > > > > > > > > > > >> > let’s try to troubleshoot
> >> first.
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> > Bobby.
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> >
> boris.stoya...@shapeblue.com
> >> > > > > > > > > > > > > > > >> > www.shapeblue.com
> >> > > > > > > > > > > > > > > >> > Amadeus House, Floral
> Street,
> >> > > London
> >> > > > > WC2E 9DPUK
> >> > > > > > > > > > > > > > > >> > @shapeblue
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >> > > On 19 Mar 2019, at
17:29,
> >> > > Jevgeni
> >> > > > > Zolotarjov <
> >> > > > > > > > > > > > > > > j.zolotar...@gmail.com>
> >> > > > > > > > > > > > > > > >> > wrote:
> >> > > > > > > > > > > > > > > >> > >
> >> > > > > > > > > > > > > > > >> > > Guys, please help with
> it.
> >> > > > > > > > > > > > > > > >> > > What can be done here?
> >> > > > > > > > > > > > > > > >> > > There is too much
> valuable
> >> > data.
> >> > > > > > > > > > > > > > > >> > >
> >> > > > > > > > > > > > > > > >> > > On Tue, Mar 19, 2019 at
> >> 4:21
> >> > PM
> >> > > > Jevgeni
> >> > > > > > Zolotarjov <
> >> > > > > > > > > > > > > > > >> > j.zolotar...@gmail.com>
> >> > > > > > > > > > > > > > > >> > > wrote:
> >> > > > > > > > > > > > > > > >> > >
> >> > > > > > > > > > > > > > > >> > >> Tried that just now and
> >> got
> >> > > > error:
> >> > > > > > > > > > > > > > > >> > >> Resource [DataCenter:1]
> is
> >> > > > unreachable:
> >> > > > > > Can't find
> >> > > > > > > > all
> >> > > > > > > > > > > > > necessary
> >> > > > > > > > > > > > > > > >> running
> >> > > > > > > > > > > > > > > >> > >> routers!
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> > >> In the log I see:
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > =============================================
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,644
> >> DEBUG
> >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> >> > > > > > > > > > > > > > > >> > >>
> >> > > (API-Job-Executor-4:ctx-7b6b69eb
> >> > > > > job-5093
> >> > > > > > > > ctx-9be30648)
> >> > > > > > > > > > > > > > > >> (logid:265a6099)
> >> > > > > > > > > > > > > > > >> > >> Restarting network
> 204...
> >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,645
> >> DEBUG
> >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> >> > > > > > > > > > > > > > > >> > >>
> >> > > (API-Job-Executor-4:ctx-7b6b69eb
> >> > > > > job-5093
> >> > > > > > > > ctx-9be30648)
> >> > > > > > > > > > > > > > > >> (logid:265a6099)
> >> > > > > > > > > > > > > > > >> > >> Performing rolling
> >> restart of
> >> > > > routers
> >> > > > > of
> >> > > > > > network
> >> > > > > > > > > > > > > > Ntwk[204|Guest|6]
> >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,649
> >> DEBUG
> >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> >> > > > > > > > > > > > > > > >> > >>
> >> > > (API-Job-Executor-4:ctx-7b6b69eb
> >> > > > > job-5093
> >> > > > > > > > ctx-9be30648)
> >> > > > > > > > > > > > > > > >> (logid:265a6099)
> >> > > > > > > > > > > > > > > >> > >> Asking VirtualRouter to
> >> > > > implemenet
> >> > > > > > > Ntwk[204|Guest|6]
> >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,658
> >> ERROR
> >> > > > > > > > > > [c.c.a.ApiAsyncJobDispatcher]
> >> > > > > > > > > > > > > > > >> > >>
> >> > > (API-Job-Executor-4:ctx-7b6b69eb
> >> > > > > job-5093)
> >> > > > > > > > > > (logid:265a6099)
> >> > > > > > > > > > > > > > > >> Unexpected
> >> > > > > > > > > > > > > > > >> > >> exception while
> executing
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > >
> >> > > > > > >
> >> > > > >
> >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > com.cloud.exception.ResourceUnavailableException:
> >> > > > > > > > > > Resource
> >> > > > > > > > > > > > > > > >> > [DataCenter:1]
> >> > > > > > > > > > > > > > > >> > >> is unreachable: Can't
> find
> >> > all
> >> > > > > necessary
> >> > > > > > running
> >> > > > > > > > > routers!
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > >
> >> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> > > > > > > > > > > > > > > Method)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > >
> >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> >> > > > > > > > > > > > > > > Source)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > >
> >> > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> >> > > > > > > > > > > > > > > >> > Source)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > java.lang.reflect.Method.invoke(Unknown
> >> > > > > > > > > Source)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > >
> com.sun.proxy.$Proxy229.restartNetwork(Unknown
> >> > > > > > > > > > > > > Source)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > >
> >> > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > >
> >> > > > > >
> >> java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> >> > > > > > > > > > > > > > > >> > >> Source)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > java.util.concurrent.FutureTask.run(Unknown
> >> > > > > > > > > > > Source)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > >
> >> > > > > >
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> >> > > > > > > > > > > > > > > >> > >> Source)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > > > > > > > > > > > >
> >> > > > > >
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >> > > > > > > > > > > > > > > >> > >> Source)
> >> > > > > > > > > > > > > > > >> > >> at
> >> > > > java.lang.Thread.run(Unknown
> >> > > > > > Source)
> >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,659
> >> DEBUG
> >> > > > > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> > > > > > > > > > > > > > > >> > >>
> >> > > (API-Job-Executor-4:ctx-7b6b69eb
> >> > > > > job-5093)
> >> > > > > > > > > > (logid:265a6099)
> >> > > > > > > > > > > > > > > Complete
> >> > > > > > > > > > > > > > > >> > async
> >> > > > > > > > > > > > > > > >> > >> job-5093, jobStatus:
> >> FAILED,
> >> > > > > resultCode: 530,
> >> > > > > > > result:
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> >
> >> > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> >> > > > > > > > > > > > > > > >> > >> [DataCenter:1] is un
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at
> >> 4:19
> >> > PM
> >> > > > Andrija
> >> > > > > > Panic <
> >> > > > > > > > > > > > > > > >> >
> andrija.pa...@shapeblue.com>
> >> > > > > > > > > > > > > > > >> > >> wrote:
> >> > > > > > > > > > > > > > > >> > >>
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>> Your network can't be
> >> > deleted
> >> > > > due to
> >> > > > > "Can't
> >> > > > > > delete
> >> > > > > > > > the
> >> > > > > > > > > > > > > network,
> >> > > > > > > > > > > > > > > not
> >> > > > > > > > > > > > > > > >> all
> >> > > > > > > > > > > > > > > >> > >>> user vms are expunged.
> Vm
> >> > > > > > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is
> in
> >> > > Stopped
> >> > > > > state" -
> >> > > > > > which is
> >> > > > > > > > > fine.
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>> You should be able to
> >> just
> >> > > > start the
> >> > > > > user
> >> > > > > > VM - but
> >> > > > > > > > if
> >> > > > > > > > > > you
> >> > > > > > > > > > > > have
> >> > > > > > > > > > > > > > > >> actually
> >> > > > > > > > > > > > > > > >> > >>> delete the VR itself,
> >> then
> >> > > just
> >> > > > do
> >> > > > > Network
> >> > > > > > restart
> >> > > > > > > > > with
> >> > > > > > > > > > > > > > "cleanup"
> >> > > > > > > > > > > > > > > >> and
> >> > > > > > > > > > > > > > > >> > it
> >> > > > > > > > > > > > > > > >> > >>> will recreate a new
VR,
> >> > after
> >> > > > which
> >> > > > > you
> >> > > > > > should be
> >> > > > > > > > able
> >> > > > > > > > > > to
> >> > > > > > > > > > > > > start
> >> > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > >> VM.
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>> Andrija
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>>
> >> andrija.pa...@shapeblue.com
> >> > > > > > > > > > > > > > > >> > >>> www.shapeblue.com
> >> > > > > > > > > > > > > > > >> > >>> Amadeus House, Floral
> >> > Street,
> >> > > > London
> >> > > > > WC2E
> >> > > > > > 9DPUK
> >> > > > > > > > > > > > > > > >> > >>> @shapeblue
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>> -----Original
> >> Message-----
> >> > > > > > > > > > > > > > > >> > >>> From: Jevgeni
> Zolotarjov
> >> <
> >> > > > > > j.zolotar...@gmail.com>
> >> > > > > > > > > > > > > > > >> > >>> Sent: 19 March 2019
> 15:10
> >> > > > > > > > > > > > > > > >> > >>> To:
> >> > > users@cloudstack.apache.org
> >> > > > > > > > > > > > > > > >> > >>> Subject: Re: Disaster
> >> after
> >> > > > > maintenance
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>> I mean I cannot delete
> >> > > network:
> >> > > > In the
> >> > > > > > management
> >> > > > > > > > > server
> >> > > > > > > > > > > > log I
> >> > > > > > > > > > > > > > see
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > > > > > > > > >> > >>>
> >> > > > > > > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >>
> >> Andrija Panić
> >>
> >
>
--
Andrija Panić
dag.sonst...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
