+1 on the advice to start from scratch.
Provisioning is failing because it can’t spin up either SSVM or proxy due to
not enough capacity. The reason might be:
* Not enough capacity either CPU or RAM. increasing overprovisioning
factors or reducing disable thresholds might help.
* Hosts in error state
* Cluster disabled
* Problem accessing primary and/or secondary storage mount from management
server host
2019-03-20 15:07:39,218 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264)
(logid:49483c7a) Trying to allocate a host and storage pools from dc:3,
pod:null,cluster:null, requested cpu: 500, requested ram: 536870912
2019-03-20 15:07:39,218 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264)
(logid:49483c7a) Is ROOT volume READY (pool already allocated)?: No
2019-03-20 15:07:39,219 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264)
(logid:49483c7a) Deploy avoids pods: null, clusters: null, hosts: null
2019-03-20 15:07:39,219 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264)
(logid:49483c7a) Searching all possible resources under this Zone: 3
2019-03-20 15:07:39,219 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-38:ctx-f824bfeb job-5119/job-7076 ctx-98888498)
(logid:bc39cd2a) No clusters found having a host with enough capacity,
returning.
2019-03-20 15:07:39,219 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264)
(logid:49483c7a) Listing clusters in order of aggregate capacity, that have
(atleast one host with) enough CPU and RAM capacity under this Zone: 3
2019-03-20 15:07:39,221 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264)
(logid:49483c7a) No clusters found having a host with enough capacity,
returning.
On 3/20/19, 10:38 AM, "Andrija Panic" <andrija.pa...@gmail.com> wrote:
Hi Jevgeni,
I would perhaps consider you continue with plan B from your separate email
thread (root volumes --> create snapshots, convert snaps to template,
download template somewhere safe - for DATA volumes, also create snapshots,
then convert to volume and download it (or simply directly download
existing DATA volume if VM is stopped).
Once you are safe, and all templates, and VM volumes are safe, you are good
to reinstall.
Seriously, I'm not sure how to proceed via ML - if this was my own setup,
probably would be able to fix it...
In next installment, start with clean 4.11.2 (4.10 was never released as an
official release and was SERIOUSLY broken), or even 4.12 which has just
been released (will be in 1-2 days).
In this new installment, please dedicate a VM (or physical server) that
will host mgmt+DB+NFS (or even better separate NFS on different server
etc.) - but certainly do NOT collocate management components with KVM role.
When you build zone successfully, you can then import all templates and
upload all volumes (which you saved previously to some external place, web
server)
This will allow you to restore your VMs - possibly with just different IPs
versus original ones.
Considering failed DB upgrades and issues you see now, I assume your env,
might be severely broken at this point, and warrants starting from
scratch...
Hope that makes sense - so again, download all root and data volumes to
safe place (consider some petrol + matches fun) and then reinstall with a
fresh and shiny infra.
Alternatively, I would try to wipe all new zones (this takes some time and
certain steps) and then continue troubleshooting with failed-to-start VRs.
Cheers
On Wed, 20 Mar 2019 at 17:59, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
wrote:
> It started with 4.10 and then gradually upgraded with all stops, when new
> releases were available.
>
>
> >>> Why do you have 3 zones in this installation - what is the setup ?
> >>> SSVM and CPVM (for whatever zone) are failing to be created...
> Its a result of attempts to create new zone and somehow move VMs to this
> new zone. These all are unsuccessful attempts.
> Before problem started there was 1 zone and There should be just 1 zone in
> reality.
>
>
> >>> yes, the VR can't be started, it get's timeout - in AGENT logs, I see
> that
> >>> it attemps to create a volume on primary storage...
> I guess this is the root cause. I checked, and primary storage is
> accessible via NFS share on both hosts. How to troubleshoot it?
>
>
> On Wed, Mar 20, 2019 at 6:29 PM Andrija Panic <andrija.pa...@gmail.com>
> wrote:
>
> > Hi,
> >
> > 2019-03-20 06:41:50,446 INFO [c.c.u.DatabaseUpgradeChecker] (main:null)
> > (logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0
> > 2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null)
> > (logid:) Running upgrade Upgrade41000to41100 to upgrade from
> > 4.10.0.0-4.11.0.0 to 4.11.0.0
> > fails due to
> > java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to
> > './cloud/#sql2-2f01-13d' (errno: 152)
> >
> > Then later...
> >
> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create a
> > deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface
> > com.cloud.dc.DataCenter; id=3
> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create a
> > deployment for VM[ConsoleProxy|v-733-VM]Scope=interface
> > com.cloud.dc.DataCenter; id=3
> >
> > 2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch
> > secondary storage VM
> > 2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to
> launch
> > secondary storage VM yet
> > 2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch
> > secondary storage VM
> >
> > so did you start with clean 4.11.2 install, or was it upgraded one - I
> see
> > in logs an upgrade from DB version 4.10 to 4.11 was tried and failed...
> > Why do you have 3 zones in this installation - what is the setup ?
> > SSVM and CPVM (for whatever zone) are failing to be created...
> >
> > yes, the VR can't be started, it get's timeout - in AGENT logs, I see
> that
> > it attemps to create a volume on primary storage...
> >
> >
> > Also, for SSVM I got this one...
> > 2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7)
> > (logid:49483c7a) No clusters found having a host with enough capacity,
> > returning.
> >
> > Andrija
> >
> > On Wed, 20 Mar 2019 at 16:39, Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >
> > wrote:
> >
> > > Basic Zone - Yes
> > >
> > > router has been actually started/created on KVM side - not created,
not
> > > started. Thats the main problem, I guess
> > >
> > > agent.log
> > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > >
> > > management log
> > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > >
> > > >> Can you confirm your zone/pod/cluster/hosts are all in Enabled
> state,
> > > i.e.
> > > YES, all green
> > >
> > > >> Can you connect your both KVM hosts can access/mount both Primary
> and
> > > Secondary Storage
> > > YES. Double checked
> > >
> > > >>>Can you also explain your infrastructure - you said you have two
> hosts
> > > only, where does CloudStack management run?
> > > 2 hosts:
> > > host1: 192.168.1.14
> > > host2: 192.168.1.5
> > >
> > > Servers are standing next to each other - connected to the same switch
> > > Management server runs on the same physical server with host1
> > >
> > > I noticed, that Virtual router gets created after I try to start any
of
> > the
> > > existing guest VM
> > > Here are logs
> > > management:
> > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > >
> > > agent on host1:
> > > https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy
> > >
> > > agent on host2:
> > > https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4
> > >
> > > But this virtual router stays in Starting state forever and hence VMs
> do
> > > not start either.
> > >
> > > On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic <andrija.pa...@gmail.com
> >
> > > wrote:
> > >
> > > > Just to confirm, you are using Basic Zone in CloudStack, right ?
> > > >
> > > > Can you confirm that router has been actually started/created on KVM
> > > side,
> > > > again, as requested please post logs (mgmt and agent - and note the
> > time
> > > > around which you tried to start VR last time it partially succeeded)
> -
> > we
> > > > can't guess what went wrong without logs.
> > > >
> > > > I would push more effort solving this one, instead of reinstalling -
> > you
> > > > might hit the issue again and then it's no good.
> > > >
> > > > Can you confirm your zone/pod/cluster/hosts are all in Enabled
state,
> > > i.e.
> > > > not disabled and hosts connected AND both SSVM and CPVM are
> > > > connectedUP/green
> > > > Is your dashboard in GUI all green - no issues there ?
> > > > Can you connect your both KVM hosts can access/mount both Primary
and
> > > > Secondary Storage
> > > >
> > > > On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > After dozen of attempts, the Virtual Router could finally be
> > recreated.
> > > > But
> > > > > its in eternal Starting status, and console prompts it required
> > upgrade
> > > > and
> > > > > Version is UNKNOWN
> > > > >
> > > > > It does not resolve the problem, I cannot move further form this
> > point.
> > > > > Any hints?
> > > > >
> > > > > Or I am condemned to do reinstall cloudstack from scratch?
> > > > >
> > > > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov <
> > > > > j.zolotar...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Under this defaultGuestNetwork, I go to Virtual Appliances.
There
> > is
> > > no
> > > > > > VMS - "no data to show"
> > > > > >
> > > > > > I dont have any network, other than this single default one.
> > > > > >
> > > > > > I've tried adding new network - Add guest network. But I am not
> > able
> > > to
> > > > > do
> > > > > > so, cause in the wizard popup, it offers empty dropdown with
> Zones
> > > > > > selection. And this wizard doesnt not allow to go further
without
> > > > > selecting
> > > > > > Zone
> > > > > >
> > > > > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic <
> > > > andrija.pa...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> you need to delete/remove all VMs inside this network (tick the
> > > > > "Expunge"
> > > > > >> button during VM deletion - if you want to really delete the
> VMs)
> > in
> > > > > order
> > > > > >> to be able to delete the network - OR simply attach this VM to
> > > another
> > > > > >> network, make this new network a DEFAULT one (NIC that is...),
> and
> > > > then
> > > > > >> detach from old network - and then effectively your VM was
> > "removed"
> > > > > from
> > > > > >> old network - after this you should be able to delete the old
> > > > network. I
> > > > > >> assume some DB incosistencies perhaps, being the reason you can
> > not
> > > > > >> restart
> > > > > >> the network.
> > > > > >>
> > > > > >> Did you try restarting some other Network - or deploying a new
> > > > network,
> > > > > >> spin a VM in it, then again try to restart this new network -
> does
> > > it
> > > > > >> work ?
> > > > > >>
> > > > > >> Andrija
> > > > > >>
> > > > > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov <
> > > > > j.zolotar...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > >>>Stop mgmt,
> > > > > >> > >>>Stop all agents
> > > > > >> > >>>Restart libvirtd (and check libvirt logs afterwards)
> > > > > >> > >>>Start agents
> > > > > >> > >>>Start mgmt.
> > > > > >> >
> > > > > >> > I did that numerous time. Nothing really suspicious
> > > > > >> > I can see that systems VMs are running - both in cloudstack
> > > console
> > > > > and
> > > > > >> > with virsh list -all
> > > > > >> >
> > > > > >> > It is apparently problem with network.
> > > > > >> > Is there a way to force recreation of defaultGuestNetwork? or
> > > force
> > > > > >> > recreation of Virtual Router.
> > > > > >> > I am unable to delete network, which is supposed to rebuild
> > > network
> > > > > with
> > > > > >> > its router. Thats the issue
> > > > > >> >
> > > > > >> > The issue with libvirtd was, that eventually at some point it
> > was
> > > > > >> updated
> > > > > >> > during 4 months of running, and not rebooted. It still
worked.
> > We
> > > > had
> > > > > to
> > > > > >> > add listen_tcp = 1 for libvirtd to start working again.
> > > > > >> >
> > > > > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic <
> > > > > andrija.pa...@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > As Sergey suggested... but i would also verify no libvirt
> > issues
> > > > or
> > > > > >> > storage
> > > > > >> > > pool issues - so perhaps:
> > > > > >> > >
> > > > > >> > > Stop mgmt,
> > > > > >> > > Stop all agents
> > > > > >> > > Restart libvirtd (and check libvirt logs afterwards)
> > > > > >> > > Start agents
> > > > > >> > > Start mgmt.
> > > > > >> > >
> > > > > >> > > What was originally issue with libvirtd ?
> > > > > >> > > That sounds fishy to me...
> > > > > >> > >
> > > > > >> > > Andrija
> > > > > >> > >
> > > > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <
> > > serg...@hotmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > > > select * from networks where removed is null;
> > > > > >> > > > select * from vm_instance where id=87;
> > > > > >> > > > select id,name from vm_instance where name like 'r%' and
> > > removed
> > > > > is
> > > > > >> > null;
> > > > > >> > > >
> > > > > >> > > > Basically since the network offering is not redundant
this
> > > error
> > > > > is
> > > > > >> > only
> > > > > >> > > > thrown when there is no router associated with your
> network.
> > > > > Usually
> > > > > >> > > > management server restart tries to implement network
> again.
> > > > Please
> > > > > >> > > restart
> > > > > >> > > > management server, save and share management server log.
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" <
> > > > > j.zolotar...@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > >>>>Check network_offering table for value in column
> > > > > >> > > > redundant_router_service for the network offering
you
> > > use.
> > > > > >> > > > in table network_offering_table all records have
> > > > > >> > > > redundant_router_service =
> > > > > >> > > > 0
> > > > > >> > > >
> > > > > >> > > > Can you also run the following:
> > > > > >> > > > >>>select name, state, removed from host where name
> > like
> > > > 'r%'
> > > > > >> > > > returns zero rows - nothing
> > > > > >> > > >
> > > > > >> > > > >>>select * from domain_router;
> > > > > >> > > > # id, element_id, public_mac_address,
> public_ip_address,
> > > > > >> > > > public_netmask,
> > > > > >> > > > guest_netmask, guest_ip_address, is_redundant_router,
> > > > > priority,
> > > > > >> > > > redundant_state, stop_pending, role,
template_version,
> > > > > >> > > scripts_version,
> > > > > >> > > > vpc_id, update_state
> > > > > >> > > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan
> 14
> > > > > 15:37:28
> > > > > >> > UTC
> > > > > >> > > > 2018',
> > > > > >> > > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL
> > > > > >> > > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan
> 14
> > > > > 15:37:28
> > > > > >> > UTC
> > > > > >> > > > 2018',
> > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > > >> > > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan
> 14
> > > > > 15:37:28
> > > > > >> > UTC
> > > > > >> > > > 2018',
> > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > > >> > > > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
> > > > > >> > > > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan
> 14
> > > > > 15:37:28
> > > > > >> > UTC
> > > > > >> > > > 2018',
> > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > > >> > > > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan
> 14
> > > > > 15:37:28
> > > > > >> > UTC
> > > > > >> > > > 2018',
> > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > > >> > > > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun
> 22
> > > > > 07:52:17
> > > > > >> > UTC
> > > > > >> > > > 2018',
> > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL,
> > > 'UPDATE_FAILED'
> > > > > >> > > > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun
> 22
> > > > > 07:52:17
> > > > > >> > UTC
> > > > > >> > > > 2018',
> > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > > >> > > > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun
> 22
> > > > > 07:52:17
> > > > > >> > UTC
> > > > > >> > > > 2018',
> > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > > >> > > > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
> > > > > >> > > > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > > 'UNKNOWN',
> > > > > >> '0',
> > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon
Nov
> > 12
> > > > > >> 15:06:49
> > > > > >> > > UTC
> > > > > >> > > > 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL,
> NULL
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > >>>select * from router_network_ref;
> > > > > >> > > > # id, router_id, network_id, guest_type
> > > > > >> > > > '1', '4', '204', 'Shared'
> > > > > >> > > > '2', '49', '204', 'Shared'
> > > > > >> > > > '3', '73', '204', 'Shared'
> > > > > >> > > > '4', '75', '204', 'Shared'
> > > > > >> > > > '5', '76', '204', 'Shared'
> > > > > >> > > > '6', '77', '204', 'Shared'
> > > > > >> > > > '7', '80', '204', 'Shared'
> > > > > >> > > > '8', '85', '204', 'Shared'
> > > > > >> > > > '9', '86', '204', 'Shared'
> > > > > >> > > > '10', '87', '204', 'Shared'
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy <
> > > > > >> > > serg...@hotmail.com
> > > > > >> > > > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Check network_offering table for value in column
> > > > > >> > > > > redundant_router_service for the network offering
> you
> > > > use.
> > > > > >> > > > > Can you also run the following:
> > > > > >> > > > > select name, state, removed from host where name
> like
> > > > 'r%'
> > > > > >> > > > > select * from domain_router;
> > > > > >> > > > > select * from router_network_ref;
> > > > > >> > > > >
> > > > > >> > > > > Cloudstack is supposed to recreate you VR. If it is
> > not
> > > > > >> happening
> > > > > >> > > > there is
> > > > > >> > > > > something fundamentally wrong. I would advise to
> > destroy
> > > > > your
> > > > > >> VR
> > > > > >> > > > again.
> > > > > >> > > > > Stop you management server. Rotate management
server
> > log
> > > > and
> > > > > >> > start
> > > > > >> > > it
> > > > > >> > > > > again. If your VR doesn't start in few min, post
> your
> > > > > complete
> > > > > >> > > > management
> > > > > >> > > > > server log and agent log again.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" <
> > > > > >> > j.zolotar...@gmail.com>
> > > > > >> > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > >>>Network offering needs to be change to
> > > > non-redundant
> > > > > >> > > > > How do I do that?
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Mar 19, 2019 at 11:47 PM Sergey
> Levitskiy
> > <
> > > > > >> > > > serg...@hotmail.com
> > > > > >> > > > > >
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Network offering needs to be change to
> > > > non-redundant.
> > > > > >> Most
> > > > > >> > > > likely
> > > > > >> > > > > old bug
> > > > > >> > > > > > is resurfaced.
> > > > > >> > > > > >
> > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <
> > > > > >> > > > j.zolotar...@gmail.com>
> > > > > >> > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > Hello
> > > > > >> > > > > >
> > > > > >> > > > > > I did exactly like you suggested.
> > > > > >> > > > > >
> > > > > >> > > > > > After UPDATE on db, I can see the router
> in
> > > > > >> cloudstack
> > > > > >> > > > console.
> > > > > >> > > > > > But attempt to restart network fails.
> > > > > >> > > > > > I get error:
> > > > > >> > > > > > Resource [DataCenter:1] is unreachable:
> > Can't
> > > > find
> > > > > >> all
> > > > > >> > > > necessary
> > > > > >> > > > > > running
> > > > > >> > > > > > routers!
> > > > > >> > > > > >
> > > > > >> > > > > > I rechecked agents on both servers. They
> > look
> > > > > >> running
> > > > > >> > OK.
> > > > > >> > > > > >
> > > > > >> > > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija
> > > Panic <
> > > > > >> > > > > > andrija.pa...@gmail.com>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Ok, so:
> > > > > >> > > > > > >
> > > > > >> > > > > > > 1.BACKUP YOUR DB - in case there is
> > issues,
> > > > you
> > > > > >> will
> > > > > >> > > > want to
> > > > > >> > > > > restore
> > > > > >> > > > > > > 2. I tried to reproduce your case on
> > 4.11.2
> > > > > >> (though
> > > > > >> > > > didn't do
> > > > > >> > > > > any
> > > > > >> > > > > > > maintenance etc.) - and could not - I
> have
> > > > > >> > artificially
> > > > > >> > > > marked
> > > > > >> > > > > > existing VR
> > > > > >> > > > > > > as removed in DB - and tried to restart
> > > > network
> > > > > >> and
> > > > > >> > it
> > > > > >> > > > worked
> > > > > >> > > > > just
> > > > > >> > > > > > fine.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Let's try to "undelete" the VR for that
> > > > network
> > > > > -
> > > > > >> > > though
> > > > > >> > > > I
> > > > > >> > > > > can't be
> > > > > >> > > > > > sure if
> > > > > >> > > > > > > this will work or not.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Find VR based on the network UUID:
> > > > > >> > > > > > > SELECT * FROM vm_instance WHERE
id
> > IN
> > > > > >> (SELECT
> > > > > >> > > > > instance_id from
> > > > > >> > > > > > nics
> > > > > >> > > > > > > where network_id IN (SELECT id FROM
> > networks
> > > > > WHERE
> > > > > >> > > > > > >
> > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> > > > > >> > > > > > > Obviously, replace this UUID with your
> > > network
> > > > > >> UUID.
> > > > > >> > > > > > > SQL above will return all VMs (user
VMs,
> > > VRs,
> > > > > etc,
> > > > > >> > > > delete or
> > > > > >> > > > > not...)
> > > > > >> > > > > > that
> > > > > >> > > > > > > have NIC inside the network with that
> > UUID -
> > > > > look
> > > > > >> for
> > > > > >> > > the
> > > > > >> > > > > r-XX-VM
> > > > > >> > > > > > being a
> > > > > >> > > > > > > router.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Now that you have the name of the VR
> (i.e.
> > > > > >> r-4-VM) -
> > > > > >> > > you
> > > > > >> > > > want
> > > > > >> > > > > to set
> > > > > >> > > > > > 2
> > > > > >> > > > > > > fields to some other value than what it
> > > might
> > > > be
> > > > > >> > > > already, as
> > > > > >> > > > > below:
> > > > > >> > > > > > >
> > > > > >> > > > > > > Need to set field "state" to Stopped:
> > > > > >> > > > > > > UPDATE cloud.vm_instance SET
> > > > > state='Stopped'
> > > > > >> > > WHERE
> > > > > >> > > > > > name="r-4-VM";
> > > > > >> > > > > > > (what state is set currently ???)
> > > > > >> > > > > > >
> > > > > >> > > > > > > Need to set "removed" to NULL value:
> > > > > >> > > > > > > UPDATE cloud.vm_instance SET
> > removed =
> > > > > NULL
> > > > > >> > WHERE
> > > > > >> > > > > > NAME="r-4-VM"; (is
> > > > > >> > > > > > > there a removed date already set ???)
> > > > > >> > > > > > >
> > > > > >> > > > > > > Obviously make sure that this VR is not
> > > > running
> > > > > on
> > > > > >> > any
> > > > > >> > > > > hypervisor
> > > > > >> > > > > > and if
> > > > > >> > > > > > > running, virsh destroy it...
> > > > > >> > > > > > >
> > > > > >> > > > > > > AFTER above has been done (VR is
> > considered
> > > to
> > > > > >> exist
> > > > > >> > > but
> > > > > >> > > > > stopped,
> > > > > >> > > > > > from
> > > > > >> > > > > > > CloudStack point of view) - try to
> RESTART
> > > the
> > > > > >> > network
> > > > > >> > > > (don't
> > > > > >> > > > > bother
> > > > > >> > > > > > > deleteing it, since there are VMs in
> that
> > > > > network)
> > > > > >> > > > > > >
> > > > > >> > > > > > > I'm not 100% positive this will fix
your
> > > > issue,
> > > > > >> but
> > > > > >> > > > doesn't
> > > > > >> > > > > hurt to
> > > > > >> > > > > > try
> > > > > >> > > > > > >
> > > > > >> > > > > > > If above doesn't work - I would still
> > take a
> > > > > look
> > > > > >> > into
> > > > > >> > > > agents
> > > > > >> > > > > and if
> > > > > >> > > > > > they
> > > > > >> > > > > > > are still connected - optionally,
> restart
> > > > agents
> > > > > >> on
> > > > > >> > > both
> > > > > >> > > > hosts
> > > > > >> > > > > once
> > > > > >> > > > > > more
> > > > > >> > > > > > > and confirm they are connected and up.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Let us know how it goes (and please
> backup
> > > DB
> > > > > once
> > > > > >> > more
> > > > > >> > > > before
> > > > > >> > > > > any
> > > > > >> > > > > > actions
> > > > > >> > > > > > > !)
> > > > > >> > > > > > >
> > > > > >> > > > > > > Andrija
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni
> > > > > Zolotarjov <
> > > > > >> > > > > > j.zolotar...@gmail.com>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Yes. Just a single network.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija
> > Panic,
> > > <
> > > > > >> > > > > andrija.pa...@gmail.com
> > > > > >> > > > > > >
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Just one more clarification - this
> is
> > > > > Isolate
> > > > > >> > > single
> > > > > >> > > > > network (not
> > > > > >> > > > > > > Shared
> > > > > >> > > > > > > > > Network, not VPC) ?
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni
> > > > > >> Zolotarjov <
> > > > > >> > > > > > j.zolotar...@gmail.com
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Name defaultGuestNetwork
> > > > > >> > > > > > > > > > ID
> > > 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > > > >> > > > > > > > > > Zone PBT zone 1
> > > > > >> > > > > > > > > > Description defaultGuestNetwork
> > > > > >> > > > > > > > > > Type Shared
> > > > > >> > > > > > > > > > State Setup
> > > > > >> > > > > > > > > > VPC ID N/A
> > > > > >> > > > > > > > > > Persistent No
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM
> > > Andrija
> > > > > >> Panic <
> > > > > >> > > > > > > andrija.pa...@gmail.com
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Share the network id and name
as
> > > seen
> > > > > from
> > > > > >> > > GUI...
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Tue, Mar 19, 2019, 19:27
> > Jevgeni
> > > > > >> > Zolotarjov
> > > > > >> > > <
> > > > > >> > > > > > > > j.zolotar...@gmail.com
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >>> 1. Confirm please rolling
> > > > restart
> > > > > is
> > > > > >> > set
> > > > > >> > > > to false
> > > > > >> > > > > > please -
> > > > > >> > > > > > > > double
> > > > > >> > > > > > > > > > > check
> > > > > >> > > > > > > > > > > > Double checked - It is set to
> > > false
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >>>>2. If so - do you know
the
> > > name
> > > > of
> > > > > >> VR
> > > > > >> > > > which you
> > > > > >> > > > > > deleted ? Is
> > > > > >> > > > > > > it
> > > > > >> > > > > > > > > > last
> > > > > >> > > > > > > > > > > > one
> > > > > >> > > > > > > > > > > > >>> ever created - if so we
> can
> > > find
> > > > > it
> > > > > >> > > > easily...
> > > > > >> > > > > > > > > > > > I dont know the name
> > > > > >> > > > > > > > > > > > Is there a way to fetch it
> from
> > > DB?
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58
> PM
> > > > > Andrija
> > > > > >> > > Panic <
> > > > > >> > > > > > > > > andrija.pa...@gmail.com
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Right...still complaining
on
> > > > missing
> > > > > >> > > running
> > > > > >> > > > > routers.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > 1. Confirm please rolling
> > > restart
> > > > is
> > > > > >> set
> > > > > >> > to
> > > > > >> > > > false
> > > > > >> > > > > please
> > > > > >> > > > > > -
> > > > > >> > > > > > > double
> > > > > >> > > > > > > > > > check
> > > > > >> > > > > > > > > > > > > 2. If so - do you know the
> > name
> > > of
> > > > > VR
> > > > > >> > which
> > > > > >> > > > you
> > > > > >> > > > > deleted
> > > > > >> > > > > > ? Is it
> > > > > >> > > > > > > > > last
> > > > > >> > > > > > > > > > > one
> > > > > >> > > > > > > > > > > > > ever created - if so we can
> > find
> > > > it
> > > > > >> > > easily...
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40
> > > > Jevgeni
> > > > > >> > > > Zolotarjov <
> > > > > >> > > > > > > > > > j.zolotar...@gmail.com
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Here is management server
> > log
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On Tue, Mar 19, 2019 at
> 7:29
> > > PM
> > > > > >> Andrija
> > > > > >> > > > Panic <
> > > > > >> > > > > > > > > > > andrija.pa...@gmail.com
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > I would also try to
> > > "undelete"
> > > > > VR
> > > > > >> in
> > > > > >> > > DB,
> > > > > >> > > > but
> > > > > >> > > > > let's
> > > > > >> > > > > > keep
> > > > > >> > > > > > > this
> > > > > >> > > > > > > > as
> > > > > >> > > > > > > > > > > last
> > > > > >> > > > > > > > > > > > > > step.
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > On Tue, Mar 19, 2019,
> > 18:24
> > > > > >> Andrija
> > > > > >> > > > Panic <
> > > > > >> > > > > > > > > > andrija.pa...@gmail.com
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > Disclaimer: from
> mobile
> > > > > device.
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > I don't see any
> special
> > > > > failure
> > > > > >> in
> > > > > >> > > > agent log,
> > > > > >> > > > > > except some
> > > > > >> > > > > > > > > long
> > > > > >> > > > > > > > > > > > > running
> > > > > >> > > > > > > > > > > > > > > > migration, timeout
for
> > > > > graceful
> > > > > >> VM
> > > > > >> > > > shutdown
> > > > > >> > > > > etc
> > > > > >> > > > > > (and
> > > > > >> > > > > > > agent
> > > > > >> > > > > > > > > > > restart)
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > You did not send mgmt
> > log
> > > > > after
> > > > > >> > last
> > > > > >> > > > network
> > > > > >> > > > > > restart
> > > > > >> > > > > > > > failure
> > > > > >> > > > > > > > > ?
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > On Tue, Mar 19, 2019,
> > > 17:29
> > > > > >> Jevgeni
> > > > > >> > > > > Zolotarjov <
> > > > > >> > > > > > > > > > > > > j.zolotar...@gmail.com
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > >> Under
> > > > > >> > > > > > > > > > > > > > > >> Infrastructure /
> Hosts:
> > > > > >> > > > > > > > > > > > > > > >> both hosts are
> enabled
> > > and
> > > > > >> green
> > > > > >> > > > (Unsecure
> > > > > >> > > > > though)
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >> agent.log -
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > >
> > > > > >> >
> > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >> I set
> > > > network.rolling.restart
> > > > > >> to
> > > > > >> > > > false now.
> > > > > >> > > > > > Restarted
> > > > > >> > > > > > > > > > management
> > > > > >> > > > > > > > > > > > > > server
> > > > > >> > > > > > > > > > > > > > > -
> > > > > >> > > > > > > > > > > > > > > >> same problem -
cannot
> > > > restart
> > > > > >> not
> > > > > >> > > > delete
> > > > > >> > > > > network
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >> On Tue, Mar 19, 2019
> at
> > > > 5:48
> > > > > PM
> > > > > >> > > Boris
> > > > > >> > > > > Stoyanov <
> > > > > >> > > > > > > > > > > > > > > >>
> > > > boris.stoya...@shapeblue.com
> > > > > >
> > > > > >> > > > > > > > > > > > > > > >> wrote:
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > > >> > Hi, you shouldn’t
> be
> > > > > worried
> > > > > >> > about
> > > > > >> > > > data
> > > > > >> > > > > as long
> > > > > >> > > > > > as you
> > > > > >> > > > > > > > > use a
> > > > > >> > > > > > > > > > > > > > separate
> > > > > >> > > > > > > > > > > > > > > >> > shared storage.
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> > What does the
> > > cloudstack
> > > > > >> console
> > > > > >> > > say
> > > > > >> > > > > about your
> > > > > >> > > > > > host?
> > > > > >> > > > > > > Is
> > > > > >> > > > > > > > > it
> > > > > >> > > > > > > > > > > up?
> > > > > >> > > > > > > > > > > > If
> > > > > >> > > > > > > > > > > > > > > it’s
> > > > > >> > > > > > > > > > > > > > > >> up
> > > > > >> > > > > > > > > > > > > > > >> > then you should be
> > able
> > > > to
> > > > > >> > deploy
> > > > > >> > > a
> > > > > >> > > > VM on
> > > > > >> > > > > it.
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> > If not, you’ll
need
> > to
> > > > > >> > investigate
> > > > > >> > > > the
> > > > > >> > > > > reason
> > > > > >> > > > > > > > > > cloudstack-agent
> > > > > >> > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > not
> > > > > >> > > > > > > > > > > > > > > >> > connected, is the
> > agent
> > > > > >> running?
> > > > > >> > > If
> > > > > >> > > > not
> > > > > >> > > > > just
> > > > > >> > > > > > start it
> > > > > >> > > > > > > > and
> > > > > >> > > > > > > > > > see
> > > > > >> > > > > > > > > > > if
> > > > > >> > > > > > > > > > > > > it
> > > > > >> > > > > > > > > > > > > > > >> > connects.
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> > If it’s running
but
> > > still
> > > > > not
> > > > > >> > > > connected
> > > > > >> > > > > please
> > > > > >> > > > > > share
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > agent.log.
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> > If nothing helps
at
> > all
> > > > you
> > > > > >> can
> > > > > >> > > > remove
> > > > > >> > > > > the host
> > > > > >> > > > > > and
> > > > > >> > > > > > > add
> > > > > >> > > > > > > > it
> > > > > >> > > > > > > > > > > > again,
> > > > > >> > > > > > > > > > > > > > but
> > > > > >> > > > > > > > > > > > > > > >> > let’s try to
> > > troubleshoot
> > > > > >> first.
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> > Bobby.
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > boris.stoya...@shapeblue.com
> > > > > >> > > > > > > > > > > > > > > >> > www.shapeblue.com
> > > > > >> > > > > > > > > > > > > > > >> > Amadeus House,
> Floral
> > > > > Street,
> > > > > >> > > London
> > > > > >> > > > > WC2E 9DPUK
> > > > > >> > > > > > > > > > > > > > > >> > @shapeblue
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >> > > On 19 Mar 2019,
> at
> > > > 17:29,
> > > > > >> > > Jevgeni
> > > > > >> > > > > Zolotarjov <
> > > > > >> > > > > > > > > > > > > > > j.zolotar...@gmail.com>
> > > > > >> > > > > > > > > > > > > > > >> > wrote:
> > > > > >> > > > > > > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > > > > > > >> > > Guys, please
help
> > > with
> > > > > it.
> > > > > >> > > > > > > > > > > > > > > >> > > What can be done
> > > here?
> > > > > >> > > > > > > > > > > > > > > >> > > There is too
much
> > > > > valuable
> > > > > >> > data.
> > > > > >> > > > > > > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > > > > > > >> > > On Tue, Mar 19,
> > 2019
> > > at
> > > > > >> 4:21
> > > > > >> > PM
> > > > > >> > > > Jevgeni
> > > > > >> > > > > > Zolotarjov <
> > > > > >> > > > > > > > > > > > > > > >> >
> > j.zolotar...@gmail.com
> > > >
> > > > > >> > > > > > > > > > > > > > > >> > > wrote:
> > > > > >> > > > > > > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > > > > > > >> > >> Tried that just
> > now
> > > > and
> > > > > >> got
> > > > > >> > > > error:
> > > > > >> > > > > > > > > > > > > > > >> > >> Resource
> > > > [DataCenter:1]
> > > > > is
> > > > > >> > > > unreachable:
> > > > > >> > > > > > Can't find
> > > > > >> > > > > > > > all
> > > > > >> > > > > > > > > > > > > necessary
> > > > > >> > > > > > > > > > > > > > > >> running
> > > > > >> > > > > > > > > > > > > > > >> > >> routers!
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> > >> In the log I
> see:
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > =============================================
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19
> > > > 14:20:39,644
> > > > > >> DEBUG
> > > > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > > >> > > > > job-5093
> > > > > >> > > > > > > > ctx-9be30648)
> > > > > >> > > > > > > > > > > > > > > >> (logid:265a6099)
> > > > > >> > > > > > > > > > > > > > > >> > >> Restarting
> network
> > > > > 204...
> > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19
> > > > 14:20:39,645
> > > > > >> DEBUG
> > > > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > > >> > > > > job-5093
> > > > > >> > > > > > > > ctx-9be30648)
> > > > > >> > > > > > > > > > > > > > > >> (logid:265a6099)
> > > > > >> > > > > > > > > > > > > > > >> > >> Performing
> rolling
> > > > > >> restart of
> > > > > >> > > > routers
> > > > > >> > > > > of
> > > > > >> > > > > > network
> > > > > >> > > > > > > > > > > > > > Ntwk[204|Guest|6]
> > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19
> > > > 14:20:39,649
> > > > > >> DEBUG
> > > > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > > >> > > > > job-5093
> > > > > >> > > > > > > > ctx-9be30648)
> > > > > >> > > > > > > > > > > > > > > >> (logid:265a6099)
> > > > > >> > > > > > > > > > > > > > > >> > >> Asking
> > VirtualRouter
> > > > to
> > > > > >> > > > implemenet
> > > > > >> > > > > > > Ntwk[204|Guest|6]
> > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19
> > > > 14:20:39,658
> > > > > >> ERROR
> > > > > >> > > > > > > > > > [c.c.a.ApiAsyncJobDispatcher]
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > > >> > > > > job-5093)
> > > > > >> > > > > > > > > > (logid:265a6099)
> > > > > >> > > > > > > > > > > > > > > >> Unexpected
> > > > > >> > > > > > > > > > > > > > > >> > >> exception while
> > > > > executing
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > >
> > > > > >>
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > >
> > com.cloud.exception.ResourceUnavailableException:
> > > > > >> > > > > > > > > > Resource
> > > > > >> > > > > > > > > > > > > > > >> > [DataCenter:1]
> > > > > >> > > > > > > > > > > > > > > >> > >> is unreachable:
> > > Can't
> > > > > find
> > > > > >> > all
> > > > > >> > > > > necessary
> > > > > >> > > > > > running
> > > > > >> > > > > > > > > routers!
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > >
> > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > >> > > > > > > > > > > > > > > Method)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > >
> > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > > > > >> > > > > > > > > > > > > > > Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > > >> > > > > > > > > > > > > > > >> > Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > java.lang.reflect.Method.invoke(Unknown
> > > > > >> > > > > > > > > Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > >
> > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown
> > > > > >> > > > > > > > > > > > > Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > >
> > > > > >> > > >
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > > > > >> > > > > > > > > > > > > > > >> > >> Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > >
> > java.util.concurrent.FutureTask.run(Unknown
> > > > > >> > > > > > > > > > > Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > > > > >> > > > > > > > > > > > > > > >> > >> Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > >
> > > > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > > > > >> > > > > > > > > > > > > > > >> > >> Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> at
> > > > > >> > > > java.lang.Thread.run(Unknown
> > > > > >> > > > > > Source)
> > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19
> > > > 14:20:39,659
> > > > > >> DEBUG
> > > > > >> > > > > > > > > > > >
> > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > > >> > > > > job-5093)
> > > > > >> > > > > > > > > > (logid:265a6099)
> > > > > >> > > > > > > > > > > > > > > Complete
> > > > > >> > > > > > > > > > > > > > > >> > async
> > > > > >> > > > > > > > > > > > > > > >> > >> job-5093,
> > jobStatus:
> > > > > >> FAILED,
> > > > > >> > > > > resultCode: 530,
> > > > > >> > > > > > > result:
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > >> > > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > > > > >> > > > > > > > > > > > > > > >> > >> [DataCenter:1]
> is
> > un
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> > >> On Tue, Mar 19,
> > 2019
> > > > at
> > > > > >> 4:19
> > > > > >> > PM
> > > > > >> > > > Andrija
> > > > > >> > > > > > Panic <
> > > > > >> > > > > > > > > > > > > > > >> >
> > > > > andrija.pa...@shapeblue.com>
> > > > > >> > > > > > > > > > > > > > > >> > >> wrote:
> > > > > >> > > > > > > > > > > > > > > >> > >>
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>> Your network
> > can't
> > > be
> > > > > >> > deleted
> > > > > >> > > > due to
> > > > > >> > > > > "Can't
> > > > > >> > > > > > delete
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > > network,
> > > > > >> > > > > > > > > > > > > > > not
> > > > > >> > > > > > > > > > > > > > > >> all
> > > > > >> > > > > > > > > > > > > > > >> > >>> user vms are
> > > > expunged.
> > > > > Vm
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > VM[User|i-2-11-VM]
> > > is
> > > > > in
> > > > > >> > > Stopped
> > > > > >> > > > > state" -
> > > > > >> > > > > > which is
> > > > > >> > > > > > > > > fine.
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>> You should be
> > able
> > > to
> > > > > >> just
> > > > > >> > > > start the
> > > > > >> > > > > user
> > > > > >> > > > > > VM - but
> > > > > >> > > > > > > > if
> > > > > >> > > > > > > > > > you
> > > > > >> > > > > > > > > > > > have
> > > > > >> > > > > > > > > > > > > > > >> actually
> > > > > >> > > > > > > > > > > > > > > >> > >>> delete the VR
> > > itself,
> > > > > >> then
> > > > > >> > > just
> > > > > >> > > > do
> > > > > >> > > > > Network
> > > > > >> > > > > > restart
> > > > > >> > > > > > > > > with
> > > > > >> > > > > > > > > > > > > > "cleanup"
> > > > > >> > > > > > > > > > > > > > > >> and
> > > > > >> > > > > > > > > > > > > > > >> > it
> > > > > >> > > > > > > > > > > > > > > >> > >>> will recreate
a
> > new
> > > > VR,
> > > > > >> > after
> > > > > >> > > > which
> > > > > >> > > > > you
> > > > > >> > > > > > should be
> > > > > >> > > > > > > > able
> > > > > >> > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > start
> > > > > >> > > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > > >> VM.
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>> Andrija
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> andrija.pa...@shapeblue.com
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > www.shapeblue.com
> > > > > >> > > > > > > > > > > > > > > >> > >>> Amadeus House,
> > > Floral
> > > > > >> > Street,
> > > > > >> > > > London
> > > > > >> > > > > WC2E
> > > > > >> > > > > > 9DPUK
> > > > > >> > > > > > > > > > > > > > > >> > >>> @shapeblue
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>> -----Original
> > > > > >> Message-----
> > > > > >> > > > > > > > > > > > > > > >> > >>> From: Jevgeni
> > > > > Zolotarjov
> > > > > >> <
> > > > > >> > > > > > j.zolotar...@gmail.com>
> > > > > >> > > > > > > > > > > > > > > >> > >>> Sent: 19 March
> > 2019
> > > > > 15:10
> > > > > >> > > > > > > > > > > > > > > >> > >>> To:
> > > > > >> > > users@cloudstack.apache.org
> > > > > >> > > > > > > > > > > > > > > >> > >>> Subject: Re:
> > > Disaster
> > > > > >> after
> > > > > >> > > > > maintenance
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>> I mean I
cannot
> > > > delete
> > > > > >> > > network:
> > > > > >> > > > In the
> > > > > >> > > > > > management
> > > > > >> > > > > > > > > server
> > > > > >> > > > > > > > > > > > log I
> > > > > >> > > > > > > > > > > > > > see
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > > > > > > > > >> > >>>
> > > > > >> > > > > > > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >>
> > > > > >> Andrija Panić
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrija Panić
> > > >
> > >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
--
Andrija Panić
