Re: Disaster after maintenance
+1 on the advice to start from scratch. Provisioning is failing because it can’t spin up either SSVM or proxy due to not enough capacity. The reason might be: * Not enough capacity either CPU or RAM. increasing overprovisioning factors or reducing disable thresholds might help. * Hosts in error state * Cluster disabled * Problem accessing primary and/or secondary storage mount from management server host 2019-03-20 15:07:39,218 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264) (logid:49483c7a) Trying to allocate a host and storage pools from dc:3, pod:null,cluster:null, requested cpu: 500, requested ram: 536870912 2019-03-20 15:07:39,218 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264) (logid:49483c7a) Is ROOT volume READY (pool already allocated)?: No 2019-03-20 15:07:39,219 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264) (logid:49483c7a) Deploy avoids pods: null, clusters: null, hosts: null 2019-03-20 15:07:39,219 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264) (logid:49483c7a) Searching all possible resources under this Zone: 3 2019-03-20 15:07:39,219 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-38:ctx-f824bfeb job-5119/job-7076 ctx-9498) (logid:bc39cd2a) No clusters found having a host with enough capacity, returning. 2019-03-20 15:07:39,219 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264) (logid:49483c7a) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 3 2019-03-20 15:07:39,221 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-37:ctx-3cad2de4 job-5120/job-7077 ctx-6b705264) (logid:49483c7a) No clusters found having a host with enough capacity, returning. On 3/20/19, 10:38 AM, "Andrija Panic" wrote: Hi Jevgeni, I would perhaps consider you continue with plan B from your separate email thread (root volumes --> create snapshots, convert snaps to template, download template somewhere safe - for DATA volumes, also create snapshots, then convert to volume and download it (or simply directly download existing DATA volume if VM is stopped). Once you are safe, and all templates, and VM volumes are safe, you are good to reinstall. Seriously, I'm not sure how to proceed via ML - if this was my own setup, probably would be able to fix it... In next installment, start with clean 4.11.2 (4.10 was never released as an official release and was SERIOUSLY broken), or even 4.12 which has just been released (will be in 1-2 days). In this new installment, please dedicate a VM (or physical server) that will host mgmt+DB+NFS (or even better separate NFS on different server etc.) - but certainly do NOT collocate management components with KVM role. When you build zone successfully, you can then import all templates and upload all volumes (which you saved previously to some external place, web server) This will allow you to restore your VMs - possibly with just different IPs versus original ones. Considering failed DB upgrades and issues you see now, I assume your env, might be severely broken at this point, and warrants starting from scratch... Hope that makes sense - so again, download all root and data volumes to safe place (consider some petrol + matches fun) and then reinstall with a fresh and shiny infra. Alternatively, I would try to wipe all new zones (this takes some time and certain steps) and then continue troubleshooting with failed-to-start VRs. Cheers On Wed, 20 Mar 2019 at 17:59, Jevgeni Zolotarjov wrote: > It started with 4.10 and then gradually upgraded with all stops, when new > releases were available. > > > >>> Why do you have 3 zones in this installation - what is the setup ? > >>> SSVM and CPVM (for whatever zone) are failing to be created... > Its a result of attempts to create new zone and somehow move VMs to this > new zone. These all are unsuccessful attempts. > Before problem started there was 1 zone and There should be just 1 zone in > reality. > > > >>> yes, the VR can't be started, it get's timeout - in AGENT logs, I see > that > >>> it attemps to create a volume on primary storage... > I guess this is the root cause. I checked, and primary storage is > accessible via NFS share on both hosts. How to troubleshoot it? > > > On Wed, Mar 20, 2019 at 6:29 PM Andrija Panic > wrote: > > > Hi, > > > > 2019-03-20 06:41:50,446 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) > >
Re: Disaster after maintenance
Hi Jevgeni, I would perhaps consider you continue with plan B from your separate email thread (root volumes --> create snapshots, convert snaps to template, download template somewhere safe - for DATA volumes, also create snapshots, then convert to volume and download it (or simply directly download existing DATA volume if VM is stopped). Once you are safe, and all templates, and VM volumes are safe, you are good to reinstall. Seriously, I'm not sure how to proceed via ML - if this was my own setup, probably would be able to fix it... In next installment, start with clean 4.11.2 (4.10 was never released as an official release and was SERIOUSLY broken), or even 4.12 which has just been released (will be in 1-2 days). In this new installment, please dedicate a VM (or physical server) that will host mgmt+DB+NFS (or even better separate NFS on different server etc.) - but certainly do NOT collocate management components with KVM role. When you build zone successfully, you can then import all templates and upload all volumes (which you saved previously to some external place, web server) This will allow you to restore your VMs - possibly with just different IPs versus original ones. Considering failed DB upgrades and issues you see now, I assume your env, might be severely broken at this point, and warrants starting from scratch... Hope that makes sense - so again, download all root and data volumes to safe place (consider some petrol + matches fun) and then reinstall with a fresh and shiny infra. Alternatively, I would try to wipe all new zones (this takes some time and certain steps) and then continue troubleshooting with failed-to-start VRs. Cheers On Wed, 20 Mar 2019 at 17:59, Jevgeni Zolotarjov wrote: > It started with 4.10 and then gradually upgraded with all stops, when new > releases were available. > > > >>> Why do you have 3 zones in this installation - what is the setup ? > >>> SSVM and CPVM (for whatever zone) are failing to be created... > Its a result of attempts to create new zone and somehow move VMs to this > new zone. These all are unsuccessful attempts. > Before problem started there was 1 zone and There should be just 1 zone in > reality. > > > >>> yes, the VR can't be started, it get's timeout - in AGENT logs, I see > that > >>> it attemps to create a volume on primary storage... > I guess this is the root cause. I checked, and primary storage is > accessible via NFS share on both hosts. How to troubleshoot it? > > > On Wed, Mar 20, 2019 at 6:29 PM Andrija Panic > wrote: > > > Hi, > > > > 2019-03-20 06:41:50,446 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) > > (logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0 > > 2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) > > (logid:) Running upgrade Upgrade41000to41100 to upgrade from > > 4.10.0.0-4.11.0.0 to 4.11.0.0 > > fails due to > > java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to > > './cloud/#sql2-2f01-13d' (errno: 152) > > > > Then later... > > > > com.cloud.exception.InsufficientServerCapacityException: Unable to > create a > > deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface > > com.cloud.dc.DataCenter; id=3 > > com.cloud.exception.InsufficientServerCapacityException: Unable to > create a > > deployment for VM[ConsoleProxy|v-733-VM]Scope=interface > > com.cloud.dc.DataCenter; id=3 > > > > 2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch > > secondary storage VM > > 2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to > launch > > secondary storage VM yet > > 2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch > > secondary storage VM > > > > so did you start with clean 4.11.2 install, or was it upgraded one - I > see > > in logs an upgrade from DB version 4.10 to 4.11 was tried and failed... > > Why do you have 3 zones in this installation - what is the setup ? > > SSVM and CPVM (for whatever zone) are failing to be created... > > > > yes, the VR can't be started, it get's timeout - in AGENT logs, I see > that > > it attemps to create a volume on primary storage... > > > > > > Also, for SSVM I got this one... > > 2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner] > > (Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7) > > (logid:49483c7a) No clusters found having a host with enough capacity, > > returning. > > > > Andrija > > > > On Wed, 20 Mar 2019 at 16:39, Jevgeni Zolotarjov > > > wrote: > > > > > Basic Zone - Yes > > > > > > router has been actually started/created on KVM side - not created, not > > > started. Thats the main problem, I guess > > > > > > agent.log > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > > > management log > > >
Re: Disaster after maintenance
It started with 4.10 and then gradually upgraded with all stops, when new releases were available. >>> Why do you have 3 zones in this installation - what is the setup ? >>> SSVM and CPVM (for whatever zone) are failing to be created... Its a result of attempts to create new zone and somehow move VMs to this new zone. These all are unsuccessful attempts. Before problem started there was 1 zone and There should be just 1 zone in reality. >>> yes, the VR can't be started, it get's timeout - in AGENT logs, I see that >>> it attemps to create a volume on primary storage... I guess this is the root cause. I checked, and primary storage is accessible via NFS share on both hosts. How to troubleshoot it? On Wed, Mar 20, 2019 at 6:29 PM Andrija Panic wrote: > Hi, > > 2019-03-20 06:41:50,446 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) > (logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0 > 2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) > (logid:) Running upgrade Upgrade41000to41100 to upgrade from > 4.10.0.0-4.11.0.0 to 4.11.0.0 > fails due to > java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to > './cloud/#sql2-2f01-13d' (errno: 152) > > Then later... > > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface > com.cloud.dc.DataCenter; id=3 > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[ConsoleProxy|v-733-VM]Scope=interface > com.cloud.dc.DataCenter; id=3 > > 2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch > secondary storage VM > 2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to launch > secondary storage VM yet > 2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch > secondary storage VM > > so did you start with clean 4.11.2 install, or was it upgraded one - I see > in logs an upgrade from DB version 4.10 to 4.11 was tried and failed... > Why do you have 3 zones in this installation - what is the setup ? > SSVM and CPVM (for whatever zone) are failing to be created... > > yes, the VR can't be started, it get's timeout - in AGENT logs, I see that > it attemps to create a volume on primary storage... > > > Also, for SSVM I got this one... > 2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner] > (Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7) > (logid:49483c7a) No clusters found having a host with enough capacity, > returning. > > Andrija > > On Wed, 20 Mar 2019 at 16:39, Jevgeni Zolotarjov > wrote: > > > Basic Zone - Yes > > > > router has been actually started/created on KVM side - not created, not > > started. Thats the main problem, I guess > > > > agent.log > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > management log > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > >> Can you confirm your zone/pod/cluster/hosts are all in Enabled state, > > i.e. > > YES, all green > > > > >> Can you connect your both KVM hosts can access/mount both Primary and > > Secondary Storage > > YES. Double checked > > > > >>>Can you also explain your infrastructure - you said you have two hosts > > only, where does CloudStack management run? > > 2 hosts: > > host1: 192.168.1.14 > > host2: 192.168.1.5 > > > > Servers are standing next to each other - connected to the same switch > > Management server runs on the same physical server with host1 > > > > I noticed, that Virtual router gets created after I try to start any of > the > > existing guest VM > > Here are logs > > management: > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > agent on host1: > > https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy > > > > agent on host2: > > https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4 > > > > But this virtual router stays in Starting state forever and hence VMs do > > not start either. > > > > On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic > > wrote: > > > > > Just to confirm, you are using Basic Zone in CloudStack, right ? > > > > > > Can you confirm that router has been actually started/created on KVM > > side, > > > again, as requested please post logs (mgmt and agent - and note the > time > > > around which you tried to start VR last time it partially succeeded) - > we > > > can't guess what went wrong without logs. > > > > > > I would push more effort solving this one, instead of reinstalling - > you > > > might hit the issue again and then it's no good. > > > > > > Can you confirm your zone/pod/cluster/hosts are all in Enabled state, > > i.e. > > > not disabled and hosts connected AND both SSVM and CPVM are > > > connectedUP/green > > >
Re: Disaster after maintenance
Hi, 2019-03-20 06:41:50,446 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0 2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Running upgrade Upgrade41000to41100 to upgrade from 4.10.0.0-4.11.0.0 to 4.11.0.0 fails due to java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to './cloud/#sql2-2f01-13d' (errno: 152) Then later... com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface com.cloud.dc.DataCenter; id=3 com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-733-VM]Scope=interface com.cloud.dc.DataCenter; id=3 2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch secondary storage VM 2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to launch secondary storage VM yet 2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch secondary storage VM so did you start with clean 4.11.2 install, or was it upgraded one - I see in logs an upgrade from DB version 4.10 to 4.11 was tried and failed... Why do you have 3 zones in this installation - what is the setup ? SSVM and CPVM (for whatever zone) are failing to be created... yes, the VR can't be started, it get's timeout - in AGENT logs, I see that it attemps to create a volume on primary storage... Also, for SSVM I got this one... 2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7) (logid:49483c7a) No clusters found having a host with enough capacity, returning. Andrija On Wed, 20 Mar 2019 at 16:39, Jevgeni Zolotarjov wrote: > Basic Zone - Yes > > router has been actually started/created on KVM side - not created, not > started. Thats the main problem, I guess > > agent.log > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > management log > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > >> Can you confirm your zone/pod/cluster/hosts are all in Enabled state, > i.e. > YES, all green > > >> Can you connect your both KVM hosts can access/mount both Primary and > Secondary Storage > YES. Double checked > > >>>Can you also explain your infrastructure - you said you have two hosts > only, where does CloudStack management run? > 2 hosts: > host1: 192.168.1.14 > host2: 192.168.1.5 > > Servers are standing next to each other - connected to the same switch > Management server runs on the same physical server with host1 > > I noticed, that Virtual router gets created after I try to start any of the > existing guest VM > Here are logs > management: > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > agent on host1: > https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy > > agent on host2: > https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4 > > But this virtual router stays in Starting state forever and hence VMs do > not start either. > > On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic > wrote: > > > Just to confirm, you are using Basic Zone in CloudStack, right ? > > > > Can you confirm that router has been actually started/created on KVM > side, > > again, as requested please post logs (mgmt and agent - and note the time > > around which you tried to start VR last time it partially succeeded) - we > > can't guess what went wrong without logs. > > > > I would push more effort solving this one, instead of reinstalling - you > > might hit the issue again and then it's no good. > > > > Can you confirm your zone/pod/cluster/hosts are all in Enabled state, > i.e. > > not disabled and hosts connected AND both SSVM and CPVM are > > connectedUP/green > > Is your dashboard in GUI all green - no issues there ? > > Can you connect your both KVM hosts can access/mount both Primary and > > Secondary Storage > > > > On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov > > > wrote: > > > > > After dozen of attempts, the Virtual Router could finally be recreated. > > But > > > its in eternal Starting status, and console prompts it required upgrade > > and > > > Version is UNKNOWN > > > > > > It does not resolve the problem, I cannot move further form this point. > > > Any hints? > > > > > > Or I am condemned to do reinstall cloudstack from scratch? > > > > > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov < > > > j.zolotar...@gmail.com> > > > wrote: > > > > > > > Under this defaultGuestNetwork, I go to Virtual Appliances. There is > no > > > > VMS - "no data to show" > > > > > > > > I dont have any network, other than this single default one. > > > > > > > > I've tried adding new network - Add guest network. But I am not able >
Re: Disaster after maintenance
Basic Zone - Yes router has been actually started/created on KVM side - not created, not started. Thats the main problem, I guess agent.log https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- management log https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A >> Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e. YES, all green >> Can you connect your both KVM hosts can access/mount both Primary and Secondary Storage YES. Double checked >>>Can you also explain your infrastructure - you said you have two hosts only, where does CloudStack management run? 2 hosts: host1: 192.168.1.14 host2: 192.168.1.5 Servers are standing next to each other - connected to the same switch Management server runs on the same physical server with host1 I noticed, that Virtual router gets created after I try to start any of the existing guest VM Here are logs management: https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A agent on host1: https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy agent on host2: https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4 But this virtual router stays in Starting state forever and hence VMs do not start either. On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic wrote: > Just to confirm, you are using Basic Zone in CloudStack, right ? > > Can you confirm that router has been actually started/created on KVM side, > again, as requested please post logs (mgmt and agent - and note the time > around which you tried to start VR last time it partially succeeded) - we > can't guess what went wrong without logs. > > I would push more effort solving this one, instead of reinstalling - you > might hit the issue again and then it's no good. > > Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e. > not disabled and hosts connected AND both SSVM and CPVM are > connectedUP/green > Is your dashboard in GUI all green - no issues there ? > Can you connect your both KVM hosts can access/mount both Primary and > Secondary Storage > > On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov > wrote: > > > After dozen of attempts, the Virtual Router could finally be recreated. > But > > its in eternal Starting status, and console prompts it required upgrade > and > > Version is UNKNOWN > > > > It does not resolve the problem, I cannot move further form this point. > > Any hints? > > > > Or I am condemned to do reinstall cloudstack from scratch? > > > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov < > > j.zolotar...@gmail.com> > > wrote: > > > > > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no > > > VMS - "no data to show" > > > > > > I dont have any network, other than this single default one. > > > > > > I've tried adding new network - Add guest network. But I am not able to > > do > > > so, cause in the wizard popup, it offers empty dropdown with Zones > > > selection. And this wizard doesnt not allow to go further without > > selecting > > > Zone > > > > > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic < > andrija.pa...@gmail.com> > > > wrote: > > > > > >> you need to delete/remove all VMs inside this network (tick the > > "Expunge" > > >> button during VM deletion - if you want to really delete the VMs) in > > order > > >> to be able to delete the network - OR simply attach this VM to another > > >> network, make this new network a DEFAULT one (NIC that is...), and > then > > >> detach from old network - and then effectively your VM was "removed" > > from > > >> old network - after this you should be able to delete the old > network. I > > >> assume some DB incosistencies perhaps, being the reason you can not > > >> restart > > >> the network. > > >> > > >> Did you try restarting some other Network - or deploying a new > network, > > >> spin a VM in it, then again try to restart this new network - does it > > >> work ? > > >> > > >> Andrija > > >> > > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov < > > j.zolotar...@gmail.com> > > >> wrote: > > >> > > >> > >>>Stop mgmt, > > >> > >>>Stop all agents > > >> > >>>Restart libvirtd (and check libvirt logs afterwards) > > >> > >>>Start agents > > >> > >>>Start mgmt. > > >> > > > >> > I did that numerous time. Nothing really suspicious > > >> > I can see that systems VMs are running - both in cloudstack console > > and > > >> > with virsh list -all > > >> > > > >> > It is apparently problem with network. > > >> > Is there a way to force recreation of defaultGuestNetwork? or force > > >> > recreation of Virtual Router. > > >> > I am unable to delete network, which is supposed to rebuild network > > with > > >> > its router. Thats the issue > > >> > > > >> > The issue with libvirtd was, that eventually at some point it was > > >> updated > > >> > during 4 months of running, and not rebooted. It still worked. We > had > > to > > >> > add listen_tcp = 1 for libvirtd to start working again. > > >> > > > >> > On Wed,
Re: Disaster after maintenance
Jevgeni, Can you also explain your infrastructure - you said you have two hosts only, where does CloudStack management run? Reason I'm asking is when checking your logs from yesterday the IP address 192.168.1.14 seems to be used for management, NFS and a KVM host? Is this the case, do you co-host everything on the same server? Regards, Dag Sonstebo Cloud Architect ShapeBlue On 20/03/2019, 12:49, "Andrija Panic" wrote: Just to confirm, you are using Basic Zone in CloudStack, right ? Can you confirm that router has been actually started/created on KVM side, again, as requested please post logs (mgmt and agent - and note the time around which you tried to start VR last time it partially succeeded) - we can't guess what went wrong without logs. I would push more effort solving this one, instead of reinstalling - you might hit the issue again and then it's no good. Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e. not disabled and hosts connected AND both SSVM and CPVM are connectedUP/green Is your dashboard in GUI all green - no issues there ? Can you connect your both KVM hosts can access/mount both Primary and Secondary Storage On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov wrote: > After dozen of attempts, the Virtual Router could finally be recreated. But > its in eternal Starting status, and console prompts it required upgrade and > Version is UNKNOWN > > It does not resolve the problem, I cannot move further form this point. > Any hints? > > Or I am condemned to do reinstall cloudstack from scratch? > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > wrote: > > > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no > > VMS - "no data to show" > > > > I dont have any network, other than this single default one. > > > > I've tried adding new network - Add guest network. But I am not able to > do > > so, cause in the wizard popup, it offers empty dropdown with Zones > > selection. And this wizard doesnt not allow to go further without > selecting > > Zone > > > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic > > wrote: > > > >> you need to delete/remove all VMs inside this network (tick the > "Expunge" > >> button during VM deletion - if you want to really delete the VMs) in > order > >> to be able to delete the network - OR simply attach this VM to another > >> network, make this new network a DEFAULT one (NIC that is...), and then > >> detach from old network - and then effectively your VM was "removed" > from > >> old network - after this you should be able to delete the old network. I > >> assume some DB incosistencies perhaps, being the reason you can not > >> restart > >> the network. > >> > >> Did you try restarting some other Network - or deploying a new network, > >> spin a VM in it, then again try to restart this new network - does it > >> work ? > >> > >> Andrija > >> > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > >> wrote: > >> > >> > >>>Stop mgmt, > >> > >>>Stop all agents > >> > >>>Restart libvirtd (and check libvirt logs afterwards) > >> > >>>Start agents > >> > >>>Start mgmt. > >> > > >> > I did that numerous time. Nothing really suspicious > >> > I can see that systems VMs are running - both in cloudstack console > and > >> > with virsh list -all > >> > > >> > It is apparently problem with network. > >> > Is there a way to force recreation of defaultGuestNetwork? or force > >> > recreation of Virtual Router. > >> > I am unable to delete network, which is supposed to rebuild network > with > >> > its router. Thats the issue > >> > > >> > The issue with libvirtd was, that eventually at some point it was > >> updated > >> > during 4 months of running, and not rebooted. It still worked. We had > to > >> > add listen_tcp = 1 for libvirtd to start working again. > >> > > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic < > andrija.pa...@gmail.com> > >> > wrote: > >> > > >> > > As Sergey suggested... but i would also verify no libvirt issues or > >> > storage > >> > > pool issues - so perhaps: > >> > > > >> > > Stop mgmt, > >> > > Stop all agents > >> > > Restart libvirtd (and check libvirt logs afterwards) > >> > > Start agents > >> > > Start mgmt. > >> > > > >> > > What was originally issue with libvirtd ? > >> > > That sounds fishy to me... > >> > > > >> > > Andrija > >> > > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy > >> > wrote: > >> > > > >> > > > select * from networks where
Re: Disaster after maintenance
Just to confirm, you are using Basic Zone in CloudStack, right ? Can you confirm that router has been actually started/created on KVM side, again, as requested please post logs (mgmt and agent - and note the time around which you tried to start VR last time it partially succeeded) - we can't guess what went wrong without logs. I would push more effort solving this one, instead of reinstalling - you might hit the issue again and then it's no good. Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e. not disabled and hosts connected AND both SSVM and CPVM are connectedUP/green Is your dashboard in GUI all green - no issues there ? Can you connect your both KVM hosts can access/mount both Primary and Secondary Storage On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov wrote: > After dozen of attempts, the Virtual Router could finally be recreated. But > its in eternal Starting status, and console prompts it required upgrade and > Version is UNKNOWN > > It does not resolve the problem, I cannot move further form this point. > Any hints? > > Or I am condemned to do reinstall cloudstack from scratch? > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > wrote: > > > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no > > VMS - "no data to show" > > > > I dont have any network, other than this single default one. > > > > I've tried adding new network - Add guest network. But I am not able to > do > > so, cause in the wizard popup, it offers empty dropdown with Zones > > selection. And this wizard doesnt not allow to go further without > selecting > > Zone > > > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic > > wrote: > > > >> you need to delete/remove all VMs inside this network (tick the > "Expunge" > >> button during VM deletion - if you want to really delete the VMs) in > order > >> to be able to delete the network - OR simply attach this VM to another > >> network, make this new network a DEFAULT one (NIC that is...), and then > >> detach from old network - and then effectively your VM was "removed" > from > >> old network - after this you should be able to delete the old network. I > >> assume some DB incosistencies perhaps, being the reason you can not > >> restart > >> the network. > >> > >> Did you try restarting some other Network - or deploying a new network, > >> spin a VM in it, then again try to restart this new network - does it > >> work ? > >> > >> Andrija > >> > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > >> wrote: > >> > >> > >>>Stop mgmt, > >> > >>>Stop all agents > >> > >>>Restart libvirtd (and check libvirt logs afterwards) > >> > >>>Start agents > >> > >>>Start mgmt. > >> > > >> > I did that numerous time. Nothing really suspicious > >> > I can see that systems VMs are running - both in cloudstack console > and > >> > with virsh list -all > >> > > >> > It is apparently problem with network. > >> > Is there a way to force recreation of defaultGuestNetwork? or force > >> > recreation of Virtual Router. > >> > I am unable to delete network, which is supposed to rebuild network > with > >> > its router. Thats the issue > >> > > >> > The issue with libvirtd was, that eventually at some point it was > >> updated > >> > during 4 months of running, and not rebooted. It still worked. We had > to > >> > add listen_tcp = 1 for libvirtd to start working again. > >> > > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic < > andrija.pa...@gmail.com> > >> > wrote: > >> > > >> > > As Sergey suggested... but i would also verify no libvirt issues or > >> > storage > >> > > pool issues - so perhaps: > >> > > > >> > > Stop mgmt, > >> > > Stop all agents > >> > > Restart libvirtd (and check libvirt logs afterwards) > >> > > Start agents > >> > > Start mgmt. > >> > > > >> > > What was originally issue with libvirtd ? > >> > > That sounds fishy to me... > >> > > > >> > > Andrija > >> > > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy > >> > wrote: > >> > > > >> > > > select * from networks where removed is null; > >> > > > select * from vm_instance where id=87; > >> > > > select id,name from vm_instance where name like 'r%' and removed > is > >> > null; > >> > > > > >> > > > Basically since the network offering is not redundant this error > is > >> > only > >> > > > thrown when there is no router associated with your network. > Usually > >> > > > management server restart tries to implement network again. Please > >> > > restart > >> > > > management server, save and share management server log. > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" < > j.zolotar...@gmail.com> > >> > > wrote: > >> > > > > >> > > > Check network_offering table for value in column > >> > > > redundant_router_service for the network offering you use. > >> > > > in table network_offering_table all records have > >> > > > redundant_router_service =
Re: Disaster after maintenance
After dozen of attempts, the Virtual Router could finally be recreated. But its in eternal Starting status, and console prompts it required upgrade and Version is UNKNOWN It does not resolve the problem, I cannot move further form this point. Any hints? Or I am condemned to do reinstall cloudstack from scratch? On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov wrote: > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no > VMS - "no data to show" > > I dont have any network, other than this single default one. > > I've tried adding new network - Add guest network. But I am not able to do > so, cause in the wizard popup, it offers empty dropdown with Zones > selection. And this wizard doesnt not allow to go further without selecting > Zone > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic > wrote: > >> you need to delete/remove all VMs inside this network (tick the "Expunge" >> button during VM deletion - if you want to really delete the VMs) in order >> to be able to delete the network - OR simply attach this VM to another >> network, make this new network a DEFAULT one (NIC that is...), and then >> detach from old network - and then effectively your VM was "removed" from >> old network - after this you should be able to delete the old network. I >> assume some DB incosistencies perhaps, being the reason you can not >> restart >> the network. >> >> Did you try restarting some other Network - or deploying a new network, >> spin a VM in it, then again try to restart this new network - does it >> work ? >> >> Andrija >> >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov >> wrote: >> >> > >>>Stop mgmt, >> > >>>Stop all agents >> > >>>Restart libvirtd (and check libvirt logs afterwards) >> > >>>Start agents >> > >>>Start mgmt. >> > >> > I did that numerous time. Nothing really suspicious >> > I can see that systems VMs are running - both in cloudstack console and >> > with virsh list -all >> > >> > It is apparently problem with network. >> > Is there a way to force recreation of defaultGuestNetwork? or force >> > recreation of Virtual Router. >> > I am unable to delete network, which is supposed to rebuild network with >> > its router. Thats the issue >> > >> > The issue with libvirtd was, that eventually at some point it was >> updated >> > during 4 months of running, and not rebooted. It still worked. We had to >> > add listen_tcp = 1 for libvirtd to start working again. >> > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic >> > wrote: >> > >> > > As Sergey suggested... but i would also verify no libvirt issues or >> > storage >> > > pool issues - so perhaps: >> > > >> > > Stop mgmt, >> > > Stop all agents >> > > Restart libvirtd (and check libvirt logs afterwards) >> > > Start agents >> > > Start mgmt. >> > > >> > > What was originally issue with libvirtd ? >> > > That sounds fishy to me... >> > > >> > > Andrija >> > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy >> > wrote: >> > > >> > > > select * from networks where removed is null; >> > > > select * from vm_instance where id=87; >> > > > select id,name from vm_instance where name like 'r%' and removed is >> > null; >> > > > >> > > > Basically since the network offering is not redundant this error is >> > only >> > > > thrown when there is no router associated with your network. Usually >> > > > management server restart tries to implement network again. Please >> > > restart >> > > > management server, save and share management server log. >> > > > >> > > > >> > > > >> > > > >> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" >> > > wrote: >> > > > >> > > > Check network_offering table for value in column >> > > > redundant_router_service for the network offering you use. >> > > > in table network_offering_table all records have >> > > > redundant_router_service = >> > > > 0 >> > > > >> > > > Can you also run the following: >> > > > >>>select name, state, removed from host where name like 'r%' >> > > > returns zero rows - nothing >> > > > >> > > > >>>select * from domain_router; >> > > > # id, element_id, public_mac_address, public_ip_address, >> > > > public_netmask, >> > > > guest_netmask, guest_ip_address, is_redundant_router, priority, >> > > > redundant_state, stop_pending, role, template_version, >> > > scripts_version, >> > > > vpc_id, update_state >> > > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', >> '0', >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 >> > UTC >> > > > 2018', >> > > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL >> > > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', >> '0', >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 >> > UTC >> > > > 2018', >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL >> > > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', >> '0', >> > > > 'VIRTUAL_ROUTER',
Re: Disaster after maintenance
> > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > > > > > > Method) > > > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > > > Source) > > > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > &g
Re: Disaster after maintenance
; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > > > > > Method) > > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > > Source) > > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > > >> > Source) > > > > > > > > > > > > > > >> > >>at > > > > > java.lang.reflect.Method.invoke(Unknown > > > > > > > > Source) > > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) >
Re: Disaster after maintenance
rchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > > > > Method) > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > Source) > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > >> > Source) > > > > > > > > > > > > > >> > >>at > > > > java.lang.reflect.Method.invoke(Unknown > > > > > > > Source) > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > > >> > >> > > >
Re: Disaster after maintenance
invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > > > >> > >>at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > &g
Re: Disaster after maintenance
> > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > >
Re: Disaster after maintenance
t; > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > > >> > >>at > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > > Method) > > > > > > > > > > > >> > >>at > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > Source) > > > > > > > > > > > >> > >>at > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > >> > Source) > > > > > > > > > > > >> > >>at > > java.lang.reflect.Method.invoke(Unknown > > > > > Source) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > > >> > >>at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > > > >> > >>
Re: Disaster after maintenance
> > >
> >
>
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > > > > > > > > >> > >>at
> > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > > > > > > Method)
> > > > > > > > > > >> > >>at
> > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > > > > > > > > > Source)
> > > > > > > > > > >> > >>at
> > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > > > > > > > >> > Source)
> > > > > > > > > > >> > >>at
> java.lang.reflect.Method.invoke(Unknown
> > > > Source)
> > > > > > > > > > >> > >>at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > > > > > > > > >> > >>at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > > > > > > > > >> > >>at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > > > > > > > >> > >>at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > > > > > > > >> > >>at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > > > > > > >> > &
Re: Disaster after maintenance
ion.java:197) > > > > > > > > > > >> > >>at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > > >> > >>at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > >> > >>at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > > > > >> > >>at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > >> > >>at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > > > > > > >> > >>at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > >
Re: Disaster after maintenance
; >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > > > > > > > >> > >>at
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > > > > > Method)
> > > > > > > > > >> > >>at
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > > > > > > > > Source)
> > > > > > > > > >> > >>at
> > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > > > > > > >> > Source)
> > > > > > > > > >> > >>at java.lang.reflect.Method.invoke(Unknown
> > > Source)
> > > > > > > > > >> > >>at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > > > > > > > >> > >>at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > > > > > > > >> > >>at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > > > > > > >> > >>at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > > > > > > >> > >>at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > > > > > >> > >>at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> &
Re: Disaster after maintenance
t; org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > > > > > > >> > >>at > > > > > > > > > >> > >> > > > > > > > > > >> >
Re: Disaster after maintenance
ator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > > > > > > >> > >>at > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > > > > > > >> > >>at > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > > > > >> > >>at > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > >> > >>at > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > Method) > > > > > > > > >> > >>at > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > Source) > > > > > > > > >> > >>at > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > >> > Source) > > > > > > > > >> > >>at java.lang.reflect.Method.invoke(Unknown > > Source) > > > > > > > > >> > >>at > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > >> > >>at > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > >> > >>at > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > >> > >>at > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > >> > >>at > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > >
Re: Disaster after maintenance
; > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > > > > >> > >>at > com.sun.proxy.$Proxy229.restartNetwork(Unknown > > > > > Source) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > > > > > >> > >>at > > > > > > > >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > > > > > >> > >>at > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > &
Re: Disaster after maintenance
gt; > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > >> > >>at > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > Method) > > > > > > >> > >>at > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > Source) > > > > > > >> > >>at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > >> > Source) > > > > > > >> > >>at java.lang.reflect.Method.invoke(Unknown Source) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > >> > >>at > > > > > > >> > >> > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > >> > >>at > > > > > > >> > >> > > > &g
Re: Disaster after maintenance
ion.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > >> > >>at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > Method) > > > > > >> > >>at > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > Source) > > > > > >> > >>at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > >> > Source) > > > > > >> > >>at java.lang.reflect.Method.invoke(Unknown Source) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > > >> > >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown > > > Source) > > > > > >> > >>at > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) &g
Re: Disaster after maintenance
;> > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > >> > >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown > > Source) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > > >> > >>at > > > > >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > > >> > >>at > > > > >> > >> > > > > >> > > > > > >> > > > > &
Re: Disaster after maintenance
; >> > > > > >> > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > >> > >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown > Source) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > >> > >>at > > > >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > > >> > >>at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > > >> > >>at > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > > >> > >> Source) > > > >> > >>at java.util.concurrent.FutureTask.run(Unknown Source) > > > >> > >>at > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > >> > >> Source) > > > >> > >>at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > >> > >> Source) > > > >> > >>at java.lang.Thread.run(Unknown Source) > > > >> > >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) > > > Complete > > > >> > async > > > >> > >> job-5093, jobStatus: FAILED, resultCode: 530, result: > > > >> > >> > > > >>
Re: Disaster after maintenance
oud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > >> > >>at
> > >> > >>
> > >> >
> > >>
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > >> > >>at
> > >> > >>
> > >> >
> > >>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > >> > >>at
> > >> > >>
> > >> >
> > >>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > >> > >>at
> > >> > >>
> > >> >
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > >> > >>at
> > >> > >>
> > >> >
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > >> > >>at
> > >> > >>
> > >> >
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > >> > >>at
> > >> > >>
> > >> >
> > >>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > >> > >>at
> > >> > >>
> > >> >
> > >>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > >> > >>at
> > java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > >> > >> Source)
> > >> > >>at java.util.concurrent.FutureTask.run(Unknown Source)
> > >> > >>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > >> > >> Source)
> > >> > >>at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > >> > >> Source)
> > >> > >>at java.lang.Thread.run(Unknown Source)
> > >> > >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099)
> > Complete
> > >> > async
> > >> > >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> > >> > >>
> > >> >
> > >>
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > >> > >> [DataCenter:1] is un
> > >> > >>
> > >> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> > >> > andrija.pa...@shapeblue.com>
> > >> > >> wrote:
> > >> > >>
> > >> > >>>
> > >> > >>> Your network can't be deleted due to "Can't delete the network,
> > not
> > >> all
> > >> > >>> user vms are expunged. Vm
> > >> > >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
> > >> > >>>
> > >> > >>> You should be able to just start the user VM - but if you have
> > >> actually
> > >> > >>> delete the VR itself, then just do Network restart with
> "cleanup"
> > >> and
> > >> > it
> > >> > >>> will recreate a new VR, after which you should be able to start
> > the
> > >> VM.
> > >> > >>>
> > >> > >>> Andrija
> > >> > >>>
> > >> > >>> andrija.pa...@shapeblue.com
> > >> > >>> www.shapeblue.com
> > >> > >>> Amadeus House, Floral Street, London WC2E 9DPUK
> > >> > >>> @shapeblue
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>> -Original Message-
> > >> > >>> From: Jevgeni Zolotarjov
> > >> > >>> Sent: 19 March 2019 15:10
> > >> > >>> To: users@cloudstac
Re: Disaster after maintenance
Element.java:243) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > >> > >>at > >> > >> > >> > > >> > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > >> > >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> > >>at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > Source) > >> > >>at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > >> > Source) > >> > >>at java.lang.reflect.Method.invoke(Unknown Source) > >> > >>at > >> > >> > >> > > >> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > >> > >>at > >> > >> > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > >> > >>at > >> > >> > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > >> > >>at > >> > >> > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > >> > >>at > >> > >> > >> > > >> > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > >> > >>at > >> > >> > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > >> > >>at > >> > >> > >> > > >> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > >> > >>at > >> > >> > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > >> > >>at > >> > >> > >> > > >> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > >> > >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > >> > >>at > >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > >> > >>at > >> > >> > >> > > >> > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > >> > >>at > >> > >> > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > >> > >>at > >> > >> > >&g
Re: Disaster after maintenance
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > >>at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>> > >>at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>> > Source)
>> > >>at java.lang.reflect.Method.invoke(Unknown Source)
>> > >>at
>> > >>
>> >
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>> > >>at
>> > >>
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>> > >>at
>> > >>
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>> > >>at
>> > >>
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> > >>at
>> > >>
>> >
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> > >>at
>> > >>
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> > >>at
>> > >>
>> >
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>> > >>at
>> > >>
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>> > >>at
>> > >>
>> >
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>> > >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
>> > >>at
>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>> > >>at
>> > >>
>> >
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> > >>at
>> > >>
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>> > >>at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> > >> Source)
>> > >>at java.util.concurrent.FutureTask.run(Unknown Source)
>> > >>at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>> > >> Source)
>> > >>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> > >> Source)
>> > >>at java.lang.Thread.run(Unknown Source)
>> > >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete
>> > async
>> > >> job-5093, jobStatus: FAILED, resultCode: 530, result:
>> > >>
>> >
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
>> > >> [DataCenter:1] is un
Re: Disaster after maintenance
work.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > >>at
> > >>
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > >>at
> > >>
> >
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > >>at
> > >>
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > >>at
> > >>
> >
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > >>at
> > >>
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > >>at
> > >>
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > >>at
> > >>
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > >>at
> > >>
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> > >>at
> > >>
> >
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > >>at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > >>at
> > >>
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > >>at
> > >>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > >>at
> > >>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > >>at
> > >>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > >>at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > >> Source)
> > >>at java.util.concurrent.FutureTask.run(Unknown Source)
> > >>at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > >> Source)
> > >>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > >> Source)
> > >>at java.lang.Thread.run(Unknown Source)
> > >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete
> > async
> > >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> > >>
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > >> [DataCenter:1] is un
> > >>
> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> > andrija.pa...@shapeblue.com>
> > >> wrote:
> > >>
> > >>>
> > >>> Your network can't be deleted due to "Can't delete the network, not
> all
> > >>> user vms are expunged. Vm
> > >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
> > >>>
> > >>> You should be able to just start the user VM - but if you have
> actually
> > >>> delete the VR itself, then just do Network restart with "cleanup" and
> > it
> > >>> will recreate a new VR, after which you should be able to start the
> VM.
> > >>>
> > >>> Andrija
> > >>>
> > >>> andrija.pa...@shapeblue.com
> > >>> www.shapeblue.com
> > >>> Amadeus House, Fl
Re: Disaster after maintenance
nterceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> >>at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >>at
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> >>at
> >>
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> >>at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >>at
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >>at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >>at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >>at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> >> Source)
> >>at java.util.concurrent.FutureTask.run(Unknown Source)
> >>at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> >> Source)
> >>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >> Source)
> >>at java.lang.Thread.run(Unknown Source)
> >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete
> async
> >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> >>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> >> [DataCenter:1] is un
> >>
> >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> andrija.pa...@shapeblue.com>
> >> wrote:
> >>
> >>>
> >>> Your network can't be deleted due to "Can't delete the network, not all
> >>> user vms are expunged. Vm
> >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
> >>>
> >>> You should be able to just start the user VM - but if you have actually
> >>> delete the VR itself, then just do Network restart with "cleanup" and
> it
> >>> will recreate a new VR, after which you should be able to start the VM.
> >>>
> >>> Andrija
> >>>
> >>> andrija.pa...@shapeblue.com
> >>> www.shapeblue.com
> >>> Amadeus House, Floral Street, London WC2E 9DPUK
> >>> @shapeblue
> >>>
> >>>
> >>>
> >>>
> >>> -Original Message-
> >>> From: Jevgeni Zolotarjov
> >>> Sent: 19 March 2019 15:10
> >>> To: users@cloudstack.apache.org
> >>> Subject: Re: Disaster after maintenance
> >>>
> >>> I mean I cannot delete network: In the management server log I see
> >>>
> >>> ==
> >>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
> >>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
> >>> instanceId: 204, cmd:
> >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> cmdInfo:
> >>>
> >>>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> >>>
> >>>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2
Re: Disaster after maintenance
ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> >>at
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >>at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> >>at
> >>
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> >>at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> >>at
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> >>at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> >>at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> >>at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> >> Source)
> >>at java.util.concurrent.FutureTask.run(Unknown Source)
> >>at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> >> Source)
> >>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >> Source)
> >>at java.lang.Thread.run(Unknown Source)
> >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete
> async
> >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> >>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> >> [DataCenter:1] is un
> >>
> >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> andrija.pa...@shapeblue.com>
> >> wrote:
> >>
> >>>
> >>> Your network can't be deleted due to "Can't delete the network, not all
> >>> user vms are expunged. Vm
> >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
> >>>
> >>> You should be able to just start the user VM - but if you have actually
> >>> delete the VR itself, then just do Network restart with "cleanup" and
> it
> >>> will recreate a new VR, after which you should be able to start the VM.
> >>>
> >>> Andrija
> >>>
> >>> andrija.pa...@shapeblue.com
> >>> www.shapeblue.com
> >>> Amadeus House, Floral Street, London WC2E 9DPUK
> >>> @shapeblue
> >>>
> >>>
> >>>
> >>>
> >>> -Original Message-
> >>> From: Jevgeni Zolotarjov
> >>> Sent: 19 March 2019 15:10
> >>> To: users@cloudstack.apache.org
> >>> Subject: Re: Disaster after maintenance
> >>>
> >>> I mean I cannot delete network: In the management server log I see
> >>>
> >>> ==
> >>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
> >>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
> >>> instanceId: 204, cmd:
> >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> cmdInfo:
> >>>
> >>>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> >>>
> >>>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> >>> cmdVersion: 0, stat
Re: Disaster after maintenance
Impl$5.runInContext(AsyncJobManagerImpl.java:581)
>>at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>>at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source)
>>at java.util.concurrent.FutureTask.run(Unknown Source)
>>at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>> Source)
>>at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> Source)
>>at java.lang.Thread.run(Unknown Source)
>> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete async
>> job-5093, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
>> [DataCenter:1] is un
>>
>> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic
>> wrote:
>>
>>>
>>> Your network can't be deleted due to "Can't delete the network, not all
>>> user vms are expunged. Vm
>>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
>>>
>>> You should be able to just start the user VM - but if you have actually
>>> delete the VR itself, then just do Network restart with "cleanup" and it
>>> will recreate a new VR, after which you should be able to start the VM.
>>>
>>> Andrija
>>>
>>> andrija.pa...@shapeblue.com
>>> www.shapeblue.com
>>> Amadeus House, Floral Street, London WC2E 9DPUK
>>> @shapeblue
>>>
>>>
>>>
>>>
>>> -Original Message-
>>> From: Jevgeni Zolotarjov
>>> Sent: 19 March 2019 15:10
>>> To: users@cloudstack.apache.org
>>> Subject: Re: Disaster after maintenance
>>>
>>> I mean I cannot delete network: In the management server log I see
>>>
>>> ==
>>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
>>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
>>> instanceId: 204, cmd:
>>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
>>>
>>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
>>>
>>> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>> result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
>>> null, lastPolled: null, created: null}
>>> 2019-03-19 14:06:36,351 WARN [o.a.c.e.o.NetworkOrchestrator]
>>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) (logid:c734edfc)
>>> Can't delete the network, not all user vms are expunged. Vm
>>> VM[User|i-2-11-VM] is in Stopped state
>>> 2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete async
>>> job-5090, jobStatus: FAILED, resultCode: 530, result:
>>>
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>> to delete network"}
>>> ==
>>>
>>>
>>> I deleted a router, expecting it to be recreated on deleting network. But
>>> I am unable to delete network because of above e
RE: Disaster after maintenance
Somewhere in your error, I think that I saw a reference to rolling reboot. Try disabling that in the global settings. ( network.rolling.restart ) paul.an...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue -Original Message- From: Jevgeni Zolotarjov Sent: 19 March 2019 17:29 To: users@cloudstack.apache.org Subject: Re: Disaster after maintenance Guys, please help with it. What can be done here? There is too much valuable data. On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov wrote: > Tried that just now and got error: > Resource [DataCenter:1] is unreachable: Can't find all necessary > running routers! > > In the log I see: > = > > 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > (logid:265a6099) Restarting network 204... > 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > (logid:265a6099) Performing rolling restart of routers of network > Ntwk[204|Guest|6] > 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > (logid:265a6099) Asking VirtualRouter to implemenet Ntwk[204|Guest|6] > 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Unexpected > exception while executing > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > com.cloud.exception.ResourceUnavailableException: Resource > [DataCenter:1] is unreachable: Can't find all necessary running routers! > at > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > at > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source) > at > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(M
Re: Disaster after maintenance
ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete async
> job-5093, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> [DataCenter:1] is un
>
> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic
> wrote:
>
>>
>> Your network can't be deleted due to "Can't delete the network, not all
>> user vms are expunged. Vm
>> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
>>
>> You should be able to just start the user VM - but if you have actually
>> delete the VR itself, then just do Network restart with "cleanup" and it
>> will recreate a new VR, after which you should be able to start the VM.
>>
>> Andrija
>>
>> andrija.pa...@shapeblue.com
>> www.shapeblue.com
>> Amadeus House, Floral Street, London WC2E 9DPUK
>> @shapeblue
>>
>>
>>
>>
>> -Original Message-
>> From: Jevgeni Zolotarjov
>> Sent: 19 March 2019 15:10
>> To: users@cloudstack.apache.org
>> Subject: Re: Disaster after maintenance
>>
>> I mean I cannot delete network: In the management server log I see
>>
>> ==
>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
>> instanceId: 204, cmd:
>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
>>
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
>>
>> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
>> null, lastPolled: null, created: null}
>> 2019-03-19 14:06:36,351 WARN [o.a.c.e.o.NetworkOrchestrator]
>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) (logid:c734edfc)
>> Can't delete the network, not all user vms are expunged. Vm
>> VM[User|i-2-11-VM] is in Stopped state
>> 2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete async
>> job-5090, jobStatus: FAILED, resultCode: 530, result:
>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>> to delete network"}
>> ==
>>
>>
>> I deleted a router, expecting it to be recreated on deleting network. But
>> I am unable to delete network because of above error
>>
>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov <
>> j.zolotar...@gmail.com>
>> wrote:
>>
>> > I've managed to make libvirtd running
>> > Now cloudstack console shows both hosts - running
>> >
>> > But now as I have removed network, VMs are unable to start.
>> >
>> > How can I recreate the network now?
>> >
>> > On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
>> >
>> > wrote:
>> >
>> >> Jevgeniy, it may be a documentation bug. Take s look:
>> >> https://github.com/apache/cloudstack-documentation/pull/27/files
>> >>
>> >> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov :
>> >>
>> >> > That's it - libvirtd failed to start on second host.
>> >> > Tried restarting, but it does not start.
>> >> >
>> >> >
>> >> > >> Do you have some NUMA constraints or anything which requires
>> >> particular
>> >> > RAM configuration?
>> >> > No
>> >> >
>> >> > libvirtd.service - Virtualization daemon
>> >> >Loaded: loaded (/usr/lib/systemd/system/libvirtd.servic
Re: Disaster after maintenance
due to "Can't delete the network, not all
> user vms are expunged. Vm
> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
>
> You should be able to just start the user VM - but if you have actually
> delete the VR itself, then just do Network restart with "cleanup" and it
> will recreate a new VR, after which you should be able to start the VM.
>
> Andrija
>
> andrija.pa...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Jevgeni Zolotarjov
> Sent: 19 March 2019 15:10
> To: users@cloudstack.apache.org
> Subject: Re: Disaster after maintenance
>
> I mean I cannot delete network: In the management server log I see
>
> ==
> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
> instanceId: 204, cmd:
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2019-03-19 14:06:36,351 WARN [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) (logid:c734edfc)
> Can't delete the network, not all user vms are expunged. Vm
> VM[User|i-2-11-VM] is in Stopped state
> 2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete async
> job-5090, jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to delete network"}
> ==
>
>
> I deleted a router, expecting it to be recreated on deleting network. But
> I am unable to delete network because of above error
>
> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov >
> wrote:
>
> > I've managed to make libvirtd running
> > Now cloudstack console shows both hosts - running
> >
> > But now as I have removed network, VMs are unable to start.
> >
> > How can I recreate the network now?
> >
> > On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
> >
> > wrote:
> >
> >> Jevgeniy, it may be a documentation bug. Take s look:
> >> https://github.com/apache/cloudstack-documentation/pull/27/files
> >>
> >> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov :
> >>
> >> > That's it - libvirtd failed to start on second host.
> >> > Tried restarting, but it does not start.
> >> >
> >> >
> >> > >> Do you have some NUMA constraints or anything which requires
> >> particular
> >> > RAM configuration?
> >> > No
> >> >
> >> > libvirtd.service - Virtualization daemon
> >> >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service;
> >> > enabled; vendor preset: enabled)
> >> >Active: failed (Result: start-limit) since Tue 2019-03-19
> >> > 13:03:07
> >> GMT;
> >> > 12s ago
> >> > Docs: man:libvirtd(8)
> >> >https://libvirt.org
> >> > Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
> >> > (code=exited,
> >> > status=1/FAILURE)
> >> > Main PID: 892 (code=exited, status=1/FAILURE)
> >> > Tasks: 19 (limit: 32768)
> >> >CGroup: /system.slice/libvirtd.service
> >> >├─11338 /usr/sbin/libvirtd -d -l
> >> >├─11909 /usr/sbin/dnsmasq
> >> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> >> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >> >└─11910 /usr/sbin/dnsmasq
> >> > --conf-file
RE: Disaster after maintenance
Your network can't be deleted due to "Can't delete the network, not all user
vms are expunged. Vm
VM[User|i-2-11-VM] is in Stopped state" - which is fine.
You should be able to just start the user VM - but if you have actually delete
the VR itself, then just do Network restart with "cleanup" and it will recreate
a new VR, after which you should be able to start the VM.
Andrija
andrija.pa...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
-Original Message-
From: Jevgeni Zolotarjov
Sent: 19 March 2019 15:10
To: users@cloudstack.apache.org
Subject: Re: Disaster after maintenance
I mean I cannot delete network: In the management server log I see
==
019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
instanceId: 204, cmd:
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2019-03-19 14:06:36,351 WARN [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) (logid:c734edfc) Can't
delete the network, not all user vms are expunged. Vm VM[User|i-2-11-VM] is in
Stopped state
2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete async
job-5090, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to delete network"}
==
I deleted a router, expecting it to be recreated on deleting network. But I am
unable to delete network because of above error
On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov
wrote:
> I've managed to make libvirtd running
> Now cloudstack console shows both hosts - running
>
> But now as I have removed network, VMs are unable to start.
>
> How can I recreate the network now?
>
> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
>
> wrote:
>
>> Jevgeniy, it may be a documentation bug. Take s look:
>> https://github.com/apache/cloudstack-documentation/pull/27/files
>>
>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov :
>>
>> > That's it - libvirtd failed to start on second host.
>> > Tried restarting, but it does not start.
>> >
>> >
>> > >> Do you have some NUMA constraints or anything which requires
>> particular
>> > RAM configuration?
>> > No
>> >
>> > libvirtd.service - Virtualization daemon
>> >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service;
>> > enabled; vendor preset: enabled)
>> >Active: failed (Result: start-limit) since Tue 2019-03-19
>> > 13:03:07
>> GMT;
>> > 12s ago
>> > Docs: man:libvirtd(8)
>> >https://libvirt.org
>> > Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
>> > (code=exited,
>> > status=1/FAILURE)
>> > Main PID: 892 (code=exited, status=1/FAILURE)
>> > Tasks: 19 (limit: 32768)
>> >CGroup: /system.slice/libvirtd.service
>> >├─11338 /usr/sbin/libvirtd -d -l
>> >├─11909 /usr/sbin/dnsmasq
>> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
>> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> >└─11910 /usr/sbin/dnsmasq
>> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
>> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> >
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to
>> > start Virtualization daemon.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
>> > libvirtd.service entered failed state.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pb
Re: Disaster after maintenance
13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: >> libvirtd.service >> > failed. >> > >> > >> > On Tue, Mar 19, 2019 at 3:04 PM Paul Angus >> > wrote: >> > >> > > Can you check that the cloudstack agent is running on the host and the >> > > agent logs (usual logs directory) >> > > Also worth checking that libvirt has started ok. Do you have some >> NUMA >> > > constraints or anything which requires particular RAM configuration? >> > > >> > > paul.an...@shapeblue.com >> > > www.shapeblue.com >> > > Amadeus House, Floral Street, London WC2E 9DPUK >> > > @shapeblue >> > > >> > > >> > > >> > > >> > > -Original Message- >> > > From: Jevgeni Zolotarjov >> > > Sent: 19 March 2019 14:49 >> > > To: users@cloudstack.apache.org >> > > Subject: Re: Disaster after maintenance >> > > >> > > Can you try migrating a VM to the server that you changed the RAM >> amount? >> > > >> > > Also: >> > > What is the hypervisor version? >> > > KVM >> > > QEMU Version : 2.0.0 >> > > Release : 1.el7.6 >> > > >> > > >> > > Host status in ACS? >> > > 1st server: Unsecure >> > > 2nd server: Disconnected >> > > >> > > Did you try to force a VM to start/deploy in this server where you >> > changed >> > > the RAM? >> > > Host status became disconnected. I don't know how to make it >> "connected" >> > > again >> > > >> > > >> > > >> > > On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner < >> > > rafaelweingart...@gmail.com> wrote: >> > > >> > > > Can you try migrating a VM to the server that you changed the RAM >> > amount? >> > > > >> > > > Also: >> > > > What is the hypervisor version? >> > > > Host status in ACS? >> > > > Did you try to force a VM to start/deploy in this server where you >> > > > changed the RAM? >> > > > >> > > > >> > > > On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov >> > > > > > > > > >> > > > wrote: >> > > > >> > > > > We have Cloudstack 4.11.2 setup running fine for few months (>4) >> The >> > > > > setup is very simple: 2 hosts We decided to do a maintenance to >> > > > > increase RAM on both servers >> > > > > >> > > > > For this we put first server to maintenance. All VMS moved to >> second >> > > > > host after a while. >> > > > > >> > > > > Then first server was shutdown, RAM increased, server turned ON. >> > > > > Now nothing starts on first server. >> > > > > >> > > > > >> > > > > Tried to delete network, but this fails as well >> > > > > >> > > > > Please help ! >> > > > > >> > > > > Here is extract from log: >> > > > > == >> > > > > 2019-03-19 12:27:53,064 DEBUG >> [o.a.c.s.SecondaryStorageManagerImpl] >> > > > > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to >> > > > > launch secondary storage VM >> > > > > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl] >> > > > > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to >> > > > > launch console proxy >> > > > > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet] >> > > > > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START=== >> > > > 192.168.5.140 >> > > > > -- GET >> > > > > >> > > > > >> > > > >> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1 >> > > > =json&_=1552998473154 >> > > > > 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer] >> > > > > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) >> CIDRs >> > > > > from which account >> > > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is >> > > > allowed >> > > > > to perform API c
Re: Disaster after maintenance
I've managed to make libvirtd running Now cloudstack console shows both hosts - running But now as I have removed network, VMs are unable to start. How can I recreate the network now? On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev wrote: > Jevgeniy, it may be a documentation bug. Take s look: > https://github.com/apache/cloudstack-documentation/pull/27/files > > вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov : > > > That's it - libvirtd failed to start on second host. > > Tried restarting, but it does not start. > > > > > > >> Do you have some NUMA constraints or anything which requires > particular > > RAM configuration? > > No > > > > libvirtd.service - Virtualization daemon > >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; > > vendor preset: enabled) > >Active: failed (Result: start-limit) since Tue 2019-03-19 13:03:07 > GMT; > > 12s ago > > Docs: man:libvirtd(8) > >https://libvirt.org > > Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, > > status=1/FAILURE) > > Main PID: 892 (code=exited, status=1/FAILURE) > > Tasks: 19 (limit: 32768) > >CGroup: /system.slice/libvirtd.service > >├─11338 /usr/sbin/libvirtd -d -l > >├─11909 /usr/sbin/dnsmasq > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro > > --dhcp-script=/usr/libexec/libvirt_leaseshelper > >└─11910 /usr/sbin/dnsmasq > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro > > --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start > > Virtualization daemon. > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit > > libvirtd.service entered failed state. > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service > > failed. > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service > > holdoff time over, scheduling restart. > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped > > Virtualization daemon. > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start request > > repeated too quickly for libvirtd.service > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start > > Virtualization daemon. > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit > > libvirtd.service entered failed state. > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service > > failed. > > > > > > On Tue, Mar 19, 2019 at 3:04 PM Paul Angus > > wrote: > > > > > Can you check that the cloudstack agent is running on the host and the > > > agent logs (usual logs directory) > > > Also worth checking that libvirt has started ok. Do you have some NUMA > > > constraints or anything which requires particular RAM configuration? > > > > > > paul.an...@shapeblue.com > > > www.shapeblue.com > > > Amadeus House, Floral Street, London WC2E 9DPUK > > > @shapeblue > > > > > > > > > > > > > > > -Original Message- > > > From: Jevgeni Zolotarjov > > > Sent: 19 March 2019 14:49 > > > To: users@cloudstack.apache.org > > > Subject: Re: Disaster after maintenance > > > > > > Can you try migrating a VM to the server that you changed the RAM > amount? > > > > > > Also: > > > What is the hypervisor version? > > > KVM > > > QEMU Version : 2.0.0 > > > Release : 1.el7.6 > > > > > > > > > Host status in ACS? > > > 1st server: Unsecure > > > 2nd server: Disconnected > > > > > > Did you try to force a VM to start/deploy in this server where you > > changed > > > the RAM? > > > Host status became disconnected. I don't know how to make it > "connected" > > > again > > > > > > > > > > > > On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner < > > > rafaelweingart...@gmail.com> wrote: > > > > > > > Can you try migrating a VM to the server that you changed the RAM > > amount? > > > > > > > > Also: > > > > What is the hypervisor version? > > > > Host status in ACS? > > > > Did you try to force a VM to start/deploy in this server where you > > > > changed the RAM? > > > > > > > > > > > > On Tue, Mar 19, 2019 at 9:3
RE: Disaster after maintenance
Libvirtd has its own logs, so you'll need to look at those I'm afraid. paul.an...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue -Original Message- From: Jevgeni Zolotarjov Sent: 19 March 2019 15:09 To: users@cloudstack.apache.org Subject: Re: Disaster after maintenance That's it - libvirtd failed to start on second host. Tried restarting, but it does not start. >> Do you have some NUMA constraints or anything which requires >> particular RAM configuration? No libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Tue 2019-03-19 13:03:07 GMT; 12s ago Docs: man:libvirtd(8) https://libvirt.org Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=1/FAILURE) Main PID: 892 (code=exited, status=1/FAILURE) Tasks: 19 (limit: 32768) CGroup: /system.slice/libvirtd.service ├─11338 /usr/sbin/libvirtd -d -l ├─11909 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper └─11910 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start Virtualization daemon. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit libvirtd.service entered failed state. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service failed. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service holdoff time over, scheduling restart. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped Virtualization daemon. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start request repeated too quickly for libvirtd.service Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start Virtualization daemon. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit libvirtd.service entered failed state. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service failed. On Tue, Mar 19, 2019 at 3:04 PM Paul Angus wrote: > Can you check that the cloudstack agent is running on the host and the > agent logs (usual logs directory) Also worth checking that libvirt has > started ok. Do you have some NUMA constraints or anything which > requires particular RAM configuration? > > paul.an...@shapeblue.com > www.shapeblue.com > Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue > > > > > -Original Message- > From: Jevgeni Zolotarjov > Sent: 19 March 2019 14:49 > To: users@cloudstack.apache.org > Subject: Re: Disaster after maintenance > > Can you try migrating a VM to the server that you changed the RAM amount? > > Also: > What is the hypervisor version? > KVM > QEMU Version : 2.0.0 > Release : 1.el7.6 > > > Host status in ACS? > 1st server: Unsecure > 2nd server: Disconnected > > Did you try to force a VM to start/deploy in this server where you changed > the RAM? > Host status became disconnected. I don't know how to make it "connected" > again > > > > On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner < > rafaelweingart...@gmail.com> wrote: > > > Can you try migrating a VM to the server that you changed the RAM amount? > > > > Also: > > What is the hypervisor version? > > Host status in ACS? > > Did you try to force a VM to start/deploy in this server where you > > changed the RAM? > > > > > > On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov > > > > > > wrote: > > > > > We have Cloudstack 4.11.2 setup running fine for few months (>4) The > > > setup is very simple: 2 hosts We decided to do a maintenance to > > > increase RAM on both servers > > > > > > For this we put first server to maintenance. All VMS moved to second > > > host after a while. > > > > > > Then first server was shutdown, RAM increased, server turned ON. > > > Now nothing starts on first server. > > > > > > > > > Tried to delete network, but this fails as well > > > > > > Please help ! > > > > > > Here is extract from log: > > > == > > > 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > > > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to > > > launch secondary storage VM > > > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl] > > > (consoleproxy-1:ctx-cbd034b9) (logid:0
Re: Disaster after maintenance
Jevgeniy, it may be a documentation bug. Take s look: https://github.com/apache/cloudstack-documentation/pull/27/files вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov : > That's it - libvirtd failed to start on second host. > Tried restarting, but it does not start. > > > >> Do you have some NUMA constraints or anything which requires particular > RAM configuration? > No > > libvirtd.service - Virtualization daemon >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; > vendor preset: enabled) >Active: failed (Result: start-limit) since Tue 2019-03-19 13:03:07 GMT; > 12s ago > Docs: man:libvirtd(8) >https://libvirt.org > Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, > status=1/FAILURE) > Main PID: 892 (code=exited, status=1/FAILURE) > Tasks: 19 (limit: 32768) >CGroup: /system.slice/libvirtd.service >├─11338 /usr/sbin/libvirtd -d -l >├─11909 /usr/sbin/dnsmasq > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro > --dhcp-script=/usr/libexec/libvirt_leaseshelper >└─11910 /usr/sbin/dnsmasq > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro > --dhcp-script=/usr/libexec/libvirt_leaseshelper > > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start > Virtualization daemon. > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit > libvirtd.service entered failed state. > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service > failed. > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service > holdoff time over, scheduling restart. > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped > Virtualization daemon. > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start request > repeated too quickly for libvirtd.service > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start > Virtualization daemon. > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit > libvirtd.service entered failed state. > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service > failed. > > > On Tue, Mar 19, 2019 at 3:04 PM Paul Angus > wrote: > > > Can you check that the cloudstack agent is running on the host and the > > agent logs (usual logs directory) > > Also worth checking that libvirt has started ok. Do you have some NUMA > > constraints or anything which requires particular RAM configuration? > > > > paul.an...@shapeblue.com > > www.shapeblue.com > > Amadeus House, Floral Street, London WC2E 9DPUK > > @shapeblue > > > > > > > > > > -Original Message- > > From: Jevgeni Zolotarjov > > Sent: 19 March 2019 14:49 > > To: users@cloudstack.apache.org > > Subject: Re: Disaster after maintenance > > > > Can you try migrating a VM to the server that you changed the RAM amount? > > > > Also: > > What is the hypervisor version? > > KVM > > QEMU Version : 2.0.0 > > Release : 1.el7.6 > > > > > > Host status in ACS? > > 1st server: Unsecure > > 2nd server: Disconnected > > > > Did you try to force a VM to start/deploy in this server where you > changed > > the RAM? > > Host status became disconnected. I don't know how to make it "connected" > > again > > > > > > > > On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner < > > rafaelweingart...@gmail.com> wrote: > > > > > Can you try migrating a VM to the server that you changed the RAM > amount? > > > > > > Also: > > > What is the hypervisor version? > > > Host status in ACS? > > > Did you try to force a VM to start/deploy in this server where you > > > changed the RAM? > > > > > > > > > On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov > > > > > > > > > wrote: > > > > > > > We have Cloudstack 4.11.2 setup running fine for few months (>4) The > > > > setup is very simple: 2 hosts We decided to do a maintenance to > > > > increase RAM on both servers > > > > > > > > For this we put first server to maintenance. All VMS moved to second > > > > host after a while. > > > > > > > > Then first server was shutdown, RAM increased, server turned ON. > > > > Now nothing starts on first server. > > > > > > > > > > > > Tried to delete network, but this fails as well > > > > > > > > Please help ! > > > > >
Re: Disaster after maintenance
That's it - libvirtd failed to start on second host. Tried restarting, but it does not start. >> Do you have some NUMA constraints or anything which requires particular RAM configuration? No libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Tue 2019-03-19 13:03:07 GMT; 12s ago Docs: man:libvirtd(8) https://libvirt.org Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=1/FAILURE) Main PID: 892 (code=exited, status=1/FAILURE) Tasks: 19 (limit: 32768) CGroup: /system.slice/libvirtd.service ├─11338 /usr/sbin/libvirtd -d -l ├─11909 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper └─11910 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start Virtualization daemon. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit libvirtd.service entered failed state. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service failed. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service holdoff time over, scheduling restart. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped Virtualization daemon. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start request repeated too quickly for libvirtd.service Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start Virtualization daemon. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit libvirtd.service entered failed state. Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service failed. On Tue, Mar 19, 2019 at 3:04 PM Paul Angus wrote: > Can you check that the cloudstack agent is running on the host and the > agent logs (usual logs directory) > Also worth checking that libvirt has started ok. Do you have some NUMA > constraints or anything which requires particular RAM configuration? > > paul.an...@shapeblue.com > www.shapeblue.com > Amadeus House, Floral Street, London WC2E 9DPUK > @shapeblue > > > > > -Original Message- > From: Jevgeni Zolotarjov > Sent: 19 March 2019 14:49 > To: users@cloudstack.apache.org > Subject: Re: Disaster after maintenance > > Can you try migrating a VM to the server that you changed the RAM amount? > > Also: > What is the hypervisor version? > KVM > QEMU Version : 2.0.0 > Release : 1.el7.6 > > > Host status in ACS? > 1st server: Unsecure > 2nd server: Disconnected > > Did you try to force a VM to start/deploy in this server where you changed > the RAM? > Host status became disconnected. I don't know how to make it "connected" > again > > > > On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner < > rafaelweingart...@gmail.com> wrote: > > > Can you try migrating a VM to the server that you changed the RAM amount? > > > > Also: > > What is the hypervisor version? > > Host status in ACS? > > Did you try to force a VM to start/deploy in this server where you > > changed the RAM? > > > > > > On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov > > > > > > wrote: > > > > > We have Cloudstack 4.11.2 setup running fine for few months (>4) The > > > setup is very simple: 2 hosts We decided to do a maintenance to > > > increase RAM on both servers > > > > > > For this we put first server to maintenance. All VMS moved to second > > > host after a while. > > > > > > Then first server was shutdown, RAM increased, server turned ON. > > > Now nothing starts on first server. > > > > > > > > > Tried to delete network, but this fails as well > > > > > > Please help ! > > > > > > Here is extract from log: > > > == > > > 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > > > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to > > > launch secondary storage VM > > > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl] > > > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to > > > launch console proxy > > > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet] > > > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START=== > > 192.168.5.140 > > > -- GET > > > > > > > > command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1 &
RE: Disaster after maintenance
Can you check that the cloudstack agent is running on the host and the agent
logs (usual logs directory)
Also worth checking that libvirt has started ok. Do you have some NUMA
constraints or anything which requires particular RAM configuration?
paul.an...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
-Original Message-
From: Jevgeni Zolotarjov
Sent: 19 March 2019 14:49
To: users@cloudstack.apache.org
Subject: Re: Disaster after maintenance
Can you try migrating a VM to the server that you changed the RAM amount?
Also:
What is the hypervisor version?
KVM
QEMU Version : 2.0.0
Release : 1.el7.6
Host status in ACS?
1st server: Unsecure
2nd server: Disconnected
Did you try to force a VM to start/deploy in this server where you changed the
RAM?
Host status became disconnected. I don't know how to make it "connected"
again
On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:
> Can you try migrating a VM to the server that you changed the RAM amount?
>
> Also:
> What is the hypervisor version?
> Host status in ACS?
> Did you try to force a VM to start/deploy in this server where you
> changed the RAM?
>
>
> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov
> >
> wrote:
>
> > We have Cloudstack 4.11.2 setup running fine for few months (>4) The
> > setup is very simple: 2 hosts We decided to do a maintenance to
> > increase RAM on both servers
> >
> > For this we put first server to maintenance. All VMS moved to second
> > host after a while.
> >
> > Then first server was shutdown, RAM increased, server turned ON.
> > Now nothing starts on first server.
> >
> >
> > Tried to delete network, but this fails as well
> >
> > Please help !
> >
> > Here is extract from log:
> > ==
> > 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to
> > launch secondary storage VM
> > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to
> > launch console proxy
> > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===
> 192.168.5.140
> > -- GET
> >
> >
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1
> =json&_=1552998473154
> > 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) CIDRs
> > from which account
> > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> allowed
> > to perform API calls: 0.0.0.0/0,::/0
> > 2019-03-19 12:27:53,208 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add
> > job-5081 into job monitoring
> > 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) submit
> async
> > job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId: 2,
> > instanceType: Network, instanceId: 204, cmd:
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> interface
> >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> EventType":"NETWORK.DELETE","_":"1552998473154"},
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: null}
> > 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be)
> > ===END===
> > 192.168.5.140 -- GET
> >
> >
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1
> =json&_=1552998473154
> > 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
> > Executing AsyncJobVO {id:5081, userId: 2, accountId: 2,
> > instanceType: Network,
> > instanceId: 204, cmd:
> > org.apache.cloudstack.api.command.user.
Re: Disaster after maintenance
that is why nothing deploys there. You need to connect this host to ACS.
otherwise, it will just be ignored. Did you check the log files in the
agent (in the host)? And, of course, in ACS?
On Tue, Mar 19, 2019 at 9:49 AM Jevgeni Zolotarjov
wrote:
> Can you try migrating a VM to the server that you changed the RAM amount?
>
> Also:
> What is the hypervisor version?
> KVM
> QEMU Version : 2.0.0
> Release : 1.el7.6
>
>
> Host status in ACS?
> 1st server: Unsecure
> 2nd server: Disconnected
>
> Did you try to force a VM to start/deploy in this server where you changed
> the RAM?
> Host status became disconnected. I don't know how to make it "connected"
> again
>
>
>
> On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > Can you try migrating a VM to the server that you changed the RAM amount?
> >
> > Also:
> > What is the hypervisor version?
> > Host status in ACS?
> > Did you try to force a VM to start/deploy in this server where you
> changed
> > the RAM?
> >
> >
> > On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > wrote:
> >
> > > We have Cloudstack 4.11.2 setup running fine for few months (>4)
> > > The setup is very simple: 2 hosts
> > > We decided to do a maintenance to increase RAM on both servers
> > >
> > > For this we put first server to maintenance. All VMS moved to second
> host
> > > after a while.
> > >
> > > Then first server was shutdown, RAM increased, server turned ON.
> > > Now nothing starts on first server.
> > >
> > >
> > > Tried to delete network, but this fails as well
> > >
> > > Please help !
> > >
> > > Here is extract from log:
> > > ==
> > > 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to launch
> > > secondary storage VM
> > > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to
> launch
> > > console proxy
> > > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> > > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===
> > 192.168.5.140
> > > -- GET
> > >
> > >
> >
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
> > > 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
> > > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) CIDRs
> from
> > > which account 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> > allowed
> > > to perform API calls: 0.0.0.0/0,::/0
> > > 2019-03-19 12:27:53,208 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add
> job-5081
> > > into job monitoring
> > > 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) submit
> > async
> > > job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId: 2,
> > > instanceType: Network, instanceId: 204, cmd:
> > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> cmdInfo:
> > >
> > >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> > >
> > >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
> > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > > result: null, initMsid: 264216221068220, completeMsid: null,
> lastUpdated:
> > > null, lastPolled: null, created: null}
> > > 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
> > > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) ===END===
> > > 192.168.5.140 -- GET
> > >
> > >
> >
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
> > > 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Executing
> > > AsyncJobVO {id:5081, userId: 2, accountId: 2, instanceType: Network,
> > > instanceId: 204, cmd:
> > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> cmdInfo:
> > >
> > >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> > >
> > >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
> > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > > result: null, initMsid: 264216221068220, completeMsid: null,
> lastUpdated:
> > > null, lastPolled: null, created: null}
> > > 2019-03-19 12:27:53,257 WARN [o.a.c.e.o.NetworkOrchestrator]
> > > (API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979)
>
Re: Disaster after maintenance
Can you try migrating a VM to the server that you changed the RAM amount?
Also:
What is the hypervisor version?
KVM
QEMU Version : 2.0.0
Release : 1.el7.6
Host status in ACS?
1st server: Unsecure
2nd server: Disconnected
Did you try to force a VM to start/deploy in this server where you changed
the RAM?
Host status became disconnected. I don't know how to make it "connected"
again
On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:
> Can you try migrating a VM to the server that you changed the RAM amount?
>
> Also:
> What is the hypervisor version?
> Host status in ACS?
> Did you try to force a VM to start/deploy in this server where you changed
> the RAM?
>
>
> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov >
> wrote:
>
> > We have Cloudstack 4.11.2 setup running fine for few months (>4)
> > The setup is very simple: 2 hosts
> > We decided to do a maintenance to increase RAM on both servers
> >
> > For this we put first server to maintenance. All VMS moved to second host
> > after a while.
> >
> > Then first server was shutdown, RAM increased, server turned ON.
> > Now nothing starts on first server.
> >
> >
> > Tried to delete network, but this fails as well
> >
> > Please help !
> >
> > Here is extract from log:
> > ==
> > 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to launch
> > secondary storage VM
> > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to launch
> > console proxy
> > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===
> 192.168.5.140
> > -- GET
> >
> >
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
> > 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) CIDRs from
> > which account 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> allowed
> > to perform API calls: 0.0.0.0/0,::/0
> > 2019-03-19 12:27:53,208 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add job-5081
> > into job monitoring
> > 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) submit
> async
> > job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId: 2,
> > instanceType: Network, instanceId: 204, cmd:
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: null}
> > 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) ===END===
> > 192.168.5.140 -- GET
> >
> >
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
> > 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Executing
> > AsyncJobVO {id:5081, userId: 2, accountId: 2, instanceType: Network,
> > instanceId: 204, cmd:
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: null}
> > 2019-03-19 12:27:53,257 WARN [o.a.c.e.o.NetworkOrchestrator]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979) (logid:16897ea6)
> > Can't delete the network, not all user vms are expunged. Vm
> > VM[User|i-2-11-VM] is in Stopped state
> > 2019-03-19 12:27:53,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Complete
> async
> > job-5081, jobStatus: FAILED, resultCode: 530, result:
> >
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> > to delete network"}
> > 2019-03-19 12:27:53,264 DEBUG
Re: Disaster after maintenance
Can you try migrating a VM to the server that you changed the RAM amount?
Also:
What is the hypervisor version?
Host status in ACS?
Did you try to force a VM to start/deploy in this server where you changed
the RAM?
On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov
wrote:
> We have Cloudstack 4.11.2 setup running fine for few months (>4)
> The setup is very simple: 2 hosts
> We decided to do a maintenance to increase RAM on both servers
>
> For this we put first server to maintenance. All VMS moved to second host
> after a while.
>
> Then first server was shutdown, RAM increased, server turned ON.
> Now nothing starts on first server.
>
>
> Tried to delete network, but this fails as well
>
> Please help !
>
> Here is extract from log:
> ==
> 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to launch
> secondary storage VM
> 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to launch
> console proxy
> 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START=== 192.168.5.140
> -- GET
>
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
> 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) CIDRs from
> which account 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is allowed
> to perform API calls: 0.0.0.0/0,::/0
> 2019-03-19 12:27:53,208 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add job-5081
> into job monitoring
> 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) submit async
> job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId: 2,
> instanceType: Network, instanceId: 204, cmd:
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) ===END===
> 192.168.5.140 -- GET
>
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
> 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Executing
> AsyncJobVO {id:5081, userId: 2, accountId: 2, instanceType: Network,
> instanceId: 204, cmd:
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2019-03-19 12:27:53,257 WARN [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979) (logid:16897ea6)
> Can't delete the network, not all user vms are expunged. Vm
> VM[User|i-2-11-VM] is in Stopped state
> 2019-03-19 12:27:53,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Complete async
> job-5081, jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to delete network"}
> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Publish async
> job-5081 complete on message bus
> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Wake up jobs
> related to job-5081
> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Update db
> status for job-5081
> 2019-03-19 12:27:53,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Wake up jobs
> joined with job-5081 and disjoin all subjobs created from job- 5081
> 2019-03-19 12:27:53,267
