I think I've seen a suggestion earlier about adding the world "solved" in the subject whenever someone finds a solution for its problem.
So, regarding this management IP pool issue, I'm not sure that's the definitive solution but it did work for me. Having all management IPs used, CloudStack was failing to start the console proxy VM. I created a dump of cloud DB and start looking for my management IPs. I found a table called op_dc_ip_address_alloc with references to all my management IPs. Long story short, 2 of the 4 IP addresses were still showing as reserved so I simply run this: mysql> update op_dc_ip_address_alloc set nic_id=NULL,reservation_id=NULL,taken=NULL where id=1; The id in the command above was the id corresponding to the IPs that were still showing as reserved. In the next 30 seconds CloudStack started also the console proxy VM and everything came back to normal. I suggest creating a full backup of the database before changing anything though. Daniel On Thu, Sep 26, 2013 at 2:24 AM, Daniel Hertanu <d...@unixmob.com> wrote: > I posted here: http://pastebin.com/uJ6jR73D the logs, maybe someone else > have seen this behavior before. Just to remind the issue, CS 4.1, XenServer > 6.1, when the only existing host is rebooted without being in maintenance > mode (and stays down for a while), all the management IPs get used (in this > case there were 171 management IPs, all show as in use now). > > Thanks, > > Daniel > > > On Wed, Sep 25, 2013 at 8:56 PM, Musayev, Ilya <imusa...@webmd.net> wrote: > >> Dan, >> >> Ping me on IRC, nick serverchief >> >> Regards >> ilya >> >> >> > -----Original Message----- >> > From: Daniel Hertanu [mailto:d...@unixmob.com] >> > Sent: Wednesday, September 25, 2013 6:44 AM >> > To: users@cloudstack.apache.org >> > Subject: Re: Management IP address pool exhausted >> > >> > Hi >> > >> > So I re-installed CS, reboot Xen host without entering it first in >> Maintenace >> > Mode, then got the management IP pool fully used. I followed again your >> > indications but to no avail. >> > In the meantime SSVM has reached number 112 (s-112-VM). >> > >> > I would really appreciate any other idea. >> > >> > Thanks, >> > >> > Daniel >> > >> > >> > On Wed, Sep 25, 2013 at 9:36 AM, Daniel Hertanu <d...@unixmob.com> >> > wrote: >> > >> > > Hello Ilya >> > > >> > > I submit an issue describing the bug: >> > > https://issues.apache.org/jira/browse/CLOUDSTACK-4735 (I picked CS >> > > version 4.1.0 as 4.1.1 is reported as non existent, but I've seen this >> > > behavior before since version 3). >> > > >> > > I followed your indications but it didn't work. After changing those >> > > records in the database, CS would still give the same error. I revert >> > > back to the initial db content, but this time nothing came back >> > > (initially the SSVM was still running, it was CPVM that didn't start). >> > > After this step I tried some tricks on my own and I ended up with a >> botched >> > environment. >> > > So I'll do a fresh install, reboot the host and try your suggestions >> > > again. >> > > >> > > Thanks, >> > > >> > > Daniel >> > > >> > > >> > > On Tue, Sep 24, 2013 at 10:46 PM, Musayev, Ilya >> > <imusa...@webmd.net>wrote: >> > > >> > >> I'm not certain if this is hypervisor specific - I've seen the same >> > >> issue for vSphere - but I though it was me, I think we may need to >> > >> revise the system vm ip space decommission process. >> > >> >> > >> Shanker and/or Daniel, >> > >> >> > >> Do you mind creating a ticket in issues.apache.org and write out >> ways >> > >> to reproduce this issue. >> > >> >> > >> --- >> > >> >> > >> *NOTE* this is purely from memory and some notes I had - proceed at >> > >> your own risk.... >> > >> >> > >> >> > >> >> > >> In the meantime, lets clean up what you have. >> > >> >> > >> The goal is to release the IPs that are still marked as reserved and >> > >> allocated. >> > >> >> > >> I'd recommend installing mysql workbench >> > >> >> > >> There are few tables we need to review: >> > >> 1) Stop cloudstack management server >> > >> 2) Backup mysql db first. >> > >> >> > >> Note the id of management network referenced in cloud.networks >> > >> Select id from cloud.networks where traffic_type="Management" >> > >> In my case, Management network has ID of 205 >> > >> SELECT * FROM cloud.nics where network_id=205; Alter to >> > >> Deallocating the ones that are marked as Reserved. >> > >> >> > >> Do the same for cloud.user_ip_address SELECT * FROM >> > >> cloud.user_ip_address where source_network_id=205; Alter from >> > >> Allocated to Free >> > >> >> > >> Stop any systemvms if running, and alter the state from Running to >> > >> Stopped or Expunging in cloud.vm_instance table. This will recreate >> > >> the systemvm from scratch. >> > >> >> > >> I think this should have you covered, if not, revert back the mysql >> db. >> > >> Do mysql dump and grep for IPs that should have been released. >> > >> >> > >> Let me know, >> > >> >> > >> Regards >> > >> ilya >> > >> >> > >> >> > >> >> > >> > -----Original Message----- >> > >> > From: Shanker Balan [mailto:shanker.ba...@shapeblue.com] >> > >> > Sent: Tuesday, September 24, 2013 9:44 AM >> > >> > To: <users@cloudstack.apache.org> >> > >> > Subject: Re: Management IP address pool exhausted >> > >> > >> > >> > >> > >> > On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <d...@unixmob.com> >> > wrote: >> > >> > >> > >> > > Hello Ilya >> > >> > > >> > >> > > I repeated the issue by rebooting the Xen host without having it >> > >> > > in maintenance mode in CS. At this moment I have the SSVM up and >> > >> > > running and a virtual router (not related to the management IPs, >> I >> > know). >> > >> > > Console Proxy VM can't be started because: >> > >> > > >> > >> > > 2013-09-24 14:56:24,410 INFO >> > >> > > [cloud.vm.VirtualMachineManagerImpl] >> > >> > > (Job-Executor-22:job-72) Insufficient capacity >> > >> > > com.cloud.exception.InsufficientAddressCapacityException: Unable >> > >> > > to get a management ip addressScope=interface com.cloud.dc.Pod; >> > >> > > id=1 >> > >> > > >> > >> > >> > >> > >> > >> > >> > >> > Hi Ilya, >> > >> > >> > >> > I have run into the same issue too. It is easily reproducible by >> > >> > having >> > >> only one >> > >> > host in the Zone and rebooting it without enabling maintenance mode >> > >> > on >> > >> it. >> > >> > >> > >> > This issue exists for both XenServer and KVM host when I tried. >> > >> > >> > >> > >> > >> > -- >> > >> > @shankerbalan >> > >> > >> > >> > M: +91 98860 60539 | O: +91 (80) 67935867 >> > >> > shanker.ba...@shapeblue.com | www.shapeblue.com | >> > >> > Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit >> > >> > 2201A, World Trade Centre, Bangalore - 560 055 >> > >> > >> > >> > This email and any attachments to it may be confidential and are >> > >> intended >> > >> > solely for the use of the individual to whom it is addressed. Any >> > >> > views >> > >> or >> > >> > opinions expressed are solely those of the author and do not >> > >> > necessarily represent those of Shape Blue Ltd or related companies. >> > >> > If you are not >> > >> the >> > >> > intended recipient of this email, you must neither take any action >> > >> > based upon its contents, nor copy or show it to anyone. Please >> > >> > contact the >> > >> sender if >> > >> > you believe you have received this email in error. Shape Blue Ltd >> > >> > is a company incorporated in England & Wales. ShapeBlue Services >> > >> > India LLP >> > >> is a >> > >> > company incorporated in India and is operated under license from >> > >> > Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company >> > >> > incorporated >> > >> in >> > >> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue >> > >> > is a registered trademark. >> > >> >> > >> >> > >> >> > > >> >> >
