updated subject with [SOLVED] tag On 16-Oct-2013, at 10:48 AM, Daniel Hertanu <d...@unixmob.com> wrote:
> I think I've seen a suggestion earlier about adding the world "solved" in > the subject whenever someone finds a solution for its problem. > > So, regarding this management IP pool issue, I'm not sure that's the > definitive solution but it did work for me. Having all management IPs used, > CloudStack was failing to start the console proxy VM. > I created a dump of cloud DB and start looking for my management IPs. I > found a table called op_dc_ip_address_alloc with references to all my > management IPs. Long story short, 2 of the 4 IP addresses were still > showing as reserved so I simply run this: > > mysql> update op_dc_ip_address_alloc set > nic_id=NULL,reservation_id=NULL,taken=NULL where id=1; > > The id in the command above was the id corresponding to the IPs that were > still showing as reserved. > > In the next 30 seconds CloudStack started also the console proxy VM and > everything came back to normal. > > I suggest creating a full backup of the database before changing anything > though. > > > Daniel > > On Thu, Sep 26, 2013 at 2:24 AM, Daniel Hertanu <d...@unixmob.com> wrote: > >> I posted here: http://pastebin.com/uJ6jR73D the logs, maybe someone else >> have seen this behavior before. Just to remind the issue, CS 4.1, XenServer >> 6.1, when the only existing host is rebooted without being in maintenance >> mode (and stays down for a while), all the management IPs get used (in this >> case there were 171 management IPs, all show as in use now). >> >> Thanks, >> >> Daniel >> >> >> On Wed, Sep 25, 2013 at 8:56 PM, Musayev, Ilya <imusa...@webmd.net> wrote: >> >>> Dan, >>> >>> Ping me on IRC, nick serverchief >>> >>> Regards >>> ilya >>> >>> >>>> -----Original Message----- >>>> From: Daniel Hertanu [mailto:d...@unixmob.com] >>>> Sent: Wednesday, September 25, 2013 6:44 AM >>>> To: users@cloudstack.apache.org >>>> Subject: Re: Management IP address pool exhausted >>>> >>>> Hi >>>> >>>> So I re-installed CS, reboot Xen host without entering it first in >>> Maintenace >>>> Mode, then got the management IP pool fully used. I followed again your >>>> indications but to no avail. >>>> In the meantime SSVM has reached number 112 (s-112-VM). >>>> >>>> I would really appreciate any other idea. >>>> >>>> Thanks, >>>> >>>> Daniel >>>> >>>> >>>> On Wed, Sep 25, 2013 at 9:36 AM, Daniel Hertanu <d...@unixmob.com> >>>> wrote: >>>> >>>>> Hello Ilya >>>>> >>>>> I submit an issue describing the bug: >>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-4735 (I picked CS >>>>> version 4.1.0 as 4.1.1 is reported as non existent, but I've seen this >>>>> behavior before since version 3). >>>>> >>>>> I followed your indications but it didn't work. After changing those >>>>> records in the database, CS would still give the same error. I revert >>>>> back to the initial db content, but this time nothing came back >>>>> (initially the SSVM was still running, it was CPVM that didn't start). >>>>> After this step I tried some tricks on my own and I ended up with a >>> botched >>>> environment. >>>>> So I'll do a fresh install, reboot the host and try your suggestions >>>>> again. >>>>> >>>>> Thanks, >>>>> >>>>> Daniel >>>>> >>>>> >>>>> On Tue, Sep 24, 2013 at 10:46 PM, Musayev, Ilya >>>> <imusa...@webmd.net>wrote: >>>>> >>>>>> I'm not certain if this is hypervisor specific - I've seen the same >>>>>> issue for vSphere - but I though it was me, I think we may need to >>>>>> revise the system vm ip space decommission process. >>>>>> >>>>>> Shanker and/or Daniel, >>>>>> >>>>>> Do you mind creating a ticket in issues.apache.org and write out >>> ways >>>>>> to reproduce this issue. >>>>>> >>>>>> --- >>>>>> >>>>>> *NOTE* this is purely from memory and some notes I had - proceed at >>>>>> your own risk.... >>>>>> >>>>>> >>>>>> >>>>>> In the meantime, lets clean up what you have. >>>>>> >>>>>> The goal is to release the IPs that are still marked as reserved and >>>>>> allocated. >>>>>> >>>>>> I'd recommend installing mysql workbench >>>>>> >>>>>> There are few tables we need to review: >>>>>> 1) Stop cloudstack management server >>>>>> 2) Backup mysql db first. >>>>>> >>>>>> Note the id of management network referenced in cloud.networks >>>>>> Select id from cloud.networks where traffic_type="Management" >>>>>> In my case, Management network has ID of 205 >>>>>> SELECT * FROM cloud.nics where network_id=205; Alter to >>>>>> Deallocating the ones that are marked as Reserved. >>>>>> >>>>>> Do the same for cloud.user_ip_address SELECT * FROM >>>>>> cloud.user_ip_address where source_network_id=205; Alter from >>>>>> Allocated to Free >>>>>> >>>>>> Stop any systemvms if running, and alter the state from Running to >>>>>> Stopped or Expunging in cloud.vm_instance table. This will recreate >>>>>> the systemvm from scratch. >>>>>> >>>>>> I think this should have you covered, if not, revert back the mysql >>> db. >>>>>> Do mysql dump and grep for IPs that should have been released. >>>>>> >>>>>> Let me know, >>>>>> >>>>>> Regards >>>>>> ilya >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Shanker Balan [mailto:shanker.ba...@shapeblue.com] >>>>>>> Sent: Tuesday, September 24, 2013 9:44 AM >>>>>>> To: <users@cloudstack.apache.org> >>>>>>> Subject: Re: Management IP address pool exhausted >>>>>>> >>>>>>> >>>>>>> On 24-Sep-2013, at 6:37 PM, Daniel Hertanu <d...@unixmob.com> >>>> wrote: >>>>>>> >>>>>>>> Hello Ilya >>>>>>>> >>>>>>>> I repeated the issue by rebooting the Xen host without having it >>>>>>>> in maintenance mode in CS. At this moment I have the SSVM up and >>>>>>>> running and a virtual router (not related to the management IPs, >>> I >>>> know). >>>>>>>> Console Proxy VM can't be started because: >>>>>>>> >>>>>>>> 2013-09-24 14:56:24,410 INFO >>>>>>>> [cloud.vm.VirtualMachineManagerImpl] >>>>>>>> (Job-Executor-22:job-72) Insufficient capacity >>>>>>>> com.cloud.exception.InsufficientAddressCapacityException: Unable >>>>>>>> to get a management ip addressScope=interface com.cloud.dc.Pod; >>>>>>>> id=1 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Ilya, >>>>>>> >>>>>>> I have run into the same issue too. It is easily reproducible by >>>>>>> having >>>>>> only one >>>>>>> host in the Zone and rebooting it without enabling maintenance mode >>>>>>> on >>>>>> it. >>>>>>> >>>>>>> This issue exists for both XenServer and KVM host when I tried. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> @shankerbalan >>>>>>> >>>>>>> M: +91 98860 60539 | O: +91 (80) 67935867 >>>>>>> shanker.ba...@shapeblue.com | www.shapeblue.com | >>>>>>> Twitter:@shapeblue ShapeBlue Services India LLP, 22nd floor, Unit >>>>>>> 2201A, World Trade Centre, Bangalore - 560 055 >>>>>>> >>>>>>> This email and any attachments to it may be confidential and are >>>>>> intended >>>>>>> solely for the use of the individual to whom it is addressed. Any >>>>>>> views >>>>>> or >>>>>>> opinions expressed are solely those of the author and do not >>>>>>> necessarily represent those of Shape Blue Ltd or related companies. >>>>>>> If you are not >>>>>> the >>>>>>> intended recipient of this email, you must neither take any action >>>>>>> based upon its contents, nor copy or show it to anyone. Please >>>>>>> contact the >>>>>> sender if >>>>>>> you believe you have received this email in error. Shape Blue Ltd >>>>>>> is a company incorporated in England & Wales. ShapeBlue Services >>>>>>> India LLP >>>>>> is a >>>>>>> company incorporated in India and is operated under license from >>>>>>> Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company >>>>>>> incorporated >>>>>> in >>>>>>> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue >>>>>>> is a registered trademark. >>>>>> >>>>>> >>>>>> >>>>> >>> >>> >>
