[Bug 1144108] Re: Unable to create instances due to QueuePool limit
** Tags added: canonistack -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1144108 Title: Unable to create instances due to QueuePool limit To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1144108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1144108] Re: Unable to create instances due to QueuePool limit
** Changed in: nova (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1144108 Title: Unable to create instances due to QueuePool limit To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1144108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1144108] Re: Unable to create instances due to QueuePool limit
I still think this is a valid bug. The problem disappeared after nova-network restart and so far it did not come back. What we would like to find out is why did this start happening. There is no proof of this but it may be ralated to long nova-network uptime. Changing the values as you suggested could only delay the problem, there is no guarantee that the root cause is caused by low thresholds. If it was the thresholds why would the problem disappear after restart? Also if you suggest that default values could cause problems should they not be updated? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1144108 Title: Unable to create instances due to QueuePool limit To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1144108/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1144108] [NEW] Unable to create instances due to QueuePool limit
Public bug reported: We had problems creating instances. Attempts were causing the following error on the compute node: 2013-03-01 11:52:41 21023 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling 2013-03-01 11:52:41 21023 ERROR nova.openstack.common.rpc.common [-] Returning exception QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30 to caller 2013-03-01 11:52:41 21023 ERROR nova.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', ' File /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py, line 276, in _process_data\nrval = self.proxy.dispatch(ctxt, version, method, **args)\n', ' File /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py, line 145, in dispatch\nreturn getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File /usr/lib/python2.7/dist-packages/nova/network/manager.py, line 258, in wrapped\nreturn func(self, context, *args, **kwargs)\n', ' File /usr/lib/python2.7/dist-packages/nova/network/manager.py, line 1120, in get_instance_nw_info\ninstance_uuid)\n', ' File /usr/lib/python2.7/dist-packages/nova/db/api.py, line 523, in virtual_interface_get_by_instance\nreturn IMPL.virtual_interface_get_by_instance(context, instance_id)\n', ' File /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py, line 126, in wrapper\nreturn f(*args, **kwargs)\n', ' File /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py, line 138, in wrapper\ndb.instance_get_by_uuid(context, instance_uuid)\n', ' File /usr/lib/python2.7/dist-packages/nova/db/api.py, line 570, in instance_get_by_uuid\nreturn IMPL.instance_get_by_uuid(context, uuid)\n', ' File /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py, line 126, in wrapper\nreturn f(*args, **kwargs)\n', ' File /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py, line 1485, in instance_get_by_uuid\nfilter_by(uuid=uuid).\\\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py, line 2156, in first\nret = list(self[0:1])\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py, line 2023, in __getitem__\nreturn list(res)\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py, line 2227, in __iter__\nreturn self._execute_and_instances(context)\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py, line 2240, in _execute_and_instances\nclose_with_result=True)\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py, line 2231, in _connection_from_session\n**kw)\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py, line 730, in connection\nclose_with_result=close_with_result)\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py, line 736, in _connection_for_bind\nreturn engine.contextual_connect(**kwargs)\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 2490, in contextual_connect\nself.pool.connect(),\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/pool.py, line 224, in connect\n return _ConnectionFairy(self).checkout()\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/pool.py, line 387, in __init__\n rec = self._connection_record = pool._do_get()\n', ' File /usr/lib/python2.7/dist-packages/sqlalchemy/pool.py, line 729, in _do_get\n (self.size(), self.overflow(), self._timeout))\n', 'TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30\n'] And this on the controller: 2013-03-01 11:52:41 21023 DEBUG amqplib [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:70 2013-03-01 11:52:41 21023 DEBUG amqplib [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:484 2013-03-01 11:52:41 21023 DEBUG amqplib [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:70 2013-03-01 11:52:41 21023 DEBUG amqplib [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:70 2013-03-01 11:52:41 21023 DEBUG amqplib [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:70 2013-03-01 11:52:41 21023 DEBUG amqplib [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:484 2013-03-01 11:52:41 21023 DEBUG amqplib [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:90 2013-03-01 11:52:41 21023 DEBUG amqplib [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:90 2013-03-01 11:52:41 21023 DEBUG amqplib [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:484 2013-03-01 11:52:41 21023 DEBUG amqplib [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py:90
[Bug 1134604] [NEW] New IPs not available after subnet changes
Public bug reported: We recently run out IP address in the subnet used for our instances and we had to extend out subnet from /24 to /23. We edited /etc/nova/nova.conf, restared nova-network using /etc/init.d/nova-network restart. ifconfig bf100 was showing correct (new) netmask so was nova-manage network list When new instances were about to use new range they failed to get IPs: On compute node: 2013-02-27 10:46:38 ERROR nova.compute.manager [req- ] [instance: ] Instance failed network setup And corresponding on our controller: 2013-02-27 10:46:38 DEBUG nova.network.manager [req- ] [instance: ] networks retrieved for instance: |[nova.db.sqlalchemy.models.Network object at 0x4997850]| from (pid=2687) allocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:982 2013-02-27 10:46:38 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py, line 275, in _process_data 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py, line 145, in dispatch 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/network/manager.py, line 257, in wrapped 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp return func(self, context, *args, **kwargs) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/network/manager.py, line 320, in allocate_for_instance 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp **kwargs) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/network/manager.py, line 257, in wrapped 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp return func(self, context, *args, **kwargs) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/network/manager.py, line 986, in allocate_for_instance 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp requested_networks=requested_networks) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/network/manager.py, line 213, in _allocate_fixed_ips 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp vpn=vpn, address=address) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/network/manager.py, line 1265, in allocate_fixed_ip 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp instance_ref['uuid']) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/db/api.py, line 442, in fixed_ip_associate_pool 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp instance_uuid, host) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py, line 111, in wrapper 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs) 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py, line 1089, in fixed_ip_associate_pool 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp raise exception.NoMoreFixedIps() 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp NoMoreFixedIps: Zero fixed ips available. 2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp 2013-02-27 10:46:38 ERROR nova.openstack.common.rpc.common [-] Returning exception Zero fixed ips available. to caller After some investigation we realised that fixed_ips table in nova DB only had IPs from the old range. We added new IPs using: INSERT INTO fixed_ips (created_at, deleted, allocated, leased, reserved, network_id, address) VALUES (CURDATE(), 0,0,0,0, 1, xx.xx.xx.xx); which allowed our instances to get IPs from new range. On compute node we are running: ii nova-common 2012.2-0ubuntu5~cloud0 OpenStack Compute - common files ii nova-compute 2012.2-0ubuntu5~cloud0 OpenStack Compute - compute node ii nova-compute-kvm 2012.2-0ubuntu5~cloud0 OpenStack Compute - compute node (KVM) ii python-nova 2012.2-0ubuntu5~cloud0 OpenStack Compute Python libraries ii python-novaclient1:2.9.0-0ubuntu1~cloud0 client library for
[Bug 974667] Re: dnsmasq does not bind to a port
/var/log/syslog copied when dnsmasq was broken. I have just tested kubuntu 11.10 iso on the same VM and it is all fine (however 11.10 does not use dnsmasq). Just to let you know - I amthe person who reported the bug and problem with ISO test so this does not actually impact multiple people. ** Attachment added: /var/log/syslog copied when dnsmasq was broken https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/974667/+attachment/3098266/+files/syslog -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to dnsmasq in Ubuntu. https://bugs.launchpad.net/bugs/974667 Title: dnsmasq does not bind to a port To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/974667/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 974667] [NEW] dnsmasq does not bind to a port
Public bug reported: When I boot into the installation CD (kubuntu-12.04-beta2-desktop-i386) DNS service does not work properly. resolv.conf points to 127.0.0.1 which seems correct, /var/run/nm-dns-dnsmasq.conf contains correct nameserver IP. I think it is to do with the fact that nothing listens on port 53. kubuntu@kubuntu:~$ sudo netstat -nlptu|grep 53 udp0 0 0.0.0.0:53530.0.0.0:* 1173/avahi-daemon: udp6 0 0 :::5353 :::* 1173/avahi-daemon: kubuntu@kubuntu:~$ dnsmasq is running: kubuntu@kubuntu:~$ ps -ef|grep /usr/sbin/dnsmasq nobody1275 1191 0 21:17 ?00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.0.1 --conf-file=/var/run/nm-dns-dnsmasq.conf --cache-size=0 --proxy-dnssec When I do lsof it show interesting results: kubuntu@kubuntu:~$ sudo lsof|grep 1275 info ommited dnsmasq 1275 nobody3r sock0,7 0t0 11578 can't identify protocol dnsmasq 1275 nobody4w FIFO0,8 0t0 11587 pipe dnsmasq 1275 nobody5u FIFO0,8 0t0 11587 pipe dnsmasq 1275 nobody6u unix 0xec3126c0 0t0 11589 socket If I kill the process, recreate /var/run/nm-dns-dnsmasq.conf and start it on the CLI manually everything is fine (I can resolve hostname and and dnsmasq binds to the port 53) kubuntu@kubuntu:~$ sudo netstat -nlput|grep 53 I am using VirtualBox 4.1.12 for testing with almost default config of my VM. It happens every time so if you need any more details I can provide them easily. ProblemType: Bug DistroRelease: Ubuntu 12.04 Package: dnsmasq-base 2.59-4 ProcVersionSignature: Ubuntu 3.2.0-20.33-generic-pae 3.2.12 Uname: Linux 3.2.0-20-generic-pae i686 ApportVersion: 1.95-0ubuntu1 Architecture: i386 CasperVersion: 1.312 Date: Thu Apr 5 21:22:21 2012 LiveMediaBuild: Kubuntu 12.04 LTS Precise Pangolin - Beta i386 (20120328) ProcEnviron: LANGUAGE= TERM=xterm LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: dnsmasq UpgradeStatus: No upgrade log present (probably fresh install) ** Affects: dnsmasq (Ubuntu) Importance: Undecided Status: New ** Tags: apport-bug i386 precise -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to dnsmasq in Ubuntu. https://bugs.launchpad.net/bugs/974667 Title: dnsmasq does not bind to a port To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/974667/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 974667] Re: dnsmasq does not bind to a port
-- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to dnsmasq in Ubuntu. https://bugs.launchpad.net/bugs/974667 Title: dnsmasq does not bind to a port To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/974667/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
