Public bug reported: logstash query: message:"cgroup\: No space left on device" AND filename:logs*screen-n-cpu.txt
http://logs.openstack.org/12/80412/8/check/check-tempest-dsvm-postgres- full/f9f6158/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-03-21_17_45_12_490 ERROR nova.compute.manager [req-630b71d6-0fbe-4e9e-99fe-019da7d29a3a FixedIPsNegativeTestJson-475659359 FixedIPsNegativeTestJson-265680949] [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Error: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Traceback (most recent call last): 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1304, in _build_instance 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] set_access_ip=set_access_ip) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 394, in decorated_function 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] return function(self, context, *args, **kwargs) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1716, in _spawn 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] LOG.exception(_('Instance failed to spawn'), instance=instance) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__ 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] six.reraise(self.type_, self.value, self.tb) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1713, in _spawn 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] block_device_info) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2241, in spawn 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] block_device_info) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3621, in _create_domain_and_network 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] power_on=power_on) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3531, in _create_domain 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] domain.XMLDesc(0)) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__ 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] six.reraise(self.type_, self.value, self.tb) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3526, in _create_domain 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] domain.createWithFlags(launch_flags) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] result = proxy_call(self._autowrap, f, *args, **kwargs) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] rv = execute(f,*args,**kwargs) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] rv = meth(*args,**kwargs) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 581, in createWithFlags 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] libvirtError: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device 26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] 26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Traceback (most recent call last): 26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1184, in _run_instance 26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] instance, image_meta, legacy_bdm_in_spec) 26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] File "/opt/stack/new/nova/nova/compute/manager.py", line 1354, in _build_instance 26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] reason=unicode(exc_info[1])) 26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] RescheduledException: Build of instance 3f281136-ed69-4bfb-bf36-a7d4aa1c0640 was re-scheduled: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device 26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] This started right around the time we started using: https://review.openstack.org/#/c/79816/ (serge's libvirt fix for https://bugs.launchpad.net/nova/+bug/1254872) ** Affects: nova Importance: Undecided Status: New ** Affects: libvirt (Ubuntu) Importance: Undecided Status: New ** Also affects: libvirt (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to libvirt in Ubuntu. https://bugs.launchpad.net/bugs/1295876 Title: libvirtError: internal error unable to add domain xxx to cgroup: No space left on device To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1295876/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
