[Bug 1244694] Re: [SRU] Creating snapshot fails due to nonexistent temporary directory
Yes, I will revert changes test this. Just did an update see a large number of changes to openstack. I'll apply that, disable patch, verify failure an then apply proposed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: [SRU] Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
I'm using RBD (their volume interface) and not a file system. However, it appears that this IS working -- I restarted the server rather than just the services and now I can correctly snapshot a running instance. This is all on a 13.10 base system, so I must have forgotten to restart some service. Is this patch being pushed into the 13.10 releases? If not, I'll manually update the other servers in my (small) cluster. Btw, I looked through the AUDIT log for things that were DENIED and it looks like qemu-ifdown is not in the libvirt-qemu file -- is this something that should be added? I've included the log output below root@zfs2:/var/log/audit# grep -i denied audit.log type=AVC msg=audit(1389036336.913:4605): apparmor=DENIED operation=exec parent=6994 pro\ file=libvirt-1f2b61f6-0cd1-4bc0-b3fe-4575e0478e0a name=/etc/qemu-ifdown pid=11733 comm=\ qemu-system-x86 requested_mask=x denied_mask=x fsuid=110 ouid=0 type=AVC msg=audit(1389036373.338:4920): apparmor=DENIED operation=capable parent=1 pro\ file=/usr/sbin/libvirtd pid=4173 comm=libvirtd pid=4173 comm=libvirtd capability=29 \ capname=audit_write type=AVC msg=audit(1389036968.075:7337): apparmor=DENIED operation=exec parent=12382 pr\ ofile=libvirt-97be2846-7e24-4538-90bf-d3783fece0ef name=/etc/qemu-ifdown pid=15063 comm\ =qemu-system-x86 requested_mask=x denied_mask=x fsuid=110 ouid=0 root@zfs2:/var/log/audit# -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
I added that to virt-aa-helper but it's still broken. The libvirt/apparmor file is below as is the log output. I don't know if it matters, but I'm using CEPH for volume store. The instance-specific apparmour file is # DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT. /var/log/libvirt/**/instance-00ea.log w, /var/lib/libvirt/**/instance-00ea.monitor rw, /var/run/libvirt/**/instance-00ea.pid rwk, /run/libvirt/**/instance-00ea.pid rwk, /var/run/libvirt/**/*.tunnelmigrate.dest.instance-00ea rw, /run/libvirt/**/*.tunnelmigrate.dest.instance-00ea rw, /var/lib/nova/instances/snapshots/tmpJXEx_R/ca1ff017f53741eea3ee7532205ff7fe.delta rw, /var/lib/nova/instances/e2b47b8a-19ae-40bd-8bea-fdb885b1e49b/console.log rw, /var/lib/nova/instances/e2b47b8a-19ae-40bd-8bea-fdb885b1e49b/console.log rw, = The log output is: 2014-01-03 18:58:16.242 2255 INFO nova.compute.resource_tracker [-] Compute_service record updated for slab-07b:slab-07b 2014-01-03 18:58:25.707 2255 AUDIT nova.compute.manager [req-b2f5930e-2235-4bdf-ad74-c15809b58aac e9bb2814b1364a738d7c313a37a18b60 305f4fa9adcc48b09b8d07d\ f5b247df7] [instance: e2b47b8a-19ae-40bd-8bea-fdb885b1e49b] instance snapshotting 2014-01-03 18:58:25.927 2255 INFO nova.virt.libvirt.driver [req-b2f5930e-2235-4bdf-ad74-c15809b58aac e9bb2814b1364a738d7c313a37a18b60 305f4fa9adcc48b09b8d\ 07df5b247df7] [instance: e2b47b8a-19ae-40bd-8bea-fdb885b1e49b] Beginning live snapshot process 2014-01-03 18:58:26.987 2255 INFO nova.virt.libvirt.driver [req-b2f5930e-2235-4bdf-ad74-c15809b58aac e9bb2814b1364a738d7c313a37a18b60 305f4fa9adcc48b09b8d\ 07df5b247df7] [instance: e2b47b8a-19ae-40bd-8bea-fdb885b1e49b] Snapshot extracted, beginning image upload 2014-01-03 18:58:27.395 2255 ERROR nova.openstack.common.rpc.amqp [req-b2f5930e-2235-4bdf-ad74-c15809b58aac e9bb2814b1364a738d7c313a37a18b60 305f4fa9adcc4\ 8b09b8d07df5b247df7] Exception during message handling 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py, line 461, i\ n _process_data 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp **args) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py, line \ 172, in dispatch 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 353, in decorate\ d_function 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 90, in wrapped 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp payload) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.py, line 73, in wrapped 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 243, in decorate\ d_function 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp pass 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 229, in decorate\ d_function 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 271, in decorate\ d_function 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 258, in decorate\ d_function 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 319, in decorate\ d_function 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp % image_id, instance=instance) 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/manager.py, line 309, in decorate\ d_function 2014-01-03 18:58:27.395 2255 TRACE nova.openstack.common.rpc.amqp *args, **kwargs)
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
I concur that the apparmor solution works. I haven't tried to figure out if it's a race or not. Is this an OpenStack issue or an Ubuntu issue? Sounds like OpenStack to me. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
I concur that the apparmor solution works. I haven't tried to figure out if it's a race or not. Is this an OpenStack issue or an Ubuntu issue? Sounds like OpenStack to me. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
Daniel Speichert (dasp) - were you using a ceph/RBD backend? Were you trying to snapshot a QCOW image? Try it with a RAW. Openstack has issues snapping QCOW (I think there's a cinder bug filed on this). I can create a snapshow of RAW images and create a volume from that snapshot. I can not, however, boot directly from the snap-shotted image. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
Daniel Speichert (dasp) - were you using a ceph/RBD backend? Were you trying to snapshot a QCOW image? Try it with a RAW. Openstack has issues snapping QCOW (I think there's a cinder bug filed on this). I can create a snapshow of RAW images and create a volume from that snapshot. I can not, however, boot directly from the snap-shotted image. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
In my previous comment, I mentioned that this was a Havana install - that's incorrect. It's grizzley from Ubuntu 13.10. Daniel Speichert (dasp), did you check the contents of the snapshot that you were able to make when the system was shut off? I too was able to make a snapshot, but I had modified /etc/apt/sources.list prior to sync'ing and shutting the virtualized system. When I booted up the original server image, the changes to /etc/apt/sources.list were still in the virtualized system. But, when I shut it down, took a snapshot and booted from the snapshot, the contents had reverted to the original contents of /etc/apt/sources.list -- i.e. it appeared that the snapshot was using the underlying volume and not the modifications to the booted image. I have no idea why this occurs, but wanted to see if you had a similar experience with your setup? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
double checking indicates that this may be specific to e.g. /etc/apt/sources.list -- it might be that the image I'm using resets that. I created a file (/foobar/baz), snapped and booted from the snap. The /etc/apt/sources.list was reset to default mirror (http://nova.clouds.archive.ubuntu.com/ubuntu/) but the /foobar/baz directory was there the package updates I had applied were also there. So, it's just the issue of not being able to do a snap of a running system. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
In my previous comment, I mentioned that this was a Havana install - that's incorrect. It's grizzley from Ubuntu 13.10. Daniel Speichert (dasp), did you check the contents of the snapshot that you were able to make when the system was shut off? I too was able to make a snapshot, but I had modified /etc/apt/sources.list prior to sync'ing and shutting the virtualized system. When I booted up the original server image, the changes to /etc/apt/sources.list were still in the virtualized system. But, when I shut it down, took a snapshot and booted from the snapshot, the contents had reverted to the original contents of /etc/apt/sources.list -- i.e. it appeared that the snapshot was using the underlying volume and not the modifications to the booted image. I have no idea why this occurs, but wanted to see if you had a similar experience with your setup? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
double checking indicates that this may be specific to e.g. /etc/apt/sources.list -- it might be that the image I'm using resets that. I created a file (/foobar/baz), snapped and booted from the snap. The /etc/apt/sources.list was reset to default mirror (http://nova.clouds.archive.ubuntu.com/ubuntu/) but the /foobar/baz directory was there the package updates I had applied were also there. So, it's just the issue of not being able to do a snap of a running system. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1244694 Title: Creating snapshot fails due to nonexistent temporary directory To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
I'm having the same problem - I had a working Havana setup (with Ceph/rbd as the cinder storage) and within the last few days, whammo, this starts occuring. I had updated ubuntu 13.10 patches across parts of the cluster. nova-compute log from the node with the exception below 2013-11-03 08:37:03.487 1796 ERROR nova.openstack.common.rpc.amqp [req-16923a1f-cf2b-43fb-bf81-3cdceb698fb0 f2b1d092d8144d\ 17aa1c51eecfd067bb 305f4fa9adcc48b09b8d07df5b247df7] Exception during message handling 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/openstack/\ common/rpc/amqp.py, line 461, in _process_data 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp **args) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/openstack/\ common/rpc/dispatcher.py, line 172, in dispatch 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 353, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.\ py, line 90, in wrapped 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp payload) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.\ py, line 73, in wrapped 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 243, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp pass 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 229, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 271, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 258, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 319, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp % image_id, instance=instance) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 309, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp *args, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 2293, in snapshot_instance 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp task_states.IMAGE_SNAPSHOT) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 2324, in _snapshot_instance 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp update_task_state) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvi\ rt/driver.py, line 1397, in snapshot 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp image_format) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvi\ rt/driver.py, line 1483, in _live_snapshot 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/eventlet/tpool.\ py, line 179, in doit 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp result = proxy_call(self._autowrap, f, *args, **kwar\ gs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/eventlet/tpool.\ py, line 139, in proxy_call 2013-11-03 08:37:03.487 1796
[Bug 1244694] Re: Creating snapshot fails due to nonexistent temporary directory
I'm having the same problem - I had a working Havana setup (with Ceph/rbd as the cinder storage) and within the last few days, whammo, this starts occuring. I had updated ubuntu 13.10 patches across parts of the cluster. nova-compute log from the node with the exception below 2013-11-03 08:37:03.487 1796 ERROR nova.openstack.common.rpc.amqp [req-16923a1f-cf2b-43fb-bf81-3cdceb698fb0 f2b1d092d8144d\ 17aa1c51eecfd067bb 305f4fa9adcc48b09b8d07df5b247df7] Exception during message handling 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/openstack/\ common/rpc/amqp.py, line 461, in _process_data 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp **args) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/openstack/\ common/rpc/dispatcher.py, line 172, in dispatch 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 353, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.\ py, line 90, in wrapped 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp payload) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/exception.\ py, line 73, in wrapped 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 243, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp pass 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 229, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 271, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 258, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 319, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp % image_id, instance=instance) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 309, in decorated_function 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp *args, **kwargs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 2293, in snapshot_instance 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp task_states.IMAGE_SNAPSHOT) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/compute/ma\ nager.py, line 2324, in _snapshot_instance 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp update_task_state) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvi\ rt/driver.py, line 1397, in snapshot 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp image_format) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/nova/virt/libvi\ rt/driver.py, line 1483, in _live_snapshot 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/eventlet/tpool.\ py, line 179, in doit 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp result = proxy_call(self._autowrap, f, *args, **kwar\ gs) 2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File /usr/lib/python2.7/dist-packages/eventlet/tpool.\ py, line 139, in proxy_call 2013-11-03 08:37:03.487 1796
