Public bug reported: After few days running a VM over RBD, under heavy loads. We get freeze the VMs and get this log:
2016-07-26 16:20:47.069+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10~cloud0 (Openstack Ubuntu Testing Bot <[email protected]> Fri, 15 Apr 2016 08:05:47 +0000), qemu version: 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.24), hostname: tckvmhv01.todoencloud LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name opennebula -S -machine pc-i440fx-trusty,accel=kvm,usb=off -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 7733f4c6-c72d-af36-2e84-cbc44c5c8da0 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-opennebula/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=c,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:sata/opennebula.img:id=sata:key=AQB69U1Xq+bxHhAAcw7ZWZlOjE0Bf88QMsTqMQ==:auth_supported=cephx\;none:mon_host=10.255.128.238\:6789,format=raw,if=none,id=drive-virtio-disk0' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/usr/src/systemrescuecd.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0 ,id=ide0-1-0 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=31 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1c:07:7f,bus=pci.0,addr=0x3 -netdev tap,fd=35,id=hostnet1,vhost=on,vhostfd=36 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:42:b6:74,bus=pci.0,addr=0x6 -netdev tap,fd=37,id=hostnet2,vhost=on,vhostfd=38 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:2b:bb:89,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on char device redirected to /dev/pts/1 (label charserial0) librbd/ExclusiveLock.cc: In function 'std::string librbd::ExclusiveLock<ImageCtxT>::encode_lock_cookie() const [with ImageCtxT = librbd::ImageCtx; std::string = std::basic_string<char>]' thread 7fded06a2a00 time 2016-07-28 23:44:09.845595 librbd/ExclusiveLock.cc: 197: FAILED assert(m_watch_handle != 0) ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9) 1: (()+0x24c4bb) [0x7fdebcbde4bb] 2: (()+0x7cf54) [0x7fdebca0ef54] 3: (()+0x7dadb) [0x7fdebca0fadb] 4: (()+0x7e827) [0x7fdebca10827] 5: (()+0x65d46) [0x7fdebc9f7d46] 6: (()+0x66218) [0x7fdebc9f8218] 7: (rbd_aio_write()+0x29) [0x7fdebc9ec629] 8: (()+0xde816) [0x7fded0108816] 9: (()+0xde910) [0x7fded0108910] 10: (()+0xb69b0) [0x7fded00e09b0] 11: (()+0xbbc44) [0x7fded00e5c44] 12: (()+0xbbc44) [0x7fded00e5c44] 13: (()+0xbc8fc) [0x7fded00e68fc] 14: (()+0xfc36a) [0x7fded012636a] 15: (()+0x49800) [0x7fdeb84ac800] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2016-07-28 21:44:10.296+0000: shutting down root@tckvmhv01:/var/log/libvirt/qemu# dpkg -l | grep ceph ii ceph-common 10.2.0-0ubuntu0.16.04.1~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster ii libcephfs1 10.2.0-0ubuntu0.16.04.1~cloud0 amd64 Ceph distributed file system client library ii python-ceph 10.2.0-0ubuntu0.16.04.1~cloud0 all Meta-package for python libraries for the Ceph libraries ii python-cephfs 10.2.0-0ubuntu0.16.04.1~cloud0 amd64 Python libraries for the Ceph libcephfs library root@tckvmhv01:/var/log/libvirt/qemu# dpkg -l | grep libvirt ii libvirt-bin 1.3.1-1ubuntu10.1~cloud0 amd64 programs for the libvirt library ii libvirt0:amd64 1.3.1-1ubuntu10.1~cloud0 amd64 library for interfacing with different virtualization systems ii python-libvirt 1.3.1-1ubuntu1~cloud0 amd64 libvirt Python bindings root@tckvmhv01:/var/log/libvirt/qemu# uname -a Linux tckvmhv01 3.13.0-86-generic #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux ** Affects: ceph (Ubuntu) Importance: Undecided Status: New ** Tags: ceph libvirtd rbd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1607694 Title: Exclusive-Lock Issue To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1607694/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
