If I'm reading it correctly his cmdline says cache=none for the rbd device, so there should be no writeback caching:
file=rbd:Primary-ubuntu-1/c3f90fb4-c1a6-4e99-a2c0-64ae4517412e:id=admin:key=AQDiDbJR2GqPABAAWCcsUQ+UQwK8z9c6LWrizw==:auth_supported=cephx\;none:mon_host=ceph-mon.csprdc.arhont.com\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none If that's actually overriden by ceph.conf setting then that is another bug I guess :-) Jan > On 21 Oct 2015, at 19:46, Jason Dillaman <dilla...@redhat.com> wrote: > > There is an edge case with cloned image writeback caching that occurs after > an attempt to read a non-existent clone RADOS object, followed by a write to > said object, followed by another read. This second read will cause the > cached write to be flushed to the OSD while the appropriate locks are not > being held. This issue is being tracked via an upstream tracker ticket [1]. > > This issue effects librbd clients using v0.94.4 and v9.x. Disabling the > cache or switching to write-through caching (rbd_cache_max_dirty = 0) should > avoid the issue until it is fixed in the next Ceph release. > > [1] http://tracker.ceph.com/issues/13559 > > -- > > Jason Dillaman > > > ----- Original Message ----- > >> From: "Andrei Mikhailovsky" <and...@arhont.com> >> To: ceph-us...@ceph.com >> Sent: Wednesday, October 21, 2015 8:17:39 AM >> Subject: [ceph-users] [urgent] KVM issues after upgrade to 0.94.4 > >> Hello guys, > >> I've upgraded to the latest Hammer release and I've just noticed a massive >> issue after the upgrade ((( > >> I am using ceph for virtual machine rbd storage over cloudstack. I am having >> issues with starting virtual routers. The libvirt error message is: > >> cat r-1407-VM.log >> 2015-10-21 11:04:59.262+0000: starting up >> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin >> QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name r-1407-VM -S -machine >> pc-i440fx-trusty,accel=kvm,usb=off -m 256 -realtime mlock=off -smp >> 1,sockets=1,cores=1,threads=1 -uuid 815d2860-cc7f-475d-bf63-02814c720fe4 >> -no-user-config -nodefaults -chardev >> socket,id=charmonitor,path=/var/lib/libvirt/qemu/r-1407-VM.monitor,server,nowait >> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >> -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device >> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive >> file=rbd:Primary-ubuntu-1/c3f90fb4-c1a6-4e99-a2c0-64ae4517412e:id=admin:key=AQDiDbJR2GqPABAAWCcsUQ+UQwK8z9c6LWrizw==:auth_supported=cephx\;none:mon_host=ceph-mon.csprdc.arhont.com\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none >> -device >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 >> -drive >> file=/usr/share/cloudstack-common/vms/systemvm.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw,cache=none >> -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 >> -netdev tap,fd=54,id=hostnet0,vhost=on,vhostfd=55 -device >> virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:2e:f7:00:18,bus=pci.0,addr=0x3,rombar=0,romfile= >> -netdev tap,fd=56,id=hostnet1,vhost=on,vhostfd=57 -device >> virtio-net-pci,netdev=hostnet1,id=net1,mac=0e:00:a9:fe:01:42,bus=pci.0,addr=0x4,rombar=0,romfile= >> -netdev tap,fd=58,id=hostnet2,vhost=on,vhostfd=59 -device >> virtio-net-pci,netdev=hostnet2,id=net2,mac=06:0c:b6:00:02:13,bus=pci.0,addr=0x5,rombar=0,romfile= >> -chardev pty,id=charserial0 -device >> isa-serial,chardev=charserial0,id=serial0 -chardev >> socket,id=charchannel0,path=/var/lib/libvirt/qemu/r-1407-VM.agent,server,nowait >> -device >> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=r-1407-VM.vport >> -device usb-tablet,id=input0 -vnc 192.168.169.2:10,password -device >> cirrus-vga,id=video0,bus=pci.0,addr=0x2 >> Domain id=42 is tainted: high-privileges >> libust[20136/20136]: Warning: HOME environment variable not set. Disabling >> LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) >> char device redirected to /dev/pts/13 (label charserial0) >> librbd/LibrbdWriteback.cc: In function 'virtual ceph_tid_t >> librbd::LibrbdWriteback::write(const object_t&, const object_locator_t&, >> uint64_t, uint64_t, const SnapContext&, const bufferlist&, utime_t, >> uint64_t, __u32, Context*)' thread 7ffa6b7fe700 time 2015-10-21 >> 12:05:07.901876 >> librbd/LibrbdWriteback.cc: 160: FAILED assert(m_ictx->owner_lock.is_locked()) >> ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a) >> 1: (()+0x17258b) [0x7ffa92ef758b] >> 2: (()+0xa9573) [0x7ffa92e2e573] >> 3: (()+0x3a90ca) [0x7ffa9312e0ca] >> 4: (()+0x3b583d) [0x7ffa9313a83d] >> 5: (()+0x7212c) [0x7ffa92df712c] >> 6: (()+0x9590f) [0x7ffa92e1a90f] >> 7: (()+0x969a3) [0x7ffa92e1b9a3] >> 8: (()+0x4782a) [0x7ffa92dcc82a] >> 9: (()+0x56599) [0x7ffa92ddb599] >> 10: (()+0x7284e) [0x7ffa92df784e] >> 11: (()+0x162b7e) [0x7ffa92ee7b7e] >> 12: (()+0x163c10) [0x7ffa92ee8c10] >> 13: (()+0x8182) [0x7ffa8ec49182] >> 14: (clone()+0x6d) [0x7ffa8e97647d] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to >> interpret this. >> terminate called after throwing an instance of 'ceph::FailedAssertion' >> 2015-10-21 11:05:08.091+0000: shutting down > >> From what I can see, there seem to be an issue with locking >> (librbd/LibrbdWriteback.cc: 160: FAILED >> assert(m_ictx->owner_lock.is_locked())). However, the r-1407-VM virtual >> router is a new router and has not been created or ran before. So, I don't >> see why there is an issue with locking. > >> Could someone please help me determine the cause of the error and how to fix >> it. I've not seen this on 0.94.1. > >> Many thanks > >> Andrei > >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com