Re: [ceph-users] qemu-kvm and cloned rbd image
On 03/05/2015 07:19 PM, Josh Durgin wrote: client.libvirt key: caps: [mon] allow r caps: [osd] allow class-read object_prefix rbd_children, allow rw class-read pool=rbd This includes everything except class-write on the pool you're using. You'll need that so that a copy_up call (used just for clones) works. That's what was getting a permissions error. You can use rwx for short. Josh thanks! That was the problem indeed. I removed class-write capability because I also use this user as the default for ceph cli commands. Without class-write this user can't erase an existing image from the pool, while at the same time being able to create new ones. I should probably come up with a better scheme if I am to utilize cloned images. Thanks again! -Kostas ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] qemu-kvm and cloned rbd image
On 03/05/2015 03:40 AM, Josh Durgin wrote: It looks like your libvirt rados user doesn't have access to whatever pool the parent image is in: librbd::AioRequest: write 0x7f1ec6ad6960 rbd_data.24413d1b58ba.0186 1523712~4096 should_complete: r = -1 -1 is EPERM, for operation not permitted. Check the libvirt user capabilites shown in ceph auth list - it should have at least r and class-read access to the pool storing the parent image. You can update it via the 'ceph auth caps' command. Josh, All images, parent, snapshot and clone reside on the same pool (libvirt-pool *) and the user (libvirt) seems to have the proper capabilities. See: client.libvirt key: caps: [mon] allow r caps: [osd] allow class-read object_prefix rbd_children, allow rw class-read pool=rbd This same pool contains other (flat) images used to back my production VMs. They are all accessed with this same user and there have been no problems so far. I just can't seem able to use cloned images. -K. * In my original email describing the problem I used 'rbd' instead of 'libvirt-pool' for the pool name for simplicity. As more and more configuration items are requested, it makes more sense to use the real pool name to avoid causing any misconceptions. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] qemu-kvm and cloned rbd image
On 03/03/2015 05:53 PM, Jason Dillaman wrote: Your procedure appears correct to me. Would you mind re-running your cloned image VM with the following ceph.conf properties: [client] rbd cache off debug rbd = 20 log file = /path/writeable/by/qemu.$pid.log If you recreate the issue, would you mind opening a ticket at http://tracker.ceph.com/projects/rbd/issues? Jason, Thanks for the reply. Recreating the issue is not a problem, I can reproduce it any time. The log file was getting a bit large, I destroyed the guest after letting it thrash for about ~3 minutes, plenty of time to hit the problem. I've uploaded it at: http://paste.scsys.co.uk/468868 (~19MB) Do you really think this is a bug and not an err on my side? -K. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] qemu-kvm and cloned rbd image
On 03/02/2015 04:16 AM, koukou73gr wrote: Hello, Today I thought I'd experiment with snapshots and cloning. So I did: rbd import --image-format=2 vm-proto.raw rbd/vm-proto rbd snap create rbd/vm-proto@s1 rbd snap protect rbd/vm-proto@s1 rbd clone rbd/vm-proto@s1 rbd/server And then proceeded to create a qemu-kvm guest with rbd/server as its backing store. The guest booted but as soon as it got to mount the root fs, things got weird: What does the qemu command line look like? [...] scsi2 : Virtio SCSI HBA scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK1.5. PQ: 0 ANSI: 5 sd 2:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sda1 sda2 sd 2:0:0:0: [sda] Attached SCSI disk dracut: Scanning devices sda2 for LVM logical volumes vg_main/lv_swap vg_main/lv_root dracut: inactive '/dev/vg_main/lv_swap' [1.00 GiB] inherit dracut: inactive '/dev/vg_main/lv_root' [6.50 GiB] inherit EXT4-fs (dm-1): INFO: recovery required on readonly filesystem This suggests the disk is being exposed as read-only via QEMU, perhaps via qemu's snapshot or other options. You can use a clone in exactly the same way as any other rbd image. If you're running QEMU manually, for example, something like: qemu-kvm -drive file=rbd:rbd/server,format=raw,cache=writeback is fine for using the clone. QEMU is supposed to be unaware of any snapshots, parents, etc. at the rbd level. Josh ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] qemu-kvm and cloned rbd image
Hi Josh, Thanks for taking a look at this. I 'm answering your questions inline. On 03/04/2015 10:01 PM, Josh Durgin wrote: [...] And then proceeded to create a qemu-kvm guest with rbd/server as its backing store. The guest booted but as soon as it got to mount the root fs, things got weird: What does the qemu command line look like? I am using libvirt, so I'll be copy-pasting from the log file: LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm -name server -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Penryn,+dca,+pdcm,+xtpr,+tm2,+est,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid ee13f9a0-b7eb-93fd-aa8c-18da9e23ba5c -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/server.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=nc,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -drive file=rbd:libvirt-pool/server:id=libvirt:key=AQAeDqRTQEknIhAA5Gqfl/CkWIfh+nR01hEgzA==:auth_supported=cephx\;none,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:73:98:a9,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 [...] scsi2 : Virtio SCSI HBA scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK1.5. PQ: 0 ANSI: 5 sd 2:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sda1 sda2 sd 2:0:0:0: [sda] Attached SCSI disk dracut: Scanning devices sda2 for LVM logical volumes vg_main/lv_swap vg_main/lv_root dracut: inactive '/dev/vg_main/lv_swap' [1.00 GiB] inherit dracut: inactive '/dev/vg_main/lv_root' [6.50 GiB] inherit EXT4-fs (dm-1): INFO: recovery required on readonly filesystem This suggests the disk is being exposed as read-only via QEMU, perhaps via qemu's snapshot or other options. You're right, the disk does seem R/O but also corrupt. The disk image was cleanly unmounted before creating the snapshot and cloning it. What is more, if I just flatten the image and start the guest again it boots fine and there is no recovery needed on the fs. There are also a some: block I/O error in device 'drive-scsi0-0-0-0': Operation not permitted (1) messages logged in /var/log/libvirt/qemu/server.log You can use a clone in exactly the same way as any other rbd image. If you're running QEMU manually, for example, something like: qemu-kvm -drive file=rbd:rbd/server,format=raw,cache=writeback is fine for using the clone. QEMU is supposed to be unaware of any snapshots, parents, etc. at the rbd level. In a sense, the parameters passed to QEMU from libvirt boil down to your suggested command line. I think it should work as well, it is written all over the place :) I'm a still a newbie wrt ceph, maybe I am missing something flat-out obvious. Thanks for your time, -K. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] qemu-kvm and cloned rbd image
On 03/04/2015 01:36 PM, koukou73gr wrote: On 03/03/2015 05:53 PM, Jason Dillaman wrote: Your procedure appears correct to me. Would you mind re-running your cloned image VM with the following ceph.conf properties: [client] rbd cache off debug rbd = 20 log file = /path/writeable/by/qemu.$pid.log If you recreate the issue, would you mind opening a ticket at http://tracker.ceph.com/projects/rbd/issues? Jason, Thanks for the reply. Recreating the issue is not a problem, I can reproduce it any time. The log file was getting a bit large, I destroyed the guest after letting it thrash for about ~3 minutes, plenty of time to hit the problem. I've uploaded it at: http://paste.scsys.co.uk/468868 (~19MB) It looks like your libvirt rados user doesn't have access to whatever pool the parent image is in: librbd::AioRequest: write 0x7f1ec6ad6960 rbd_data.24413d1b58ba.0186 1523712~4096 should_complete: r = -1 -1 is EPERM, for operation not permitted. Check the libvirt user capabilites shown in ceph auth list - it should have at least r and class-read access to the pool storing the parent image. You can update it via the 'ceph auth caps' command. Josh ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] qemu-kvm and cloned rbd image
Your procedure appears correct to me. Would you mind re-running your cloned image VM with the following ceph.conf properties: [client] rbd cache off debug rbd = 20 log file = /path/writeable/by/qemu.$pid.log If you recreate the issue, would you mind opening a ticket at http://tracker.ceph.com/projects/rbd/issues? Thanks, -- Jason Dillaman Red Hat dilla...@redhat.com http://www.redhat.com - Original Message - From: koukou73gr koukou7...@yahoo.com To: ceph-users@lists.ceph.com Sent: Monday, March 2, 2015 7:16:08 AM Subject: [ceph-users] qemu-kvm and cloned rbd image Hello, Today I thought I'd experiment with snapshots and cloning. So I did: rbd import --image-format=2 vm-proto.raw rbd/vm-proto rbd snap create rbd/vm-proto@s1 rbd snap protect rbd/vm-proto@s1 rbd clone rbd/vm-proto@s1 rbd/server And then proceeded to create a qemu-kvm guest with rbd/server as its backing store. The guest booted but as soon as it got to mount the root fs, things got weird: [...] scsi2 : Virtio SCSI HBA scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK1.5. PQ: 0 ANSI: 5 sd 2:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sda1 sda2 sd 2:0:0:0: [sda] Attached SCSI disk dracut: Scanning devices sda2 for LVM logical volumes vg_main/lv_swap vg_main/lv_root dracut: inactive '/dev/vg_main/lv_swap' [1.00 GiB] inherit dracut: inactive '/dev/vg_main/lv_root' [6.50 GiB] inherit EXT4-fs (dm-1): INFO: recovery required on readonly filesystem EXT4-fs (dm-1): write access will be enabled during recovery sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 b0 e0 d8 00 00 08 00 Buffer I/O error on device dm-1, logical block 1058331 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 6f ba c8 00 00 08 00 [ ... snip ... snip ... more or less the same messages ] end_request: I/O error, dev sda, sector 3129880 end_request: I/O error, dev sda, sector 11518432 end_request: I/O error, dev sda, sector 3194664 end_request: I/O error, dev sda, sector 3129824 end_request: I/O error, dev sda, sector 3194376 end_request: I/O error, dev sda, sector 11579664 end_request: I/O error, dev sda, sector 3129448 end_request: I/O error, dev sda, sector 3197856 end_request: I/O error, dev sda, sector 3129400 end_request: I/O error, dev sda, sector 7385360 end_request: I/O error, dev sda, sector 11515912 end_request: I/O error, dev sda, sector 11514112 __ratelimit: 12 callbacks suppressed sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 af b0 80 00 00 10 00 __ratelimit: 12 callbacks suppressed __ratelimit: 13 callbacks suppressed Buffer I/O error on device dm-1, logical block 1048592 lost page write due to I/O error on dm-1 Buffer I/O error on device dm-1, logical block 1048593 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f bf 00 00 00 08 00 Buffer I/O error on device dm-1, logical block 480 lost page write due to I/O error on dm-1 [... snip... more of the same ... ] Buffer I/O error on device dm-1, logical block 475 lost page write due to I/O error on dm-1 Buffer I/O error on device dm-1, logical block 476 lost page write due to I/O error on dm-1 Buffer I/O error on device dm-1, logical block 477 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f be 30 00 00 10 00 Buffer I/O error on device dm-1, logical block 454 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f be 10 00 00 18 00 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f
[ceph-users] qemu-kvm and cloned rbd image
Hello, Today I thought I'd experiment with snapshots and cloning. So I did: rbd import --image-format=2 vm-proto.raw rbd/vm-proto rbd snap create rbd/vm-proto@s1 rbd snap protect rbd/vm-proto@s1 rbd clone rbd/vm-proto@s1 rbd/server And then proceeded to create a qemu-kvm guest with rbd/server as its backing store. The guest booted but as soon as it got to mount the root fs, things got weird: [...] scsi2 : Virtio SCSI HBA scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK1.5. PQ: 0 ANSI: 5 sd 2:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB) sd 2:0:0:0: [sda] Write Protect is off sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sda1 sda2 sd 2:0:0:0: [sda] Attached SCSI disk dracut: Scanning devices sda2 for LVM logical volumes vg_main/lv_swap vg_main/lv_root dracut: inactive '/dev/vg_main/lv_swap' [1.00 GiB] inherit dracut: inactive '/dev/vg_main/lv_root' [6.50 GiB] inherit EXT4-fs (dm-1): INFO: recovery required on readonly filesystem EXT4-fs (dm-1): write access will be enabled during recovery sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] abort sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 b0 e0 d8 00 00 08 00 Buffer I/O error on device dm-1, logical block 1058331 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 6f ba c8 00 00 08 00 [ ... snip ... snip ... more or less the same messages ] end_request: I/O error, dev sda, sector 3129880 end_request: I/O error, dev sda, sector 11518432 end_request: I/O error, dev sda, sector 3194664 end_request: I/O error, dev sda, sector 3129824 end_request: I/O error, dev sda, sector 3194376 end_request: I/O error, dev sda, sector 11579664 end_request: I/O error, dev sda, sector 3129448 end_request: I/O error, dev sda, sector 3197856 end_request: I/O error, dev sda, sector 3129400 end_request: I/O error, dev sda, sector 7385360 end_request: I/O error, dev sda, sector 11515912 end_request: I/O error, dev sda, sector 11514112 __ratelimit: 12 callbacks suppressed sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 af b0 80 00 00 10 00 __ratelimit: 12 callbacks suppressed __ratelimit: 13 callbacks suppressed Buffer I/O error on device dm-1, logical block 1048592 lost page write due to I/O error on dm-1 Buffer I/O error on device dm-1, logical block 1048593 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f bf 00 00 00 08 00 Buffer I/O error on device dm-1, logical block 480 lost page write due to I/O error on dm-1 [... snip... more of the same ... ] Buffer I/O error on device dm-1, logical block 475 lost page write due to I/O error on dm-1 Buffer I/O error on device dm-1, logical block 476 lost page write due to I/O error on dm-1 Buffer I/O error on device dm-1, logical block 477 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f be 30 00 00 10 00 Buffer I/O error on device dm-1, logical block 454 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f be 10 00 00 18 00 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f be 08 00 00 08 00 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 2f bd 88 00 00 08 00 __ratelimit: 5 callbacks suppressed Buffer I/O error on device dm-1, logical block 433 lost page write due to I/O error on dm-1 sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated sd 2:0:0:0: [sda] CDB: Write(10): 2a 00
