- Le 17 Sep 19, à 18:27, Daniel Berteaud a
écrit :
> Hi there.
> I'm working on moving my NFS setup to ZFS over iSCSI. I'm using a CentOS 7.6
> box
> with ZoL 0.8.1, with the LIO backend (but this shouldn't be relevent, see
> further). For the PVE side, I'm running PVE6 with all updates applied.
> Except a few minor issues I found in the LIO backend (for which I sent a patch
> serie earlier today), most things do work nicely. Except one which is
> important
> to me : I can't move disk from ZFS over iSCSI to any other storage.
> Destination
> storage type doesn't matter, but the porblem is 100% reproducible when the
> source storage is ZFS over iSCSI
> A few seconds after I started disk move, the guest FS will "panic". For
> example,
> with an el7 guest using XFS, I get :
> kernel: sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> kernel: sd 2:0:0:0: [sda] Sense Key : Aborted Command [current]
> kernel: sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
> kernel: sd 2:0:0:0: [sda] CDB: Read(10) 28 00 00 79 7f a8 00 00 08 00
> kernel: blk_update_request: I/O error, dev sda, sector 7962536
> kernel: sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> kernel: sd 2:0:0:0: [sda] Sense Key : Aborted Command [current]
> kernel: sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
> kernel: sd 2:0:0:0: [sda] CDB: Read(10) 28 00 00 79 7f a8 00 00 08 00
> kernel: blk_update_request: I/O error, dev sda, sector 7962536
> kernel: sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> kernel: sd 2:0:0:0: [sda] Sense Key : Aborted Command [current]
> kernel: sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
> kernel: sd 2:0:0:0: [sda] CDB: Read(10) 28 00 00 bc 0e 28 00 00 08 00
> kernel: blk_update_request: I/O error, dev sda, sector 12324392
> And the system completely crash. The data itself is not impacted. I can
> restart
> the guest and everything appears OK. It doesn't matter if I let the disk move
> operation terminates or if I cancel it.
> Moving the disk offline works as expected.
> Sparse or non sparse zvol backend doesn't matter either.
> I searched a lot about this issue, and found at least two other persons having
> the same, or a very similar issue :
>* One using ZoL but with SCST, see [
>https://sourceforge.net/p/scst/mailman/message/35241011/ |
> https://sourceforge.net/p/scst/mailman/message/35241011/ ]
>* Another, using OmniOS, so with Comstar, see [
>
> https://forum.proxmox.com/threads/storage-iscsi-move-results-to-io-error.38848/
>|
>
> https://forum.proxmox.com/threads/storage-iscsi-move-results-to-io-error.38848/
> ]
> Both are likely running PVE5, so it looks like it's not a recently introduced
> regression.
> I also was able to reproduce the issue with a FreeNAS storage, so using ctld.
> As
> the issue is present with so many different stack, I think we can eliminate an
> issue on the storage side. The problem is most likely on qemu, in it's iSCSI
> block implementation.
> The SCST-Devel thread is interesting, but infortunately, it's beyond my skills
> here.
> Any advice on how to debug this further ? I can reproduce it whenever I want,
> on
> a test setup. I'm happy to provide any usefull informations
> Regards, Daniel
Forgot to mention. When moving a disk offline, from ZFS over iSCSI to something
else (in my case to an NFS storage), I do have warnings like this :
create full clone of drive scsi0 (zfs-test:vm-132-disk-0)
Formatting '/mnt/pve/nfs-dumps/images/132/vm-132-disk-0.qcow2', fmt=qcow2
size=53687091200 cluster_size=65536 preallocation=metadata lazy_refcounts=off
refcount_bits=16
transferred: 0 bytes remaining: 53687091200 bytes total: 53687091200 bytes
progression: 0.00 %
qemu-img: iSCSI GET_LBA_STATUS failed at lba 0: SENSE KEY:ILLEGAL_REQUEST(5)
ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 4194303: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 8388606: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 12582909: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 16777212: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 20971515: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 25165818: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 29360121: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 33554424: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
qemu-img: iSCSI GET_LBA_STATUS failed at lba 37748727: SENSE
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
