On Tue, Aug 21, 2018 at 04:07:08PM -0500, Mike Christie wrote:
> On 08/08/2018 02:31 PM, Greg Edwards wrote:
>> When T10 PI is enabled on a backing device for the iblock backstore, the
>> PI SGL for the entire command is attached to the first bio only. This
>> works fine if the command is covered by a single bio, but results in
>> integrity verification errors for the other bios in a multi-bio command.
>>
>
> Did you hit this with a older distro kernel?
>
> It looks like iblock_get_bio will alloc a bio that has enough vecs for
> the entire cmd (bi_max_vecs will equal sgl_nents). So it is not clear to
> me how does the bio_add_page call ever return a value other than
> sg->length, and we end up doing another iblock_get_bio call?
I hit it with the tip of Linus' tree, but it depended on some other
in-flight changes. Those other changes are now in Linus' tree for 4.19,
with the exception of [1].
Without [1], when doing a large read I/O through vhost + iblock to a T10
PI enabled device (I used scsi_debug), you first hit the vhost
VHOST_SCSI_PREALLOC_PROT_SGLS limitation noted in [1].
Once the limitation on I/O size is no longer gated by
VHOST_SCSI_PREALLOC_PROT_SGLS, the next issue I hit is the one this
patch addresses. I should have been more precise in my commit message.
The failure is actually a bio_integrity_alloc() failure to allocate the
bip_vec when cmd->t_prot_nents exceeds 256 (BIO_MAX_PAGES), which
results in the following failure on the host:
[ 53.780723] Unable to allocate bio_integrity_payload
and the following failure on the client:
[ 28.724432] sd 0:0:1:0: [sda] tag#40 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 28.736127] sd 0:0:1:0: [sda] tag#40 Sense Key : Not Ready [current]
[ 28.744567] sd 0:0:1:0: [sda] tag#40 Add. Sense: Logical unit communication
failure
[ 28.754724] sd 0:0:1:0: [sda] tag#40 CDB: Read(10) 28 20 00 00 00 00 00 38
00 00
[ 28.766190] print_req_error: I/O error, dev sda, sector 0
By splitting up the PI SGL across the bios, you avoid ever trying to
allocate a too-large bip_vec (I was testing with 32 MiB I/Os).
Here is how I am testing:
L1 VM:
# modprobe scsi_debug dif=1 dix=1 guard=0 dev_size_mb=6144
# targetcli <<EOF
/backstores/block create dev=/dev/sda name=scsi_debug
/vhost create wwn=naa.50014055d17a5a87
/vhost/naa.50014055d17a5a87/tpg1/luns/ create /backstores/block/scsi_debug
EOF
L2 VM is booted with QEMU vhost option 't10_pi=on', which depends on
QEMU patches [2]:
-device vhost-scsi-pci,wwpn=naa.50014055d17a5a87,t10_pi=on \
then in L2 VM:
# cat /sys/block/sda/queue/max_hw_sectors_kb >
/sys/block/sda/queue/max_sectors_kb
# dd if=/dev/sda of=/dev/null bs=32M iflag=direct count=1
The end goal being to have a vehicle to test large I/Os through
virtio_scsi to a PI enabled device.
Greg
[1]
https://lists.linuxfoundation.org/pipermail/virtualization/2018-August/039040.html
[2] https://lists.nongnu.org/archive/html/qemu-devel/2018-08/msg01298.html
