Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On Mon, Nov 25, 2019 at 09:16:10AM +, Denis Plotnikov wrote:
>
>
> On 06.11.2019 15:03, Michael S. Tsirkin wrote:
> > On Thu, Oct 24, 2019 at 11:34:34AM +, Denis Lunev wrote:
> >> On 10/24/19 12:28 AM, Michael S. Tsirkin wrote:
> >>> On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
> From: "Denis V. Lunev"
>
> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
> field reported by SCSI controler. Thus typical sequential read with
> 1 MB size results in the following pattern of the IO from the guest:
> 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd]
> 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd]
> 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd]
> 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0]
> 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0]
> 8,16 115759 2.768539782 0 C R 2097120 + 32 [0]
> The IO was generated by
> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
>
> This effectively means that on rotational disks we will observe 3 IOPS
> for each 2 MBs processed. This definitely negatively affects both
> guest and host IO performance.
>
> The cure is relatively simple - we should report lengthy scatter-gather
> ability of the SCSI controller. Fortunately the situation here is very
> good. VirtIO transport layer can accomodate 1024 items in one request
> while we are using only 128. This situation is present since almost
> very beginning. 2 items are dedicated for request metadata thus we
> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
>
> The following pattern is observed after the patch:
> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
> which is much better.
>
> The dark side of this patch is that we are tweaking guest visible
> parameter, though this should be relatively safe as above transport
> layer support is present in QEMU/host Linux for a very long time.
> The patch adds configurable property for VirtIO SCSI with a new default
> and hardcode option for VirtBlock which does not provide good
> configurable framework.
>
> Unfortunately the commit can not be applied as is. For the real cure we
> need guest to be fixed to accomodate that queue length, which is done
> only in the latest 4.14 kernel. Thus we are going to expose the property
> and tweak it on machine type level.
>
> The problem with the old kernels is that they have
> max_segments <= virtqueue_size restriction which cause the guest
> crashing in the case of violation.
> >>> This isn't just in the guests: virtio spec also seems to imply this,
> >>> or at least be vague on this point.
> >>>
> >>> So I think it'll need a feature bit.
> >>> Doing that in a safe way will also allow being compatible with old guests.
> >>>
> >>> The only downside is it's a bit more work as we need to
> >>> spec this out and add guest support.
> >>>
> To fix the case described above in the old kernels we can increase
> virtqueue_size to 256 and max_segments to 254. The pitfall here is
> that seabios allows the virtqueue_size-s < 128, however, the seabios
> patch extending that value to 256 is pending.
> >>> And the fix here is just to limit large vq size to virtio 1.0.
> >>> In that mode it's fine I think:
> >>>
> >>>
> >>> /* check if the queue is available */
> >>> if (vp->use_modern) {
> >>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> >>> if (num > MAX_QUEUE_NUM) {
> >>> vp_write(>common, virtio_pci_common_cfg, queue_size,
> >>> MAX_QUEUE_NUM);
> >>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> >>> }
> >>> } else {
> >>> num = vp_read(>legacy, virtio_pci_legacy, queue_num);
> >>> }
> The same seabios snippet, but more detailed:
>
> vp_find_vq()
> {
> ...
> /* check if the queue is available */
> if (vp->use_modern) {
> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> if (num > MAX_QUEUE_NUM) {
> vp_write(>common, virtio_pci_common_cfg, queue_size,
> MAX_QUEUE_NUM);
> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
So how about we drop this last line in bios?
Will fix things for existing hypervisors.
spec does not say guests need to re-read it.
> }
> } else {
> num = vp_read(>legacy, virtio_pci_legacy, queue_num);
> }
> if (!num) {
>
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
Ping!
On 25.11.2019 12:16, Denis Plotnikov wrote:
>
>
> On 06.11.2019 15:03, Michael S. Tsirkin wrote:
>> On Thu, Oct 24, 2019 at 11:34:34AM +, Denis Lunev wrote:
>>> On 10/24/19 12:28 AM, Michael S. Tsirkin wrote:
On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
> From: "Denis V. Lunev"
>
> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
> field reported by SCSI controler. Thus typical sequential read with
> 1 MB size results in the following pattern of the IO from the guest:
> 8,16 1 15754 2.766095122 2071 D R 2095104 + 1008 [dd]
> 8,16 1 15755 2.766108785 2071 D R 2096112 + 1008 [dd]
> 8,16 1 15756 2.766113486 2071 D R 2097120 + 32 [dd]
> 8,16 1 15757 2.767668961 0 C R 2095104 + 1008 [0]
> 8,16 1 15758 2.768534315 0 C R 2096112 + 1008 [0]
> 8,16 1 15759 2.768539782 0 C R 2097120 + 32 [0]
> The IO was generated by
> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
>
> This effectively means that on rotational disks we will observe 3
> IOPS
> for each 2 MBs processed. This definitely negatively affects both
> guest and host IO performance.
>
> The cure is relatively simple - we should report lengthy
> scatter-gather
> ability of the SCSI controller. Fortunately the situation here is
> very
> good. VirtIO transport layer can accomodate 1024 items in one request
> while we are using only 128. This situation is present since almost
> very beginning. 2 items are dedicated for request metadata thus we
> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
>
> The following pattern is observed after the patch:
> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
> which is much better.
>
> The dark side of this patch is that we are tweaking guest visible
> parameter, though this should be relatively safe as above transport
> layer support is present in QEMU/host Linux for a very long time.
> The patch adds configurable property for VirtIO SCSI with a new
> default
> and hardcode option for VirtBlock which does not provide good
> configurable framework.
>
> Unfortunately the commit can not be applied as is. For the real
> cure we
> need guest to be fixed to accomodate that queue length, which is done
> only in the latest 4.14 kernel. Thus we are going to expose the
> property
> and tweak it on machine type level.
>
> The problem with the old kernels is that they have
> max_segments <= virtqueue_size restriction which cause the guest
> crashing in the case of violation.
This isn't just in the guests: virtio spec also seems to imply this,
or at least be vague on this point.
So I think it'll need a feature bit.
Doing that in a safe way will also allow being compatible with old
guests.
The only downside is it's a bit more work as we need to
spec this out and add guest support.
> To fix the case described above in the old kernels we can increase
> virtqueue_size to 256 and max_segments to 254. The pitfall here is
> that seabios allows the virtqueue_size-s < 128, however, the seabios
> patch extending that value to 256 is pending.
And the fix here is just to limit large vq size to virtio 1.0.
In that mode it's fine I think:
/* check if the queue is available */
if (vp->use_modern) {
num = vp_read(>common, virtio_pci_common_cfg, queue_size);
if (num > MAX_QUEUE_NUM) {
vp_write(>common, virtio_pci_common_cfg, queue_size,
MAX_QUEUE_NUM);
num = vp_read(>common, virtio_pci_common_cfg,
queue_size);
}
} else {
num = vp_read(>legacy, virtio_pci_legacy, queue_num);
}
> The same seabios snippet, but more detailed:
>
> vp_find_vq()
> {
> ...
> /* check if the queue is available */
> if (vp->use_modern) {
> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> if (num > MAX_QUEUE_NUM) {
> vp_write(>common, virtio_pci_common_cfg, queue_size,
> MAX_QUEUE_NUM);
> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> }
> } else {
> num = vp_read(>legacy, virtio_pci_legacy, queue_num);
> }
> if (!num) {
> dprintf(1, "ERROR: queue size is 0\n");
> goto fail;
> }
> if (num > MAX_QUEUE_NUM) {
> dprintf(1, "ERROR: queue size %d > %d\n", num, MAX_QUEUE_NUM);
> goto fail;
> }
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On 06.11.2019 15:03, Michael S. Tsirkin wrote:
> On Thu, Oct 24, 2019 at 11:34:34AM +, Denis Lunev wrote:
>> On 10/24/19 12:28 AM, Michael S. Tsirkin wrote:
>>> On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
From: "Denis V. Lunev"
Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
field reported by SCSI controler. Thus typical sequential read with
1 MB size results in the following pattern of the IO from the guest:
8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd]
8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd]
8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd]
8,16 115757 2.767668961 0 C R 2095104 + 1008 [0]
8,16 115758 2.768534315 0 C R 2096112 + 1008 [0]
8,16 115759 2.768539782 0 C R 2097120 + 32 [0]
The IO was generated by
dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
This effectively means that on rotational disks we will observe 3 IOPS
for each 2 MBs processed. This definitely negatively affects both
guest and host IO performance.
The cure is relatively simple - we should report lengthy scatter-gather
ability of the SCSI controller. Fortunately the situation here is very
good. VirtIO transport layer can accomodate 1024 items in one request
while we are using only 128. This situation is present since almost
very beginning. 2 items are dedicated for request metadata thus we
should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
The following pattern is observed after the patch:
8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
which is much better.
The dark side of this patch is that we are tweaking guest visible
parameter, though this should be relatively safe as above transport
layer support is present in QEMU/host Linux for a very long time.
The patch adds configurable property for VirtIO SCSI with a new default
and hardcode option for VirtBlock which does not provide good
configurable framework.
Unfortunately the commit can not be applied as is. For the real cure we
need guest to be fixed to accomodate that queue length, which is done
only in the latest 4.14 kernel. Thus we are going to expose the property
and tweak it on machine type level.
The problem with the old kernels is that they have
max_segments <= virtqueue_size restriction which cause the guest
crashing in the case of violation.
>>> This isn't just in the guests: virtio spec also seems to imply this,
>>> or at least be vague on this point.
>>>
>>> So I think it'll need a feature bit.
>>> Doing that in a safe way will also allow being compatible with old guests.
>>>
>>> The only downside is it's a bit more work as we need to
>>> spec this out and add guest support.
>>>
To fix the case described above in the old kernels we can increase
virtqueue_size to 256 and max_segments to 254. The pitfall here is
that seabios allows the virtqueue_size-s < 128, however, the seabios
patch extending that value to 256 is pending.
>>> And the fix here is just to limit large vq size to virtio 1.0.
>>> In that mode it's fine I think:
>>>
>>>
>>> /* check if the queue is available */
>>> if (vp->use_modern) {
>>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
>>> if (num > MAX_QUEUE_NUM) {
>>> vp_write(>common, virtio_pci_common_cfg, queue_size,
>>> MAX_QUEUE_NUM);
>>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
>>> }
>>> } else {
>>> num = vp_read(>legacy, virtio_pci_legacy, queue_num);
>>> }
The same seabios snippet, but more detailed:
vp_find_vq()
{
...
/* check if the queue is available */
if (vp->use_modern) {
num = vp_read(>common, virtio_pci_common_cfg, queue_size);
if (num > MAX_QUEUE_NUM) {
vp_write(>common, virtio_pci_common_cfg, queue_size,
MAX_QUEUE_NUM);
num = vp_read(>common, virtio_pci_common_cfg, queue_size);
}
} else {
num = vp_read(>legacy, virtio_pci_legacy, queue_num);
}
if (!num) {
dprintf(1, "ERROR: queue size is 0\n");
goto fail;
}
if (num > MAX_QUEUE_NUM) {
dprintf(1, "ERROR: queue size %d > %d\n", num, MAX_QUEUE_NUM);
goto fail;
}
...
}
It turned out that the problem is here, but not because of the seabios code.
The virtqueue size is written and then incorrect value is re-read.
Thanks to Roman Kagan (rka...@virtuozzo.com) for investigating
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On 13.11.2019 16:18, Michael S. Tsirkin wrote:
> On Wed, Nov 13, 2019 at 12:38:48PM +, Denis Plotnikov wrote:
>>
>> On 06.11.2019 15:03, Michael S. Tsirkin wrote:
>>> On Thu, Oct 24, 2019 at 11:34:34AM +, Denis Lunev wrote:
On 10/24/19 12:28 AM, Michael S. Tsirkin wrote:
> On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
>> From: "Denis V. Lunev"
>>
>> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
>> field reported by SCSI controler. Thus typical sequential read with
>> 1 MB size results in the following pattern of the IO from the guest:
>> 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd]
>> 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd]
>> 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd]
>> 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0]
>> 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0]
>> 8,16 115759 2.768539782 0 C R 2097120 + 32 [0]
>> The IO was generated by
>> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
>>
>> This effectively means that on rotational disks we will observe 3 IOPS
>> for each 2 MBs processed. This definitely negatively affects both
>> guest and host IO performance.
>>
>> The cure is relatively simple - we should report lengthy scatter-gather
>> ability of the SCSI controller. Fortunately the situation here is very
>> good. VirtIO transport layer can accomodate 1024 items in one request
>> while we are using only 128. This situation is present since almost
>> very beginning. 2 items are dedicated for request metadata thus we
>> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
>>
>> The following pattern is observed after the patch:
>> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
>> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
>> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
>> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
>> which is much better.
>>
>> The dark side of this patch is that we are tweaking guest visible
>> parameter, though this should be relatively safe as above transport
>> layer support is present in QEMU/host Linux for a very long time.
>> The patch adds configurable property for VirtIO SCSI with a new default
>> and hardcode option for VirtBlock which does not provide good
>> configurable framework.
>>
>> Unfortunately the commit can not be applied as is. For the real cure we
>> need guest to be fixed to accomodate that queue length, which is done
>> only in the latest 4.14 kernel. Thus we are going to expose the property
>> and tweak it on machine type level.
>>
>> The problem with the old kernels is that they have
>> max_segments <= virtqueue_size restriction which cause the guest
>> crashing in the case of violation.
> This isn't just in the guests: virtio spec also seems to imply this,
> or at least be vague on this point.
>
> So I think it'll need a feature bit.
> Doing that in a safe way will also allow being compatible with old guests.
>
> The only downside is it's a bit more work as we need to
> spec this out and add guest support.
>
>> To fix the case described above in the old kernels we can increase
>> virtqueue_size to 256 and max_segments to 254. The pitfall here is
>> that seabios allows the virtqueue_size-s < 128, however, the seabios
>> patch extending that value to 256 is pending.
> And the fix here is just to limit large vq size to virtio 1.0.
> In that mode it's fine I think:
>
>
> /* check if the queue is available */
> if (vp->use_modern) {
> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> if (num > MAX_QUEUE_NUM) {
> vp_write(>common, virtio_pci_common_cfg, queue_size,
> MAX_QUEUE_NUM);
> num = vp_read(>common, virtio_pci_common_cfg,
> queue_size);
> }
> } else {
> num = vp_read(>legacy, virtio_pci_legacy, queue_num);
> }
you mean to put the code like this into virtio_pci_realize() inside QEMU?
If no, can you pls clarify which component should be touched.
Den
>>> I mean:
>>>- add an API to change the default queue size
>>>- add a validate features callback, in there check and for modern
>>> flag set in features increase the queue size
>>>
>>> maybe all this is too much work, we could block this
>>> for transitional devices, but your patch does not do it,
>>> you need to check that legacy is enabled not that modern
>>> is not disabled.
>> To develop the idea of how to adjust queue size further I'd
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On Wed, Nov 13, 2019 at 12:38:48PM +, Denis Plotnikov wrote:
>
>
> On 06.11.2019 15:03, Michael S. Tsirkin wrote:
> > On Thu, Oct 24, 2019 at 11:34:34AM +, Denis Lunev wrote:
> >> On 10/24/19 12:28 AM, Michael S. Tsirkin wrote:
> >>> On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
> From: "Denis V. Lunev"
>
> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
> field reported by SCSI controler. Thus typical sequential read with
> 1 MB size results in the following pattern of the IO from the guest:
> 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd]
> 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd]
> 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd]
> 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0]
> 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0]
> 8,16 115759 2.768539782 0 C R 2097120 + 32 [0]
> The IO was generated by
> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
>
> This effectively means that on rotational disks we will observe 3 IOPS
> for each 2 MBs processed. This definitely negatively affects both
> guest and host IO performance.
>
> The cure is relatively simple - we should report lengthy scatter-gather
> ability of the SCSI controller. Fortunately the situation here is very
> good. VirtIO transport layer can accomodate 1024 items in one request
> while we are using only 128. This situation is present since almost
> very beginning. 2 items are dedicated for request metadata thus we
> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
>
> The following pattern is observed after the patch:
> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
> which is much better.
>
> The dark side of this patch is that we are tweaking guest visible
> parameter, though this should be relatively safe as above transport
> layer support is present in QEMU/host Linux for a very long time.
> The patch adds configurable property for VirtIO SCSI with a new default
> and hardcode option for VirtBlock which does not provide good
> configurable framework.
>
> Unfortunately the commit can not be applied as is. For the real cure we
> need guest to be fixed to accomodate that queue length, which is done
> only in the latest 4.14 kernel. Thus we are going to expose the property
> and tweak it on machine type level.
>
> The problem with the old kernels is that they have
> max_segments <= virtqueue_size restriction which cause the guest
> crashing in the case of violation.
> >>> This isn't just in the guests: virtio spec also seems to imply this,
> >>> or at least be vague on this point.
> >>>
> >>> So I think it'll need a feature bit.
> >>> Doing that in a safe way will also allow being compatible with old guests.
> >>>
> >>> The only downside is it's a bit more work as we need to
> >>> spec this out and add guest support.
> >>>
> To fix the case described above in the old kernels we can increase
> virtqueue_size to 256 and max_segments to 254. The pitfall here is
> that seabios allows the virtqueue_size-s < 128, however, the seabios
> patch extending that value to 256 is pending.
> >>> And the fix here is just to limit large vq size to virtio 1.0.
> >>> In that mode it's fine I think:
> >>>
> >>>
> >>> /* check if the queue is available */
> >>> if (vp->use_modern) {
> >>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> >>> if (num > MAX_QUEUE_NUM) {
> >>> vp_write(>common, virtio_pci_common_cfg, queue_size,
> >>> MAX_QUEUE_NUM);
> >>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> >>> }
> >>> } else {
> >>> num = vp_read(>legacy, virtio_pci_legacy, queue_num);
> >>> }
> >> you mean to put the code like this into virtio_pci_realize() inside QEMU?
> >>
> >> If no, can you pls clarify which component should be touched.
> >>
> >> Den
> > I mean:
> > - add an API to change the default queue size
> > - add a validate features callback, in there check and for modern
> > flag set in features increase the queue size
> >
> > maybe all this is too much work, we could block this
> > for transitional devices, but your patch does not do it,
> > you need to check that legacy is enabled not that modern
> > is not disabled.
> To develop the idea of how to adjust queue size further I'd like to
> summarize what we have:
>
> 1. Variatly of gusts without(?) queue size
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On 06.11.2019 15:03, Michael S. Tsirkin wrote:
> On Thu, Oct 24, 2019 at 11:34:34AM +, Denis Lunev wrote:
>> On 10/24/19 12:28 AM, Michael S. Tsirkin wrote:
>>> On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
From: "Denis V. Lunev"
Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
field reported by SCSI controler. Thus typical sequential read with
1 MB size results in the following pattern of the IO from the guest:
8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd]
8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd]
8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd]
8,16 115757 2.767668961 0 C R 2095104 + 1008 [0]
8,16 115758 2.768534315 0 C R 2096112 + 1008 [0]
8,16 115759 2.768539782 0 C R 2097120 + 32 [0]
The IO was generated by
dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
This effectively means that on rotational disks we will observe 3 IOPS
for each 2 MBs processed. This definitely negatively affects both
guest and host IO performance.
The cure is relatively simple - we should report lengthy scatter-gather
ability of the SCSI controller. Fortunately the situation here is very
good. VirtIO transport layer can accomodate 1024 items in one request
while we are using only 128. This situation is present since almost
very beginning. 2 items are dedicated for request metadata thus we
should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
The following pattern is observed after the patch:
8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
which is much better.
The dark side of this patch is that we are tweaking guest visible
parameter, though this should be relatively safe as above transport
layer support is present in QEMU/host Linux for a very long time.
The patch adds configurable property for VirtIO SCSI with a new default
and hardcode option for VirtBlock which does not provide good
configurable framework.
Unfortunately the commit can not be applied as is. For the real cure we
need guest to be fixed to accomodate that queue length, which is done
only in the latest 4.14 kernel. Thus we are going to expose the property
and tweak it on machine type level.
The problem with the old kernels is that they have
max_segments <= virtqueue_size restriction which cause the guest
crashing in the case of violation.
>>> This isn't just in the guests: virtio spec also seems to imply this,
>>> or at least be vague on this point.
>>>
>>> So I think it'll need a feature bit.
>>> Doing that in a safe way will also allow being compatible with old guests.
>>>
>>> The only downside is it's a bit more work as we need to
>>> spec this out and add guest support.
>>>
To fix the case described above in the old kernels we can increase
virtqueue_size to 256 and max_segments to 254. The pitfall here is
that seabios allows the virtqueue_size-s < 128, however, the seabios
patch extending that value to 256 is pending.
>>> And the fix here is just to limit large vq size to virtio 1.0.
>>> In that mode it's fine I think:
>>>
>>>
>>> /* check if the queue is available */
>>> if (vp->use_modern) {
>>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
>>> if (num > MAX_QUEUE_NUM) {
>>> vp_write(>common, virtio_pci_common_cfg, queue_size,
>>> MAX_QUEUE_NUM);
>>> num = vp_read(>common, virtio_pci_common_cfg, queue_size);
>>> }
>>> } else {
>>> num = vp_read(>legacy, virtio_pci_legacy, queue_num);
>>> }
>> you mean to put the code like this into virtio_pci_realize() inside QEMU?
>>
>> If no, can you pls clarify which component should be touched.
>>
>> Den
> I mean:
> - add an API to change the default queue size
> - add a validate features callback, in there check and for modern
> flag set in features increase the queue size
>
> maybe all this is too much work, we could block this
> for transitional devices, but your patch does not do it,
> you need to check that legacy is enabled not that modern
> is not disabled.
To develop the idea of how to adjust queue size further I'd like to
summarize what we have:
1. Variatly of gusts without(?) queue size limitations which can support
queue sizes up to MAX(1024)
2. seabios setups with two possible max queue size limitations: 128 and
256 (recently commited)
3. non-sebios setups with unknown max queue-size limitations
Taking into account that queue size may be
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On Wed, Oct 23, 2019 at 05:28:17PM -0400, Michael S. Tsirkin wrote: > On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote: > > From: "Denis V. Lunev" > > > > Linux guests submit IO requests no longer than PAGE_SIZE * max_seg > > field reported by SCSI controler. Thus typical sequential read with > > 1 MB size results in the following pattern of the IO from the guest: > > 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd] > > 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd] > > 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd] > > 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0] > > 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0] > > 8,16 115759 2.768539782 0 C R 2097120 + 32 [0] > > The IO was generated by > > dd if=/dev/sda of=/dev/null bs=1024 iflag=direct > > > > This effectively means that on rotational disks we will observe 3 IOPS > > for each 2 MBs processed. This definitely negatively affects both > > guest and host IO performance. > > > > The cure is relatively simple - we should report lengthy scatter-gather > > ability of the SCSI controller. Fortunately the situation here is very > > good. VirtIO transport layer can accomodate 1024 items in one request > > while we are using only 128. This situation is present since almost > > very beginning. 2 items are dedicated for request metadata thus we > > should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg. > > > > The following pattern is observed after the patch: > > 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd] > > 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd] > > 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0] > > 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0] > > which is much better. > > > > The dark side of this patch is that we are tweaking guest visible > > parameter, though this should be relatively safe as above transport > > layer support is present in QEMU/host Linux for a very long time. > > The patch adds configurable property for VirtIO SCSI with a new default > > and hardcode option for VirtBlock which does not provide good > > configurable framework. > > > > Unfortunately the commit can not be applied as is. For the real cure we > > need guest to be fixed to accomodate that queue length, which is done > > only in the latest 4.14 kernel. Thus we are going to expose the property > > and tweak it on machine type level. > > > > The problem with the old kernels is that they have > > max_segments <= virtqueue_size restriction which cause the guest > > crashing in the case of violation. > > This isn't just in the guests: virtio spec also seems to imply this, > or at least be vague on this point. > > So I think it'll need a feature bit. > Doing that in a safe way will also allow being compatible with old guests. The spec is quite explicit about this: 2.6.5 The Virtqueue Descriptor Table The number of descriptors in the table is defined by the queue size for this virtqueue: this is the maximum possible descriptor chain length. and: 2.6.5.3.1 Driver Requirements: Indirect Descriptors A driver MUST NOT create a descriptor chain longer than the Queue Size of the device. If some drivers or devices allow longer descriptor chains today that's an implementation quirk but a new feature bit is definitely required to officially allow longer descriptor chains. Stefan signature.asc Description: PGP signature
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On Thu, Oct 24, 2019 at 11:34:34AM +, Denis Lunev wrote:
> On 10/24/19 12:28 AM, Michael S. Tsirkin wrote:
> > On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
> >> From: "Denis V. Lunev"
> >>
> >> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
> >> field reported by SCSI controler. Thus typical sequential read with
> >> 1 MB size results in the following pattern of the IO from the guest:
> >> 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd]
> >> 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd]
> >> 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd]
> >> 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0]
> >> 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0]
> >> 8,16 115759 2.768539782 0 C R 2097120 + 32 [0]
> >> The IO was generated by
> >> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
> >>
> >> This effectively means that on rotational disks we will observe 3 IOPS
> >> for each 2 MBs processed. This definitely negatively affects both
> >> guest and host IO performance.
> >>
> >> The cure is relatively simple - we should report lengthy scatter-gather
> >> ability of the SCSI controller. Fortunately the situation here is very
> >> good. VirtIO transport layer can accomodate 1024 items in one request
> >> while we are using only 128. This situation is present since almost
> >> very beginning. 2 items are dedicated for request metadata thus we
> >> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
> >>
> >> The following pattern is observed after the patch:
> >> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
> >> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
> >> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
> >> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
> >> which is much better.
> >>
> >> The dark side of this patch is that we are tweaking guest visible
> >> parameter, though this should be relatively safe as above transport
> >> layer support is present in QEMU/host Linux for a very long time.
> >> The patch adds configurable property for VirtIO SCSI with a new default
> >> and hardcode option for VirtBlock which does not provide good
> >> configurable framework.
> >>
> >> Unfortunately the commit can not be applied as is. For the real cure we
> >> need guest to be fixed to accomodate that queue length, which is done
> >> only in the latest 4.14 kernel. Thus we are going to expose the property
> >> and tweak it on machine type level.
> >>
> >> The problem with the old kernels is that they have
> >> max_segments <= virtqueue_size restriction which cause the guest
> >> crashing in the case of violation.
> > This isn't just in the guests: virtio spec also seems to imply this,
> > or at least be vague on this point.
> >
> > So I think it'll need a feature bit.
> > Doing that in a safe way will also allow being compatible with old guests.
> >
> > The only downside is it's a bit more work as we need to
> > spec this out and add guest support.
> >
> >> To fix the case described above in the old kernels we can increase
> >> virtqueue_size to 256 and max_segments to 254. The pitfall here is
> >> that seabios allows the virtqueue_size-s < 128, however, the seabios
> >> patch extending that value to 256 is pending.
> >
> > And the fix here is just to limit large vq size to virtio 1.0.
> > In that mode it's fine I think:
> >
> >
> >/* check if the queue is available */
> >if (vp->use_modern) {
> >num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> >if (num > MAX_QUEUE_NUM) {
> >vp_write(>common, virtio_pci_common_cfg, queue_size,
> > MAX_QUEUE_NUM);
> >num = vp_read(>common, virtio_pci_common_cfg, queue_size);
> >}
> >} else {
> >num = vp_read(>legacy, virtio_pci_legacy, queue_num);
> >}
>
> you mean to put the code like this into virtio_pci_realize() inside QEMU?
>
> If no, can you pls clarify which component should be touched.
>
> Den
I mean:
- add an API to change the default queue size
- add a validate features callback, in there check and for modern
flag set in features increase the queue size
maybe all this is too much work, we could block this
for transitional devices, but your patch does not do it,
you need to check that legacy is enabled not that modern
is not disabled.
--
MST
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On 10/24/19 12:28 AM, Michael S. Tsirkin wrote:
> On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
>> From: "Denis V. Lunev"
>>
>> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
>> field reported by SCSI controler. Thus typical sequential read with
>> 1 MB size results in the following pattern of the IO from the guest:
>> 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd]
>> 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd]
>> 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd]
>> 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0]
>> 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0]
>> 8,16 115759 2.768539782 0 C R 2097120 + 32 [0]
>> The IO was generated by
>> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
>>
>> This effectively means that on rotational disks we will observe 3 IOPS
>> for each 2 MBs processed. This definitely negatively affects both
>> guest and host IO performance.
>>
>> The cure is relatively simple - we should report lengthy scatter-gather
>> ability of the SCSI controller. Fortunately the situation here is very
>> good. VirtIO transport layer can accomodate 1024 items in one request
>> while we are using only 128. This situation is present since almost
>> very beginning. 2 items are dedicated for request metadata thus we
>> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
>>
>> The following pattern is observed after the patch:
>> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
>> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
>> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
>> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
>> which is much better.
>>
>> The dark side of this patch is that we are tweaking guest visible
>> parameter, though this should be relatively safe as above transport
>> layer support is present in QEMU/host Linux for a very long time.
>> The patch adds configurable property for VirtIO SCSI with a new default
>> and hardcode option for VirtBlock which does not provide good
>> configurable framework.
>>
>> Unfortunately the commit can not be applied as is. For the real cure we
>> need guest to be fixed to accomodate that queue length, which is done
>> only in the latest 4.14 kernel. Thus we are going to expose the property
>> and tweak it on machine type level.
>>
>> The problem with the old kernels is that they have
>> max_segments <= virtqueue_size restriction which cause the guest
>> crashing in the case of violation.
> This isn't just in the guests: virtio spec also seems to imply this,
> or at least be vague on this point.
>
> So I think it'll need a feature bit.
> Doing that in a safe way will also allow being compatible with old guests.
>
> The only downside is it's a bit more work as we need to
> spec this out and add guest support.
>
>> To fix the case described above in the old kernels we can increase
>> virtqueue_size to 256 and max_segments to 254. The pitfall here is
>> that seabios allows the virtqueue_size-s < 128, however, the seabios
>> patch extending that value to 256 is pending.
>
> And the fix here is just to limit large vq size to virtio 1.0.
> In that mode it's fine I think:
>
>
>/* check if the queue is available */
>if (vp->use_modern) {
>num = vp_read(>common, virtio_pci_common_cfg, queue_size);
>if (num > MAX_QUEUE_NUM) {
>vp_write(>common, virtio_pci_common_cfg, queue_size,
> MAX_QUEUE_NUM);
>num = vp_read(>common, virtio_pci_common_cfg, queue_size);
>}
>} else {
>num = vp_read(>legacy, virtio_pci_legacy, queue_num);
>}
you mean to put the code like this into virtio_pci_realize() inside QEMU?
If no, can you pls clarify which component should be touched.
Den
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On Mon, Oct 21, 2019 at 02:24:55PM +0100, Stefan Hajnoczi wrote: > On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote: > > From: "Denis V. Lunev" > > > > Linux guests submit IO requests no longer than PAGE_SIZE * max_seg > > field reported by SCSI controler. Thus typical sequential read with > > 1 MB size results in the following pattern of the IO from the guest: > > 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd] > > 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd] > > 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd] > > 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0] > > 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0] > > 8,16 115759 2.768539782 0 C R 2097120 + 32 [0] > > The IO was generated by > > dd if=/dev/sda of=/dev/null bs=1024 iflag=direct > > > > This effectively means that on rotational disks we will observe 3 IOPS > > for each 2 MBs processed. This definitely negatively affects both > > guest and host IO performance. > > > > The cure is relatively simple - we should report lengthy scatter-gather > > ability of the SCSI controller. Fortunately the situation here is very > > good. VirtIO transport layer can accomodate 1024 items in one request > > while we are using only 128. This situation is present since almost > > very beginning. 2 items are dedicated for request metadata thus we > > should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg. > > > > The following pattern is observed after the patch: > > 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd] > > 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd] > > 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0] > > 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0] > > which is much better. > > > > The dark side of this patch is that we are tweaking guest visible > > parameter, though this should be relatively safe as above transport > > layer support is present in QEMU/host Linux for a very long time. > > The patch adds configurable property for VirtIO SCSI with a new default > > and hardcode option for VirtBlock which does not provide good > > configurable framework. > > > > Unfortunately the commit can not be applied as is. For the real cure we > > need guest to be fixed to accomodate that queue length, which is done > > only in the latest 4.14 kernel. Thus we are going to expose the property > > and tweak it on machine type level. > > > > The problem with the old kernels is that they have > > max_segments <= virtqueue_size restriction which cause the guest > > crashing in the case of violation. > > To fix the case described above in the old kernels we can increase > > virtqueue_size to 256 and max_segments to 254. The pitfall here is > > that seabios allows the virtqueue_size-s < 128, however, the seabios > > patch extending that value to 256 is pending. > > If I understand correctly you are relying on Indirect Descriptor support > in the guest driver in order to exceed the Virtqueue Descriptor Table > size. > > Unfortunately the "max_segments <= virtqueue_size restriction" is > required by the VIRTIO 1.1 specification: > > 2.6.5.3.1 Driver Requirements: Indirect Descriptors > > A driver MUST NOT create a descriptor chain longer than the Queue > Size of the device. > > So this idea seems to be in violation of the specification? > > There is a bug in hw/block/virtio-blk.c:virtio_blk_update_config() and > hw/scsi/virtio-scsi.c:virtio_scsi_get_config(): > > virtio_stl_p(vdev, _max, 128 - 2); > > This number should be the minimum of blk_get_max_iov() and > virtio_queue_get_num(), minus 2 for the header and footer. > > I looked at the Linux SCSI driver code and it seems each HBA has a > single max_segments number - it does not vary on a per-device basis. > This could be a problem if two host block device with different > max_segments are exposed to the guest through the same virtio-scsi > controller. Another bug? :( > > Anyway, if you want ~1024 descriptors you should set Queue Size to 1024. > I don't see a spec-compliant way of doing it otherwise. Hopefully I > have overlooked something and there is a nice way to solve this. > > Stefan We can extend the spec of course. And we can also have different vq sizes between legacy and modern interfaces. -- MST
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote:
> From: "Denis V. Lunev"
>
> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg
> field reported by SCSI controler. Thus typical sequential read with
> 1 MB size results in the following pattern of the IO from the guest:
> 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd]
> 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd]
> 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd]
> 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0]
> 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0]
> 8,16 115759 2.768539782 0 C R 2097120 + 32 [0]
> The IO was generated by
> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct
>
> This effectively means that on rotational disks we will observe 3 IOPS
> for each 2 MBs processed. This definitely negatively affects both
> guest and host IO performance.
>
> The cure is relatively simple - we should report lengthy scatter-gather
> ability of the SCSI controller. Fortunately the situation here is very
> good. VirtIO transport layer can accomodate 1024 items in one request
> while we are using only 128. This situation is present since almost
> very beginning. 2 items are dedicated for request metadata thus we
> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg.
>
> The following pattern is observed after the patch:
> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd]
> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd]
> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0]
> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0]
> which is much better.
>
> The dark side of this patch is that we are tweaking guest visible
> parameter, though this should be relatively safe as above transport
> layer support is present in QEMU/host Linux for a very long time.
> The patch adds configurable property for VirtIO SCSI with a new default
> and hardcode option for VirtBlock which does not provide good
> configurable framework.
>
> Unfortunately the commit can not be applied as is. For the real cure we
> need guest to be fixed to accomodate that queue length, which is done
> only in the latest 4.14 kernel. Thus we are going to expose the property
> and tweak it on machine type level.
>
> The problem with the old kernels is that they have
> max_segments <= virtqueue_size restriction which cause the guest
> crashing in the case of violation.
This isn't just in the guests: virtio spec also seems to imply this,
or at least be vague on this point.
So I think it'll need a feature bit.
Doing that in a safe way will also allow being compatible with old guests.
The only downside is it's a bit more work as we need to
spec this out and add guest support.
> To fix the case described above in the old kernels we can increase
> virtqueue_size to 256 and max_segments to 254. The pitfall here is
> that seabios allows the virtqueue_size-s < 128, however, the seabios
> patch extending that value to 256 is pending.
And the fix here is just to limit large vq size to virtio 1.0.
In that mode it's fine I think:
/* check if the queue is available */
if (vp->use_modern) {
num = vp_read(>common, virtio_pci_common_cfg, queue_size);
if (num > MAX_QUEUE_NUM) {
vp_write(>common, virtio_pci_common_cfg, queue_size,
MAX_QUEUE_NUM);
num = vp_read(>common, virtio_pci_common_cfg, queue_size);
}
} else {
num = vp_read(>legacy, virtio_pci_legacy, queue_num);
}
> CC: "Michael S. Tsirkin"
> CC: Stefan Hajnoczi
> CC: Kevin Wolf
> CC: Max Reitz
> CC: Gerd Hoffmann
> Signed-off-by: Denis V. Lunev
> Signed-off-by: Denis Plotnikov
> ---
> hw/block/virtio-blk.c | 3 ++-
> hw/scsi/vhost-scsi.c| 2 ++
> hw/scsi/virtio-scsi.c | 4 +++-
> include/hw/virtio/virtio-blk.h | 1 +
> include/hw/virtio/virtio-scsi.h | 1 +
> 5 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> index 06e57a4d39..b2eaeeaf67 100644
> --- a/hw/block/virtio-blk.c
> +++ b/hw/block/virtio-blk.c
> @@ -903,7 +903,7 @@ static void virtio_blk_update_config(VirtIODevice *vdev,
> uint8_t *config)
> blk_get_geometry(s->blk, );
> memset(, 0, sizeof(blkcfg));
> virtio_stq_p(vdev, , capacity);
> -virtio_stl_p(vdev, _max, 128 - 2);
> +virtio_stl_p(vdev, _max, s->conf.max_segments);
> virtio_stw_p(vdev, , conf->cyls);
> virtio_stl_p(vdev, _size, blk_size);
> virtio_stw_p(vdev, _io_size, conf->min_io_size / blk_size);
> @@ -1240,6 +1240,7 @@ static Property virtio_blk_properties[] = {
> conf.max_discard_sectors, BDRV_REQUEST_MAX_SECTORS),
> DEFINE_PROP_UINT32("max-write-zeroes-sectors", VirtIOBlock,
> conf.max_write_zeroes_sectors,
>
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On 10/23/19 5:17 PM, Stefan Hajnoczi wrote: > On Tue, Oct 22, 2019 at 04:01:57AM +, Denis Lunev wrote: >> On 10/21/19 4:24 PM, Stefan Hajnoczi wrote: >>> On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote: From: "Denis V. Lunev" Linux guests submit IO requests no longer than PAGE_SIZE * max_seg field reported by SCSI controler. Thus typical sequential read with 1 MB size results in the following pattern of the IO from the guest: 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd] 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd] 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd] 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0] 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0] 8,16 115759 2.768539782 0 C R 2097120 + 32 [0] The IO was generated by dd if=/dev/sda of=/dev/null bs=1024 iflag=direct This effectively means that on rotational disks we will observe 3 IOPS for each 2 MBs processed. This definitely negatively affects both guest and host IO performance. The cure is relatively simple - we should report lengthy scatter-gather ability of the SCSI controller. Fortunately the situation here is very good. VirtIO transport layer can accomodate 1024 items in one request while we are using only 128. This situation is present since almost very beginning. 2 items are dedicated for request metadata thus we should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg. The following pattern is observed after the patch: 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd] 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd] 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0] 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0] which is much better. The dark side of this patch is that we are tweaking guest visible parameter, though this should be relatively safe as above transport layer support is present in QEMU/host Linux for a very long time. The patch adds configurable property for VirtIO SCSI with a new default and hardcode option for VirtBlock which does not provide good configurable framework. Unfortunately the commit can not be applied as is. For the real cure we need guest to be fixed to accomodate that queue length, which is done only in the latest 4.14 kernel. Thus we are going to expose the property and tweak it on machine type level. The problem with the old kernels is that they have max_segments <= virtqueue_size restriction which cause the guest crashing in the case of violation. To fix the case described above in the old kernels we can increase virtqueue_size to 256 and max_segments to 254. The pitfall here is that seabios allows the virtqueue_size-s < 128, however, the seabios patch extending that value to 256 is pending. >>> If I understand correctly you are relying on Indirect Descriptor support >>> in the guest driver in order to exceed the Virtqueue Descriptor Table >>> size. >>> >>> Unfortunately the "max_segments <= virtqueue_size restriction" is >>> required by the VIRTIO 1.1 specification: >>> >>> 2.6.5.3.1 Driver Requirements: Indirect Descriptors >>> >>> A driver MUST NOT create a descriptor chain longer than the Queue >>> Size of the device. >>> >>> So this idea seems to be in violation of the specification? >>> >>> There is a bug in hw/block/virtio-blk.c:virtio_blk_update_config() and >>> hw/scsi/virtio-scsi.c:virtio_scsi_get_config(): >>> >>> virtio_stl_p(vdev, _max, 128 - 2); >>> >>> This number should be the minimum of blk_get_max_iov() and >>> virtio_queue_get_num(), minus 2 for the header and footer. >>> >>> I looked at the Linux SCSI driver code and it seems each HBA has a >>> single max_segments number - it does not vary on a per-device basis. >>> This could be a problem if two host block device with different >>> max_segments are exposed to the guest through the same virtio-scsi >>> controller. Another bug? :( >>> >>> Anyway, if you want ~1024 descriptors you should set Queue Size to 1024. >>> I don't see a spec-compliant way of doing it otherwise. Hopefully I >>> have overlooked something and there is a nice way to solve this. >>> >>> Stefan >> you are perfectly correct. We need actually 3 changes to improve >> guest behavior: >> 1) This patch, which adds property but does not change anything >> useful > This patch is problematic because it causes existing guest drivers to > violate the VIRTIO specification (or fail) if the value is set too high. > In addition, it does not take into account the virtqueue size so the > default value is too low when the user sets -device ...,queue-size=1024. > > Let's calculate blkcfg.seg_max based on the
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On Tue, Oct 22, 2019 at 04:01:57AM +, Denis Lunev wrote: > On 10/21/19 4:24 PM, Stefan Hajnoczi wrote: > > On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote: > >> From: "Denis V. Lunev" > >> > >> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg > >> field reported by SCSI controler. Thus typical sequential read with > >> 1 MB size results in the following pattern of the IO from the guest: > >> 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd] > >> 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd] > >> 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd] > >> 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0] > >> 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0] > >> 8,16 115759 2.768539782 0 C R 2097120 + 32 [0] > >> The IO was generated by > >> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct > >> > >> This effectively means that on rotational disks we will observe 3 IOPS > >> for each 2 MBs processed. This definitely negatively affects both > >> guest and host IO performance. > >> > >> The cure is relatively simple - we should report lengthy scatter-gather > >> ability of the SCSI controller. Fortunately the situation here is very > >> good. VirtIO transport layer can accomodate 1024 items in one request > >> while we are using only 128. This situation is present since almost > >> very beginning. 2 items are dedicated for request metadata thus we > >> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg. > >> > >> The following pattern is observed after the patch: > >> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd] > >> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd] > >> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0] > >> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0] > >> which is much better. > >> > >> The dark side of this patch is that we are tweaking guest visible > >> parameter, though this should be relatively safe as above transport > >> layer support is present in QEMU/host Linux for a very long time. > >> The patch adds configurable property for VirtIO SCSI with a new default > >> and hardcode option for VirtBlock which does not provide good > >> configurable framework. > >> > >> Unfortunately the commit can not be applied as is. For the real cure we > >> need guest to be fixed to accomodate that queue length, which is done > >> only in the latest 4.14 kernel. Thus we are going to expose the property > >> and tweak it on machine type level. > >> > >> The problem with the old kernels is that they have > >> max_segments <= virtqueue_size restriction which cause the guest > >> crashing in the case of violation. > >> To fix the case described above in the old kernels we can increase > >> virtqueue_size to 256 and max_segments to 254. The pitfall here is > >> that seabios allows the virtqueue_size-s < 128, however, the seabios > >> patch extending that value to 256 is pending. > > If I understand correctly you are relying on Indirect Descriptor support > > in the guest driver in order to exceed the Virtqueue Descriptor Table > > size. > > > > Unfortunately the "max_segments <= virtqueue_size restriction" is > > required by the VIRTIO 1.1 specification: > > > > 2.6.5.3.1 Driver Requirements: Indirect Descriptors > > > > A driver MUST NOT create a descriptor chain longer than the Queue > > Size of the device. > > > > So this idea seems to be in violation of the specification? > > > > There is a bug in hw/block/virtio-blk.c:virtio_blk_update_config() and > > hw/scsi/virtio-scsi.c:virtio_scsi_get_config(): > > > > virtio_stl_p(vdev, _max, 128 - 2); > > > > This number should be the minimum of blk_get_max_iov() and > > virtio_queue_get_num(), minus 2 for the header and footer. > > > > I looked at the Linux SCSI driver code and it seems each HBA has a > > single max_segments number - it does not vary on a per-device basis. > > This could be a problem if two host block device with different > > max_segments are exposed to the guest through the same virtio-scsi > > controller. Another bug? :( > > > > Anyway, if you want ~1024 descriptors you should set Queue Size to 1024. > > I don't see a spec-compliant way of doing it otherwise. Hopefully I > > have overlooked something and there is a nice way to solve this. > > > > Stefan > you are perfectly correct. We need actually 3 changes to improve > guest behavior: > 1) This patch, which adds property but does not change anything > useful This patch is problematic because it causes existing guest drivers to violate the VIRTIO specification (or fail) if the value is set too high. In addition, it does not take into account the virtqueue size so the default value is too low when the user sets -device ...,queue-size=1024. Let's calculate blkcfg.seg_max based on the virtqueue size as mentioned in my previous email instead.
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On 21.10.2019 16:24, Stefan Hajnoczi wrote: > On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote: >> From: "Denis V. Lunev" >> >> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg >> field reported by SCSI controler. Thus typical sequential read with >> 1 MB size results in the following pattern of the IO from the guest: >>8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd] >>8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd] >>8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd] >>8,16 115757 2.767668961 0 C R 2095104 + 1008 [0] >>8,16 115758 2.768534315 0 C R 2096112 + 1008 [0] >>8,16 115759 2.768539782 0 C R 2097120 + 32 [0] >> The IO was generated by >>dd if=/dev/sda of=/dev/null bs=1024 iflag=direct >> >> This effectively means that on rotational disks we will observe 3 IOPS >> for each 2 MBs processed. This definitely negatively affects both >> guest and host IO performance. >> >> The cure is relatively simple - we should report lengthy scatter-gather >> ability of the SCSI controller. Fortunately the situation here is very >> good. VirtIO transport layer can accomodate 1024 items in one request >> while we are using only 128. This situation is present since almost >> very beginning. 2 items are dedicated for request metadata thus we >> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg. >> >> The following pattern is observed after the patch: >>8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd] >>8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd] >>8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0] >>8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0] >> which is much better. >> >> The dark side of this patch is that we are tweaking guest visible >> parameter, though this should be relatively safe as above transport >> layer support is present in QEMU/host Linux for a very long time. >> The patch adds configurable property for VirtIO SCSI with a new default >> and hardcode option for VirtBlock which does not provide good >> configurable framework. >> >> Unfortunately the commit can not be applied as is. For the real cure we >> need guest to be fixed to accomodate that queue length, which is done >> only in the latest 4.14 kernel. Thus we are going to expose the property >> and tweak it on machine type level. >> >> The problem with the old kernels is that they have >> max_segments <= virtqueue_size restriction which cause the guest >> crashing in the case of violation. >> To fix the case described above in the old kernels we can increase >> virtqueue_size to 256 and max_segments to 254. The pitfall here is >> that seabios allows the virtqueue_size-s < 128, however, the seabios >> patch extending that value to 256 is pending. > If I understand correctly you are relying on Indirect Descriptor support > in the guest driver in order to exceed the Virtqueue Descriptor Table > size. > > Unfortunately the "max_segments <= virtqueue_size restriction" is > required by the VIRTIO 1.1 specification: > >2.6.5.3.1 Driver Requirements: Indirect Descriptors > >A driver MUST NOT create a descriptor chain longer than the Queue >Size of the device. > > So this idea seems to be in violation of the specification? > > There is a bug in hw/block/virtio-blk.c:virtio_blk_update_config() and > hw/scsi/virtio-scsi.c:virtio_scsi_get_config(): > >virtio_stl_p(vdev, _max, 128 - 2); > > This number should be the minimum of blk_get_max_iov() and > virtio_queue_get_num(), minus 2 for the header and footer. Stefan, It seems VitrioSCSI don't have a direct link to blk, apart of VirtIOBlock->blk, and the link to a blk comes with each scsi request. I suspect that idea here is that a single virtioscsi can serve several blk-s. If my assumption is corect, then we can't get blk_get_max_iov() on virtioscsi configuration stage and we shouldn't take into account max_iov and limit max_segments with virtio_queue_get_num()-2 only. Is it so, or is there any other details to take into account? Thanks! Denis > > I looked at the Linux SCSI driver code and it seems each HBA has a > single max_segments number - it does not vary on a per-device basis. > This could be a problem if two host block device with different > max_segments are exposed to the guest through the same virtio-scsi > controller. Another bug? :( > > Anyway, if you want ~1024 descriptors you should set Queue Size to 1024. > I don't see a spec-compliant way of doing it otherwise. Hopefully I > have overlooked something and there is a nice way to solve this. > > Stefan
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On 10/21/19 4:24 PM, Stefan Hajnoczi wrote: > On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote: >> From: "Denis V. Lunev" >> >> Linux guests submit IO requests no longer than PAGE_SIZE * max_seg >> field reported by SCSI controler. Thus typical sequential read with >> 1 MB size results in the following pattern of the IO from the guest: >> 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd] >> 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd] >> 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd] >> 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0] >> 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0] >> 8,16 115759 2.768539782 0 C R 2097120 + 32 [0] >> The IO was generated by >> dd if=/dev/sda of=/dev/null bs=1024 iflag=direct >> >> This effectively means that on rotational disks we will observe 3 IOPS >> for each 2 MBs processed. This definitely negatively affects both >> guest and host IO performance. >> >> The cure is relatively simple - we should report lengthy scatter-gather >> ability of the SCSI controller. Fortunately the situation here is very >> good. VirtIO transport layer can accomodate 1024 items in one request >> while we are using only 128. This situation is present since almost >> very beginning. 2 items are dedicated for request metadata thus we >> should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg. >> >> The following pattern is observed after the patch: >> 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd] >> 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd] >> 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0] >> 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0] >> which is much better. >> >> The dark side of this patch is that we are tweaking guest visible >> parameter, though this should be relatively safe as above transport >> layer support is present in QEMU/host Linux for a very long time. >> The patch adds configurable property for VirtIO SCSI with a new default >> and hardcode option for VirtBlock which does not provide good >> configurable framework. >> >> Unfortunately the commit can not be applied as is. For the real cure we >> need guest to be fixed to accomodate that queue length, which is done >> only in the latest 4.14 kernel. Thus we are going to expose the property >> and tweak it on machine type level. >> >> The problem with the old kernels is that they have >> max_segments <= virtqueue_size restriction which cause the guest >> crashing in the case of violation. >> To fix the case described above in the old kernels we can increase >> virtqueue_size to 256 and max_segments to 254. The pitfall here is >> that seabios allows the virtqueue_size-s < 128, however, the seabios >> patch extending that value to 256 is pending. > If I understand correctly you are relying on Indirect Descriptor support > in the guest driver in order to exceed the Virtqueue Descriptor Table > size. > > Unfortunately the "max_segments <= virtqueue_size restriction" is > required by the VIRTIO 1.1 specification: > > 2.6.5.3.1 Driver Requirements: Indirect Descriptors > > A driver MUST NOT create a descriptor chain longer than the Queue > Size of the device. > > So this idea seems to be in violation of the specification? > > There is a bug in hw/block/virtio-blk.c:virtio_blk_update_config() and > hw/scsi/virtio-scsi.c:virtio_scsi_get_config(): > > virtio_stl_p(vdev, _max, 128 - 2); > > This number should be the minimum of blk_get_max_iov() and > virtio_queue_get_num(), minus 2 for the header and footer. > > I looked at the Linux SCSI driver code and it seems each HBA has a > single max_segments number - it does not vary on a per-device basis. > This could be a problem if two host block device with different > max_segments are exposed to the guest through the same virtio-scsi > controller. Another bug? :( > > Anyway, if you want ~1024 descriptors you should set Queue Size to 1024. > I don't see a spec-compliant way of doing it otherwise. Hopefully I > have overlooked something and there is a nice way to solve this. > > Stefan you are perfectly correct. We need actually 3 changes to improve guest behavior: 1) This patch, which adds property but does not change anything useful 2) The patch to SeaBIOS, which extends maximum allowed queue size. Right now virtque > 128 results in assert (pending in SeaBIOS list). 3) Increase queue size and max_segments inside machine type. We have done that to 256 and 256 -2 respectively. I think that this exact patch with property does not harm and upon the acceptance could start a discussion about default queue length extension. Den
Re: [PATCH] virtio: fix IO request length in virtio SCSI/block #PSBM-78839
On Fri, Oct 18, 2019 at 02:55:47PM +0300, Denis Plotnikov wrote: > From: "Denis V. Lunev" > > Linux guests submit IO requests no longer than PAGE_SIZE * max_seg > field reported by SCSI controler. Thus typical sequential read with > 1 MB size results in the following pattern of the IO from the guest: > 8,16 115754 2.766095122 2071 D R 2095104 + 1008 [dd] > 8,16 115755 2.766108785 2071 D R 2096112 + 1008 [dd] > 8,16 115756 2.766113486 2071 D R 2097120 + 32 [dd] > 8,16 115757 2.767668961 0 C R 2095104 + 1008 [0] > 8,16 115758 2.768534315 0 C R 2096112 + 1008 [0] > 8,16 115759 2.768539782 0 C R 2097120 + 32 [0] > The IO was generated by > dd if=/dev/sda of=/dev/null bs=1024 iflag=direct > > This effectively means that on rotational disks we will observe 3 IOPS > for each 2 MBs processed. This definitely negatively affects both > guest and host IO performance. > > The cure is relatively simple - we should report lengthy scatter-gather > ability of the SCSI controller. Fortunately the situation here is very > good. VirtIO transport layer can accomodate 1024 items in one request > while we are using only 128. This situation is present since almost > very beginning. 2 items are dedicated for request metadata thus we > should publish VIRTQUEUE_MAX_SIZE - 2 as max_seg. > > The following pattern is observed after the patch: > 8,16 1 9921 2.662721340 2063 D R 2095104 + 1024 [dd] > 8,16 1 9922 2.662737585 2063 D R 2096128 + 1024 [dd] > 8,16 1 9923 2.665188167 0 C R 2095104 + 1024 [0] > 8,16 1 9924 2.665198777 0 C R 2096128 + 1024 [0] > which is much better. > > The dark side of this patch is that we are tweaking guest visible > parameter, though this should be relatively safe as above transport > layer support is present in QEMU/host Linux for a very long time. > The patch adds configurable property for VirtIO SCSI with a new default > and hardcode option for VirtBlock which does not provide good > configurable framework. > > Unfortunately the commit can not be applied as is. For the real cure we > need guest to be fixed to accomodate that queue length, which is done > only in the latest 4.14 kernel. Thus we are going to expose the property > and tweak it on machine type level. > > The problem with the old kernels is that they have > max_segments <= virtqueue_size restriction which cause the guest > crashing in the case of violation. > To fix the case described above in the old kernels we can increase > virtqueue_size to 256 and max_segments to 254. The pitfall here is > that seabios allows the virtqueue_size-s < 128, however, the seabios > patch extending that value to 256 is pending. If I understand correctly you are relying on Indirect Descriptor support in the guest driver in order to exceed the Virtqueue Descriptor Table size. Unfortunately the "max_segments <= virtqueue_size restriction" is required by the VIRTIO 1.1 specification: 2.6.5.3.1 Driver Requirements: Indirect Descriptors A driver MUST NOT create a descriptor chain longer than the Queue Size of the device. So this idea seems to be in violation of the specification? There is a bug in hw/block/virtio-blk.c:virtio_blk_update_config() and hw/scsi/virtio-scsi.c:virtio_scsi_get_config(): virtio_stl_p(vdev, _max, 128 - 2); This number should be the minimum of blk_get_max_iov() and virtio_queue_get_num(), minus 2 for the header and footer. I looked at the Linux SCSI driver code and it seems each HBA has a single max_segments number - it does not vary on a per-device basis. This could be a problem if two host block device with different max_segments are exposed to the guest through the same virtio-scsi controller. Another bug? :( Anyway, if you want ~1024 descriptors you should set Queue Size to 1024. I don't see a spec-compliant way of doing it otherwise. Hopefully I have overlooked something and there is a nice way to solve this. Stefan signature.asc Description: PGP signature
