On Sat, May 17, 2025 at 11:11:54PM +0300, Nir Soffer wrote: > Like macOS we have similar issue on Linux. For TCP socket the send > buffer size is 2626560 bytes (~2.5 MiB) and we get good performance. > However for unix socket the default and maximum buffer size is 212992 > bytes (208 KiB) and we see poor performance when using one NBD > connection, up to 4 times slower than macOS on the same machine. > > Tracing shows that for every 2 MiB payload (qemu uses 2 MiB io size), we > do 1 recvmsg call with TCP socket, and 10 recvmsg calls with unix > socket. > > Fixing this issue requires changing the maximum send buffer size (the > receive buffer size is ignored). This can be done using: > > $ cat /etc/sysctl.d/net-mem-max.conf > net.core.wmem_max = 2097152 > > $ sudo sysctl -p /etc/sysctl.d/net-mem-max.conf > > With this we can set the socket buffer size to 2 MiB. With the defaults > the value requested by qemu is clipped to the maximum size and has no > effect. > > I tested on 2 machines: > - Fedora 42 VM on MacBook Pro M2 Max > - Dell PowerEdge R640 (Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz) > > On the older Dell machine we see very little improvement, up to 1.03 > higher throughput. On the M2 machine we see up to 2.67 times higher > throughput. The following results are from the M2 machine. > > Reading from qemu-nbd with qemu-img convert. In this test buffer size of > 4m is optimal (2.28 times faster). > > | buffer size | time | user | system | > |-------------|---------|---------|---------| > | default | 4.292 | 0.243 | 1.604 | > | 524288 | 2.167 | 0.058 | 1.288 | > | 1048576 | 2.041 | 0.060 | 1.238 | > | 2097152 | 1.884 | 0.060 | 1.191 | > | 4194304 | 1.881 | 0.054 | 1.196 | > > Writing to qemu-nbd with qemu-img convert. In this test buffer size of > 1m is optimal (2.67 times faster). > > | buffer size | time | user | system | > |-------------|---------|---------|---------| > | default | 3.113 | 0.334 | 1.094 | > | 524288 | 1.173 | 0.179 | 0.654 | > | 1048576 | 1.164 | 0.164 | 0.670 | > | 2097152 | 1.227 | 0.197 | 0.663 | > | 4194304 | 1.227 | 0.198 | 0.666 | > > Computing a blkhash with nbdcopy. In this test buffer size of 512k is > optimal (1.19 times faster). > > | buffer size | time | user | system | > |-------------|---------|---------|---------| > | default | 2.140 | 4.483 | 2.681 | > | 524288 | 1.794 | 4.467 | 2.572 | > | 1048576 | 1.807 | 4.447 | 2.644 | > | 2097152 | 1.822 | 4.461 | 2.698 | > | 4194304 | 1.827 | 4.465 | 2.700 | > > Computing a blkhash with blksum. In this test buffer size of 4m is > optimal (2.65 times faster). > > | buffer size | time | user | system | > |-------------|---------|---------|---------| > | default | 3.582 | 4.595 | 2.392 | > | 524288 | 1.499 | 4.384 | 1.482 | > | 1048576 | 1.377 | 4.381 | 1.345 | > | 2097152 | 1.388 | 4.389 | 1.354 | > | 4194304 | 1.352 | 4.395 | 1.302 | > > Signed-off-by: Nir Soffer <nir...@gmail.com> > --- > nbd/common.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-)
Reviewed-by: Daniel P. Berrangé <berra...@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
