On both macOS and Linux, the default send buffer size is too small causing poor performance when reading and writing to qemu-nbd. A simple way to experience this is to compare TCP and unix sockets, showing that TCP socket is much faster. Programs like nbdcopy partly mitigate this by using multiple NBD connections.
On macOS the default send buffer size is 8192 bytes. Increasing the send buffer size to 2 MiB shows up to *12.6 times higher throughput* and lower cpu usage. On Linux the default and maximum buffer size is 212992 bytes. Increasing the send buffer size to 2 MiB shows up to *2.7 times higher throughput* and lower cpu usage. On older machine we see very little improvement, up to 1.03 times higher throughput. We likely have the same issue on other platforms. It should be easy to enable this change for more platform by defining UNIX_STREAM_SOCKET_SEND_BUFFER_SIZE. Changes since v3: - Modify only NBD sockets, since not all sockets need a larger buffer size, and the fixed memory overhead per socker may be a significant amount of extra memory overhead on a host. (Daniel) - Add qio_channel_socket_set_send_buffer() helper to set buffer size from nbd code. - Remove the check for SOCK_STREAM since NBD does not support SOCK_DGRAM. - Warn if we fail to set the socket send buffer size. v3 was here: https://lists.gnu.org/archive/html/qemu-devel/2025-04/msg05096.html Nir Soffer (3): io: Add helper for setting socket send buffer size nbd: Set unix socket send buffer on macOS nbd: Set unix socket send buffer on Linux include/io/channel-socket.h | 13 +++++++++++++ io/channel-socket.c | 11 +++++++++++ nbd/client-connection.c | 3 +++ nbd/common.c | 26 ++++++++++++++++++++++++++ nbd/nbd-internal.h | 5 +++++ nbd/server.c | 2 ++ 6 files changed, 60 insertions(+) -- 2.39.5 (Apple Git-154)
