This series reduces per-packet socket overhead in the virtio-vsock RX
worker.
Patch 1: separate socket lookup from locked packet processing.
Patch 2: factor out the RX skb metadata-budget check.
Patch 3: keep one socket lock across a bounded run of established
STREAM/RW packets for the same native socket, and reuse the locked socket
for later same-flow packets.
Patch 4: coalesce default write-space notifications while that
lock is held.
Patch 5: defer the default readable callback until after the batch
unlocks.
Performance:
Tested on a single-stream vsock connection (one Guest receiver thread on
CPU0, virtio-vsock IRQ on Guest CPU1, fresh QEMU Guest per state, with 16
balanced AB/BA pairs per payload for each baseline/patchset comparison).
The host uses vhost-vsock.
For throughput, vsock_perf runs on the host as the
sender, with a custom single-threaded Guest receiver pinned to Guest CPU0
using blocking recv(). The fixed-64K table uses a 64K buffer; the matched
table uses the sender's request size. The virtio-vsock IRQ/RX worker is
pinned to Guest CPU1. To normalize run times across payloads into a
bounded, deterministic window and avoid long runs being skewed by host
background noise or virtualization scheduling jitter, transfer sizes scale
proportionally with the request size (64B/64M, 256B/256M, etc.), capped at
2G for 4K–64K. The custom receiver keeps the recv() size and count
explicit; vsock_perf's receiver uses poll()+read().
Latency uses a custom user-space vsock ping-pong tool.
The measurements below were collected for the RX batching series.
This benchmark does not run vsock_perf on both ends. Instead, it uses
vsock_perf as the sender on the host (vhost-vsock) side and a custom
blocking recv() program without poll() on the Guest side, with the receiver
thread pinned to Guest CPU0 (and the virtio-vsock RX worker pinned to Guest
CPU1). Plain blocking recv() combined with strict core pinning gives
audit-level determinism for tracing a single-threaded TID.
With all patches applied:
End-to-end RX throughput (64K Guest recv() size, SO_RCVLOWAT=1; Gbit/s):
payload baseline RX PATCH RX throughput change (95% CI)
(Gbit/s) (Gbit/s)
64B 0.1290 0.2489 +92.91% [+89.31%, +96.58%]
256B 0.5780 0.9727 +68.29% [+66.14%, +70.48%]
512B 1.1688 1.8316 +56.70% [+53.85%, +59.61%]
1K 2.3059 3.3377 +44.75% [+43.39%, +46.12%]
4K 3.8787 5.7310 +47.76% [+45.20%, +50.36%]
8K 4.5983 5.4575 +18.68% [+16.42%, +20.99%]
16K 5.0806 5.5615 +9.47% [ +8.05%, +10.90%]
64K 5.4842 5.6423 +2.88% [ +1.13%, +4.67%]
Fixed-syscall stress test (receiver recv() size = sender write size;
Gbit/s):
payload baseline PATCH throughput change (95% CI) recv()
RX RX (base/patch)
(Gbit/s) (Gbit/s)
64B 0.1384 0.2532 +82.99% [+74.19%, +92.24%] 1048577/1048577
256B 0.4042 0.8983 +122.25% [+113.06%, +131.85%] 1048577/1048577
512B 0.8056 1.6035 +99.05% [+89.34%, +109.26%] 1048577/1048577
1K 1.6030 2.7158 +69.42% [+63.22%, +75.85%] 1048577/1048577
4K 3.0194 4.9586 +64.22% [+58.43%, +70.23%] 536799/525225
8K 3.8316 5.1255 +33.77% [+31.00%, +36.59%] 274864/263013
16K 4.5702 5.3808 +17.74% [+14.87%, +20.67%] 144958/131332
64K 5.4736 5.7118 +4.35% [ +2.79%, +5.94%] 49039/32938
Request-response latency (Ping-Pong RTT, 10,000 requests):
payload baseline mean RTT (us) PATCH mean RTT (us) change (95% CI)
64B 224.148 224.151 +0.00%
[-2.26%, +2.32%]
256B 222.037 224.499 +1.11%
[-1.66%, +3.95%]
4K 225.982 227.026 +0.46%
[-1.15%, +2.10%]
Guest CPU1 aggregate cycles per received byte (fixed 64K receiver, with perf):
Buffer size baseline cycles/B PATCH cycles/B cycles change (95% CI)
64B 65.282 57.843 -11.40% [-13.77%, -8.95%]
256B 14.710 13.813 -6.10% [ -7.89%, -4.28%]
512B 7.709 7.188 -6.75% [ -8.79%, -4.66%]
1K 4.075 3.862 -5.23% [ -6.80%, -3.63%]
4K 2.034 1.887 -7.21% [ -8.67%, -5.72%]
8K 1.664 1.568 -5.78% [ -7.83%, -3.69%]
16K 1.450 1.385 -4.48% [ -6.08%, -2.86%]
64K 1.338 1.317 -1.51% [ -3.21%, +0.22%]
P1-P3 contribution:
payload baseline RX PATCH RX throughput change (95% CI)
(Gbit/s) (Gbit/s)
64B 0.1285 0.2462 +91.59% [+79.74%, +104.22%]
256B 0.5909 0.9666 +63.59% [+61.03%, +66.19%]
512B 1.1817 1.8350 +55.29% [+46.76%, +64.32%]
1K 2.3252 3.3600 +44.50% [+38.93%, +50.30%]
4K 3.8436 5.5661 +44.82% [+40.30%, +49.47%]
8K 4.6609 5.4115 +16.10% [ +8.91%, +23.77%]
16K 5.1196 5.6399 +10.16% [ +4.07%, +16.61%]
64K 5.6095 5.7062 +1.72% [-3.19%, +6.89%]
Fixed-syscall stress test (receiver recv() size = sender write size;
no perf; Gbit/s):
payload baseline PATCH throughput change (95% CI) recv()
RX RX (base/patch)
(Gbit/s) (Gbit/s)
64B 0.1446 0.2540 +75.71% [+63.67%, +88.62%] 1048577/1048577
256B 0.4211 0.9061 +115.18% [+72.13%, +169.00%] 1048577/1048577
512B 0.7980 1.6112 +101.90% [+95.30%, +108.72%] 1048577/1048577
1K 1.4965 2.6986 +80.33% [+68.93%, +92.49%] 1048577/1048577
4K 2.9384 4.8590 +65.36% [+60.17%, +70.72%] 539544/524982
8K 3.8170 5.1887 +35.94% [+27.66%, +44.75%] 274857/262991
16K 4.6856 5.5996 +19.51% [+15.36%, +23.81%] 147518/131351
64K 5.5182 5.8800 +6.56% [-0.83%, +14.49%] 48492/32910
Guest CPU1 aggregate cycles per received byte (fixed 64K receiver, with perf):
Buffer size baseline cycles/B PATCH cycles/B cycles change (95% CI)
64B 66.728 61.220 -8.25% [-18.99%, +3.91%]
256B 14.363 14.515 +1.06% [ -4.22%, +6.64%]
512B 7.792 7.578 -2.74% [ -6.55%, +1.23%]
1K 3.988 4.032 +1.09% [ -1.64%, +3.88%]
4K 2.063 1.973 -4.38% [ -8.23%, -0.37%]
8K 1.702 1.667 -2.06% [ -7.99%, +4.25%]
16K 1.473 1.475 +0.17% [ -5.06%, +5.68%]
64K 1.385 1.358 -1.96% [-10.61%, +7.53%]
P4-P5 contribution (combined):
End-to-end RX throughput (64K Guest recv() size, SO_RCVLOWAT=1; Gbit/s):
payload P1-P3 RX P1-P5 RX throughput change (95% CI)
(Gbit/s) (Gbit/s)
64B 0.2449 0.2463 +0.55% [-3.86%, +5.17%]
256B 0.9649 0.9931 +2.93% [+0.83%, +5.07%]
512B 1.8205 1.8260 +0.30% [-4.27%, +5.10%]
1K 3.3431 3.3420 -0.03% [-2.15%, +2.12%]
4K 5.6541 5.8428 +3.34% [+1.10%, +5.63%]
8K 5.4280 5.3364 -1.69% [-15.55%, +14.46%]
16K 5.5935 5.6701 +1.37% [-1.41%, +4.23%]
64K 5.6917 5.8351 +2.52% [-0.20%, +5.31%]
Fixed-syscall stress test (receiver recv() size = sender write size;
Gbit/s):
payload P1-P3 P1-P5 throughput change (95% CI) recv()
RX RX (P1-P3/P1-P5)
(Gbit/s) (Gbit/s)
64B 0.2553 0.2575 +0.87% [-3.68%, +5.63%] 1048577/1048577
256B 0.9069 0.9183 +1.26% [-3.96%, +6.76%] 1048577/1048577
512B 1.6048 1.6459 +2.56% [+0.98%, +4.18%] 1048577/1048577
1K 2.6939 2.7357 +1.55% [-1.35%, +4.54%] 1048577/1048577
4K 4.9168 4.9678 +1.04% [-1.84%, +4.00%] 524965/525283
8K 5.1476 5.2005 +1.03% [-4.02%, +6.34%] 262926/263156
16K 5.5964 5.6609 +1.15% [-2.86%, +5.33%] 131311/131475
64K 5.9439 5.9073 -0.62% [-3.00%, +1.83%] 32875/32937
Guest CPU1 aggregate cycles per received byte (fixed 64K receiver, with perf):
payload P1-P3 P1-P5 change (95% CI)
64B 60.281 58.258 -3.36% [-8.74%, +2.35%]
256B 14.242 13.648 -4.17% [-9.81%, +1.82%]
512B 7.547 7.211 -4.45% [-9.73%, +1.13%]
1K 3.987 3.853 -3.35% [-8.52%, +2.12%]
4K 1.983 1.880 -5.17% [-7.97%, -2.27%]
8K 1.665 1.620 -2.75% [-9.03%, +3.96%]
16K 1.409 1.346 -4.52% [-10.94%, +2.37%]
64K 1.322 1.281 -3.11% [-6.49%, +0.38%]
