On Wed, Feb 4, 2026 at 11:44 PM Simon Schippers <[email protected]> wrote: > > On 2/3/26 04:48, Jason Wang wrote: > > On Mon, Feb 2, 2026 at 4:19 AM Simon Schippers > > <[email protected]> wrote: > >> > >> On 1/30/26 02:51, Jason Wang wrote: > >>> On Thu, Jan 29, 2026 at 5:25 PM Simon Schippers > >>> <[email protected]> wrote: > >>>> > >>>> On 1/29/26 02:14, Jason Wang wrote: > >>>>> On Wed, Jan 28, 2026 at 3:54 PM Simon Schippers > >>>>> <[email protected]> wrote: > >>>>>> > >>>>>> On 1/28/26 08:03, Jason Wang wrote: > >>>>>>> On Wed, Jan 28, 2026 at 12:48 AM Simon Schippers > >>>>>>> <[email protected]> wrote: > >>>>>>>> > >>>>>>>> On 1/23/26 10:54, Simon Schippers wrote: > >>>>>>>>> On 1/23/26 04:05, Jason Wang wrote: > >>>>>>>>>> On Thu, Jan 22, 2026 at 1:35 PM Jason Wang <[email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Jan 21, 2026 at 5:33 PM Simon Schippers > >>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 1/9/26 07:02, Jason Wang wrote: > >>>>>>>>>>>>> On Thu, Jan 8, 2026 at 3:41 PM Simon Schippers > >>>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 1/8/26 04:38, Jason Wang wrote: > >>>>>>>>>>>>>>> On Thu, Jan 8, 2026 at 5:06 AM Simon Schippers > >>>>>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Introduce {tun,tap}_ring_consume() helpers that wrap > >>>>>>>>>>>>>>>> __ptr_ring_consume() > >>>>>>>>>>>>>>>> and wake the corresponding netdev subqueue when consuming an > >>>>>>>>>>>>>>>> entry frees > >>>>>>>>>>>>>>>> space in the underlying ptr_ring. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Stopping of the netdev queue when the ptr_ring is full will > >>>>>>>>>>>>>>>> be introduced > >>>>>>>>>>>>>>>> in an upcoming commit. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Co-developed-by: Tim Gebauer <[email protected]> > >>>>>>>>>>>>>>>> Signed-off-by: Tim Gebauer <[email protected]> > >>>>>>>>>>>>>>>> Signed-off-by: Simon Schippers > >>>>>>>>>>>>>>>> <[email protected]> > >>>>>>>>>>>>>>>> --- > >>>>>>>>>>>>>>>> drivers/net/tap.c | 23 ++++++++++++++++++++++- > >>>>>>>>>>>>>>>> drivers/net/tun.c | 25 +++++++++++++++++++++++-- > >>>>>>>>>>>>>>>> 2 files changed, 45 insertions(+), 3 deletions(-) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> diff --git a/drivers/net/tap.c b/drivers/net/tap.c > >>>>>>>>>>>>>>>> index 1197f245e873..2442cf7ac385 100644 > >>>>>>>>>>>>>>>> --- a/drivers/net/tap.c > >>>>>>>>>>>>>>>> +++ b/drivers/net/tap.c > >>>>>>>>>>>>>>>> @@ -753,6 +753,27 @@ static ssize_t tap_put_user(struct > >>>>>>>>>>>>>>>> tap_queue *q, > >>>>>>>>>>>>>>>> return ret ? ret : total; > >>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> +static void *tap_ring_consume(struct tap_queue *q) > >>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>> + struct ptr_ring *ring = &q->ring; > >>>>>>>>>>>>>>>> + struct net_device *dev; > >>>>>>>>>>>>>>>> + void *ptr; > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> + spin_lock(&ring->consumer_lock); > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> + ptr = __ptr_ring_consume(ring); > >>>>>>>>>>>>>>>> + if (unlikely(ptr && > >>>>>>>>>>>>>>>> __ptr_ring_consume_created_space(ring, 1))) { > >>>>>>>>>>>>>>>> + rcu_read_lock(); > >>>>>>>>>>>>>>>> + dev = rcu_dereference(q->tap)->dev; > >>>>>>>>>>>>>>>> + netif_wake_subqueue(dev, q->queue_index); > >>>>>>>>>>>>>>>> + rcu_read_unlock(); > >>>>>>>>>>>>>>>> + } > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> + spin_unlock(&ring->consumer_lock); > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> + return ptr; > >>>>>>>>>>>>>>>> +} > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> static ssize_t tap_do_read(struct tap_queue *q, > >>>>>>>>>>>>>>>> struct iov_iter *to, > >>>>>>>>>>>>>>>> int noblock, struct sk_buff *skb) > >>>>>>>>>>>>>>>> @@ -774,7 +795,7 @@ static ssize_t tap_do_read(struct > >>>>>>>>>>>>>>>> tap_queue *q, > >>>>>>>>>>>>>>>> TASK_INTERRUPTIBLE); > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> /* Read frames from the queue */ > >>>>>>>>>>>>>>>> - skb = ptr_ring_consume(&q->ring); > >>>>>>>>>>>>>>>> + skb = tap_ring_consume(q); > >>>>>>>>>>>>>>>> if (skb) > >>>>>>>>>>>>>>>> break; > >>>>>>>>>>>>>>>> if (noblock) { > >>>>>>>>>>>>>>>> diff --git a/drivers/net/tun.c b/drivers/net/tun.c > >>>>>>>>>>>>>>>> index 8192740357a0..7148f9a844a4 100644 > >>>>>>>>>>>>>>>> --- a/drivers/net/tun.c > >>>>>>>>>>>>>>>> +++ b/drivers/net/tun.c > >>>>>>>>>>>>>>>> @@ -2113,13 +2113,34 @@ static ssize_t tun_put_user(struct > >>>>>>>>>>>>>>>> tun_struct *tun, > >>>>>>>>>>>>>>>> return total; > >>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> +static void *tun_ring_consume(struct tun_file *tfile) > >>>>>>>>>>>>>>>> +{ > >>>>>>>>>>>>>>>> + struct ptr_ring *ring = &tfile->tx_ring; > >>>>>>>>>>>>>>>> + struct net_device *dev; > >>>>>>>>>>>>>>>> + void *ptr; > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> + spin_lock(&ring->consumer_lock); > >>>>>>>>>>>>>>>> + > >>>>>>>>>>>>>>>> + ptr = __ptr_ring_consume(ring); > >>>>>>>>>>>>>>>> + if (unlikely(ptr && > >>>>>>>>>>>>>>>> __ptr_ring_consume_created_space(ring, 1))) { > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I guess it's the "bug" I mentioned in the previous patch that > >>>>>>>>>>>>>>> leads to > >>>>>>>>>>>>>>> the check of __ptr_ring_consume_created_space() here. If it's > >>>>>>>>>>>>>>> true, > >>>>>>>>>>>>>>> another call to tweak the current API. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> + rcu_read_lock(); > >>>>>>>>>>>>>>>> + dev = rcu_dereference(tfile->tun)->dev; > >>>>>>>>>>>>>>>> + netif_wake_subqueue(dev, tfile->queue_index); > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This would cause the producer TX_SOFTIRQ to run on the same > >>>>>>>>>>>>>>> cpu which > >>>>>>>>>>>>>>> I'm not sure is what we want. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> What else would you suggest calling to wake the queue? > >>>>>>>>>>>>> > >>>>>>>>>>>>> I don't have a good method in my mind, just want to point out > >>>>>>>>>>>>> its implications. > >>>>>>>>>>>> > >>>>>>>>>>>> I have to admit I'm a bit stuck at this point, particularly with > >>>>>>>>>>>> this > >>>>>>>>>>>> aspect. > >>>>>>>>>>>> > >>>>>>>>>>>> What is the correct way to pass the producer CPU ID to the > >>>>>>>>>>>> consumer? > >>>>>>>>>>>> Would it make sense to store smp_processor_id() in the tfile > >>>>>>>>>>>> inside > >>>>>>>>>>>> tun_net_xmit(), or should it instead be stored in the skb > >>>>>>>>>>>> (similar to the > >>>>>>>>>>>> XDP bit)? In the latter case, my concern is that this > >>>>>>>>>>>> information may > >>>>>>>>>>>> already be significantly outdated by the time it is used. > >>>>>>>>>>>> > >>>>>>>>>>>> Based on that, my idea would be for the consumer to wake the > >>>>>>>>>>>> producer by > >>>>>>>>>>>> invoking a new function (e.g., tun_wake_queue()) on the producer > >>>>>>>>>>>> CPU via > >>>>>>>>>>>> smp_call_function_single(). > >>>>>>>>>>>> Is this a reasonable approach? > >>>>>>>>>>> > >>>>>>>>>>> I'm not sure but it would introduce costs like IPI. > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> More generally, would triggering TX_SOFTIRQ on the consumer CPU > >>>>>>>>>>>> be > >>>>>>>>>>>> considered a deal-breaker for the patch set? > >>>>>>>>>>> > >>>>>>>>>>> It depends on whether or not it has effects on the performance. > >>>>>>>>>>> Especially when vhost is pinned. > >>>>>>>>>> > >>>>>>>>>> I meant we can benchmark to see the impact. For example, pin vhost > >>>>>>>>>> to > >>>>>>>>>> a specific CPU and the try to see the impact of the TX_SOFTIRQ. > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> I ran benchmarks with vhost pinned to CPU 0 using taskset -p -c 0 > >>>>>>>>> ... > >>>>>>>>> for both the stock and patched versions. The benchmarks were run > >>>>>>>>> with > >>>>>>>>> the full patch series applied, since testing only patches 1-3 would > >>>>>>>>> not > >>>>>>>>> be meaningful - the queue is never stopped in that case, so no > >>>>>>>>> TX_SOFTIRQ is triggered. > >>>>>>>>> > >>>>>>>>> Compared to the non-pinned CPU benchmarks in the cover letter, > >>>>>>>>> performance is lower for pktgen with a single thread but higher with > >>>>>>>>> four threads. The results show no regression for the patched > >>>>>>>>> version, > >>>>>>>>> with even slight performance improvements observed: > >>>>>>>>> > >>>>>>>>> +-------------------------+-----------+----------------+ > >>>>>>>>> | pktgen benchmarks to | Stock | Patched with | > >>>>>>>>> | Debian VM, i5 6300HQ, | | fq_codel qdisc | > >>>>>>>>> | 100M packets | | | > >>>>>>>>> | vhost pinned to core 0 | | | > >>>>>>>>> +-----------+-------------+-----------+----------------+ > >>>>>>>>> | TAP | Transmitted | 452 Kpps | 454 Kpps | > >>>>>>>>> | + +-------------+-----------+----------------+ > >>>>>>>>> | vhost-net | Lost | 1154 Kpps | 0 | > >>>>>>>>> +-----------+-------------+-----------+----------------+ > >>>>>>>>> > >>>>>>>>> +-------------------------+-----------+----------------+ > >>>>>>>>> | pktgen benchmarks to | Stock | Patched with | > >>>>>>>>> | Debian VM, i5 6300HQ, | | fq_codel qdisc | > >>>>>>>>> | 100M packets | | | > >>>>>>>>> | vhost pinned to core 0 | | | > >>>>>>>>> | *4 threads* | | | > >>>>>>>>> +-----------+-------------+-----------+----------------+ > >>>>>>>>> | TAP | Transmitted | 71 Kpps | 79 Kpps | > >>>>>>>>> | + +-------------+-----------+----------------+ > >>>>>>>>> | vhost-net | Lost | 1527 Kpps | 0 | > >>>>>>>>> +-----------+-------------+-----------+----------------+ > >>>>>>> > >>>>>>> The PPS seems to be low. I'd suggest using testpmd (rxonly) mode in > >>>>>>> the guest or an xdp program that did XDP_DROP in the guest. > >>>>>> > >>>>>> I forgot to mention that these PPS values are per thread. > >>>>>> So overall we have 71 Kpps * 4 = 284 Kpps and 79 Kpps * 4 = 326 Kpps, > >>>>>> respectively. For packet loss, that comes out to 1154 Kpps * 4 = > >>>>>> 4616 Kpps and 0, respectively. > >>>>>> > >>>>>> Sorry about that! > >>>>>> > >>>>>> The pktgen benchmarks with a single thread look fine, right? > >>>>> > >>>>> Still looks very low. E.g I just have a run of pktgen (using > >>>>> pktgen_sample03_burst_single_flow.sh) without a XDP_DROP in the guest, > >>>>> I can get 1Mpps. > >>>> > >>>> Keep in mind that I am using an older CPU (i5-6300HQ). For the > >>>> single-threaded tests I always used pktgen_sample01_simple.sh, and for > >>>> the multi-threaded tests I always used pktgen_sample02_multiqueue.sh. > >>>> > >>>> Using pktgen_sample03_burst_single_flow.sh as you did fails for me (even > >>>> though the same parameters work fine for sample01 and sample02): > >>>> > >>>> samples/pktgen/pktgen_sample03_burst_single_flow.sh -i tap0 -m > >>>> 52:54:00:12:34:56 -d 10.0.0.2 -n 100000000 > >>>> /samples/pktgen/functions.sh: line 79: echo: write error: Operation not > >>>> supported > >>>> ERROR: Write error(1) occurred > >>>> cmd: "burst 32 > /proc/net/pktgen/tap0@0" > >>>> > >>>> ...and I do not know what I am doing wrong, even after looking at > >>>> Documentation/networking/pktgen.rst. Every burst size except 1 fails. > >>>> Any clues? > >>> > >>> Please use -b 0, and I'm Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz. > >> > >> I tried using "-b 0", and while it worked, there was no noticeable > >> performance improvement. > >> > >>> > >>> Another thing I can think of is to disable > >>> > >>> 1) mitigations in both guest and host > >>> 2) any kernel debug features in both host and guest > >> > >> I also rebuilt the kernel with everything disabled under > >> "Kernel hacking", but that didn’t make any difference either. > >> > >> Because of this, I ran "pktgen_sample01_simple.sh" and > >> "pktgen_sample02_multiqueue.sh" on my AMD Ryzen 5 5600X system. The > >> results were about 374 Kpps with TAP and 1192 Kpps with TAP+vhost_net, > >> with very similar performance between the stock and patched kernels. > >> > >> Personally, I think the low performance is to blame on the hardware. > > > > Let's double confirm this by: > > > > 1) make sure pktgen is using 100% CPU > > 2) Perf doesn't show anything strange for pktgen thread > > > > Thanks > > > > I ran pktgen using pktgen_sample01_simple.sh and, in parallel, started a > 100 second perf stat measurement covering all kpktgend threads. > > Across all configurations, a single CPU was fully utilized. > > Apart from that, the patched variants show a higher branch frequency and > a slightly increased number of context switches. > > > The detailed results are provided below: > > Processor: Ryzen 5 5600X > > pktgen command: > sudo perf stat samples/pktgen/pktgen_sample01_simple.sh -i tap0 -m > 52:54:00:12:34:56 -d 10.0.0.2 -n 10000000000 > > perf stat command: > sudo perf stat --timeout 100000 -p $(pgrep kpktgend | tr '\n' ,) -o X.txt > > > Results: > Stock TAP: > 46.997 context-switches # 467,2 cs/sec > cs_per_second > 0 cpu-migrations # 0,0 > migrations/sec migrations_per_second > 0 page-faults # 0,0 > faults/sec page_faults_per_second > 100.587,69 msec task-clock # 1,0 CPUs > CPUs_utilized > 8.491.586.483 branch-misses # 10,9 % > branch_miss_rate (50,24%) > 77.734.761.406 branches # 772,8 M/sec > branch_frequency (66,85%) > 382.420.291.585 cpu-cycles # 3,8 GHz > cycles_frequency (66,85%) > 377.612.185.141 instructions # 1,0 > instructions insn_per_cycle (66,85%) > 84.012.185.936 stalled-cycles-frontend # 0,22 > frontend_cycles_idle (66,35%) > > 100,100414494 seconds time elapsed > > > Stock TAP+vhost-net: > 47.087 context-switches # 468,1 cs/sec > cs_per_second > 0 cpu-migrations # 0,0 > migrations/sec migrations_per_second > 0 page-faults # 0,0 > faults/sec page_faults_per_second > 100.594,09 msec task-clock # 1,0 CPUs > CPUs_utilized > 8.034.703.613 branch-misses # 11,1 % > branch_miss_rate (50,24%) > 72.477.989.922 branches # 720,5 M/sec > branch_frequency (66,86%) > 382.218.276.832 cpu-cycles # 3,8 GHz > cycles_frequency (66,85%) > 349.555.577.281 instructions # 0,9 > instructions insn_per_cycle (66,85%) > 83.917.644.262 stalled-cycles-frontend # 0,22 > frontend_cycles_idle (66,35%) > > 100,100520402 seconds time elapsed > > > Patched TAP: > 47.862 context-switches # 475,8 cs/sec > cs_per_second > 0 cpu-migrations # 0,0 > migrations/sec migrations_per_second > 0 page-faults # 0,0 > faults/sec page_faults_per_second > 100.589,30 msec task-clock # 1,0 CPUs > CPUs_utilized > 9.337.258.794 branch-misses # 9,4 % > branch_miss_rate (50,19%) > 99.518.421.676 branches # 989,4 M/sec > branch_frequency (66,85%) > 382.508.244.894 cpu-cycles # 3,8 GHz > cycles_frequency (66,85%) > 312.582.270.975 instructions # 0,8 > instructions insn_per_cycle (66,85%) > 76.338.503.984 stalled-cycles-frontend # 0,20 > frontend_cycles_idle (66,39%) > > 100,101262454 seconds time elapsed > > > Patched TAP+vhost-net: > 47.892 context-switches # 476,1 cs/sec > cs_per_second > 0 cpu-migrations # 0,0 > migrations/sec migrations_per_second > 0 page-faults # 0,0 > faults/sec page_faults_per_second > 100.581,95 msec task-clock # 1,0 CPUs > CPUs_utilized > 9.083.588.313 branch-misses # 10,1 % > branch_miss_rate (50,28%) > 90.300.124.712 branches # 897,8 M/sec > branch_frequency (66,85%) > 382.374.510.376 cpu-cycles # 3,8 GHz > cycles_frequency (66,85%) > 340.089.181.199 instructions # 0,9 > instructions insn_per_cycle (66,85%) > 78.151.408.955 stalled-cycles-frontend # 0,20 > frontend_cycles_idle (66,31%) > > 100,101212911 seconds time elapsed
Thanks for sharing. I have more questions: 1) The number of CPU and vCPUs 2) If you pin vhost or vCPU threads 3) what does perf top looks like or perf top -p $pid_of_vhost > > >> > >> Thanks! > >> > >>> > >>> Thanks > >>> > >>>> > >>>> Thanks! > >>>> > >>>>> > >>>>>> > >>>>>> I'll still look into using an XDP program that does XDP_DROP in the > >>>>>> guest. > >>>>>> > >>>>>> Thanks! > >>>>> > >>>>> Thanks > >>>>> > >>>>>> > >>>>>>> > >>>>>>>>> > >>>>>>>>> +------------------------+-------------+----------------+ > >>>>>>>>> | iperf3 TCP benchmarks | Stock | Patched with | > >>>>>>>>> | to Debian VM 120s | | fq_codel qdisc | > >>>>>>>>> | vhost pinned to core 0 | | | > >>>>>>>>> +------------------------+-------------+----------------+ > >>>>>>>>> | TAP | 22.0 Gbit/s | 22.0 Gbit/s | > >>>>>>>>> | + | | | > >>>>>>>>> | vhost-net | | | > >>>>>>>>> +------------------------+-------------+----------------+ > >>>>>>>>> > >>>>>>>>> +---------------------------+-------------+----------------+ > >>>>>>>>> | iperf3 TCP benchmarks | Stock | Patched with | > >>>>>>>>> | to Debian VM 120s | | fq_codel qdisc | > >>>>>>>>> | vhost pinned to core 0 | | | > >>>>>>>>> | *4 iperf3 client threads* | | | > >>>>>>>>> +---------------------------+-------------+----------------+ > >>>>>>>>> | TAP | 21.4 Gbit/s | 21.5 Gbit/s | > >>>>>>>>> | + | | | > >>>>>>>>> | vhost-net | | | > >>>>>>>>> +---------------------------+-------------+----------------+ > >>>>>>>> > >>>>>>>> What are your thoughts on this? > >>>>>>>> > >>>>>>>> Thanks! > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > > >
