I saw two noteworthy items in your 'vppctl show runtime' output: 1. All of the packets received/processed appear to be handled by the gtpu4-input node. They also all appear to be received/handled by a single worker thread. 2. Nearly all of the packets are being dropped. I mean the packets that were actually received and processed - not the packets that were counted as an rx-miss.
Regarding #1 - I imagine that one thread might be receiving all inbound packets because you're sending across a single GTPU tunnel (i.e. a single stream). If this is true, AFAIK most NICs will hash all of the packets to the same queue. This means you will likely be constrained to handling however many packets can be handled by a single thread and increasing the number of workers or rx queues won't help. You might be able to utilize multiple workers/queues by sending across multiple tunnels. I know very little about GTPU use cases so I don't know whether it's practical for you to use multiple tunnels. Regarding #2 - Out of 8004674 packets received by dpdk-input, 7977696 packets end up being dropped by error-drop. It would probably be useful to look at a packet trace and see what node is sending the packets to error-drop. If packets are being passed to error-drop by gtpu4-input, maybe they do not match any configured tunnel. -Matt On Tue, Sep 28, 2021 at 9:24 AM Akash S R <akashsr.akas...@gmail.com> wrote: > Hello, > > I have tried increasing the workers from 2 to 5 and rx/tx queues from 1024 > (default) to 2048 ,also decreasing till 256 but of no use. > As you told, vector is very high (> 100). Please let us know if there is > any other way or reason for the same. > > > Thread 1 vpp_wk_0 (lcore 2) > Time 41.3, 10 sec internal node vector rate 79.42 loops/sec 3628.25 > vector rates in 1.9388e5, out 6.5319e2, drop 1.9322e5, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > TenGigabitEthernet4/0/0-output active 19735 26978 > 0 2.12e3 1.37 > TenGigabitEthernet4/0/0-tx active 19735 26969 > 0 1.43e3 1.37 > dpdk-input polling 10241833 8004674 > 0 2.81e3 .78 > drop active 92422 7977696 > 0 1.31e3 86.32 > error-drop active 92422 7977696 > 0 6.82e1 86.32 > ethernet-input active 93109 8004674 > 0 1.84e2 85.97 > gtpu4-input active 93106 8004671 > 0 3.29e2 85.97 > ip4-drop active 2 2 > 0 1.33e4 1.00 > ip4-input active 93106 8004671 > 0 6.82e2 85.97 > ip4-input-no-checksum active 93106 8004673 > 0 2.37e2 85.97 > ip4-local active 93106 8004671 > 0 2.66e2 85.97 > ip4-lookup active 112841 8031651 > 0 3.55e2 71.18 > ip4-policer-classify active 93106 8004671 > 0 1.35e3 85.97 > ip4-rewrite active 19735 26978 > 0 2.43e3 1.37 > ip4-udp-lookup active 93106 8004671 > 0 3.17e2 85.97 > ip6-input active 1 1 > 0 1.99e4 1.00 > ip6-not-enabled active 1 1 > 0 2.59e4 1.00 > unix-epoll-input polling 9998 0 > 0 3.30e3 0.00 > --------------- > Thread 2 vpp_wk_1 (lcore 3) > Time 41.3, 10 sec internal node vector rate 0.00 loops/sec 988640.31 > vector rates in 0.0000e0, out 0.0000e0, drop 0.0000e0, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > dpdk-input polling 37858263 0 > 0 1.14e3 0.00 > unix-epoll-input polling 36940 0 > 0 3.54e3 0.00 > --------------- > Thread 3 vpp_wk_2 (lcore 4) > Time 41.3, 10 sec internal node vector rate 0.00 loops/sec 983700.23 > vector rates in 4.8441e-2, out 0.0000e0, drop 4.8441e-2, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > dpdk-input polling 37820862 2 > 0 2.16e10 0.00 > drop active 2 2 > 0 9.34e3 1.00 > error-drop active 2 2 > 0 7.78e3 1.00 > ethernet-input active 2 2 > 0 2.44e4 1.00 > ip6-input active 2 2 > 0 1.06e4 1.00 > ip6-not-enabled active 2 2 > 0 8.75e3 1.00 > unix-epoll-input polling 36906 0 > 0 3.54e3 0.00 > --------------- > Thread 4 vpp_wk_3 (lcore 5) > Time 41.3, 10 sec internal node vector rate 0.00 loops/sec 985307.67 > vector rates in 0.0000e0, out 0.0000e0, drop 0.0000e0, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > dpdk-input polling 37834354 0 > 0 1.14e3 0.00 > unix-epoll-input polling 36918 0 > 0 3.47e3 0.00 > > Thanks, > Akash > > On Tue, Sep 28, 2021 at 4:09 PM Akash S R via lists.fd.io > <akashsr.akashsr=gmail....@lists.fd.io> wrote: > >> Thanks mate, that makes sense. Will check it out and get back . >> >> >> Thanks, >> Akash >> >> On Tue, Sep 28, 2021, 16:00 Benoit Ganne (bganne) <bga...@cisco.com> >> wrote: >> >>> Rx-miss means the NIC must drop packets on RX because the rx queue was >>> full, usually because VPP cannot keep up with the incoming packet rate. >>> You can check it with the output of 'show run'. If you see a big average >>> vector size (100 or more) it means VPP is busy. >>> To improve that you must increase the number of VPP workers (and rx >>> queues). >>> Rx-misses can also be caused by traffic spikes. In that case you can >>> increase the rx queues size to absorb the bursts. >>> >>> Best >>> ben >>> >>> > -----Original Message----- >>> > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Akash S R >>> > Sent: mardi 28 septembre 2021 11:43 >>> > To: vpp-dev <vpp-dev@lists.fd.io> >>> > Subject: [vpp-dev] rx-miss while sending packets to Interface >>> (IMPORTANT) >>> > >>> > Hello mates, >>> > >>> > Its been a long time, me , raising a query out here to you guys :) >>> (Nah, >>> > Please ignore this junk.) >>> > >>> > I have a question on rx-miss. Whenever we fire packets at some high >>> rate, >>> > say 1GBPS or more, I get rx-miss on the interface with some packet >>> count. >>> > I referred to a link below from the VPP forum: >>> https://lists.fd.io/g/vpp- >>> > dev/topic/78179815#17985 >>> > >>> > TenGigabitEthernet4/0/1 3 up 1500/0/0/0 rx >>> > packets 8622948 >>> > rx >>> > bytes 1103737344 >>> > ip4 >>> > 8622948 >>> > rx- >>> > miss 283721416 >>> > >>> > But the buffer is available and allocated is less. I reduced the >>> queues to >>> > 256 and threads to 4. >>> > But still the issue is not resolved. >>> > DBGvpp# show dpdk buffer >>> > name="vpp pool 0" available = 13984 allocated = 2816 total = >>> 16800 >>> > >>> > May I know the reason for this rx-miss and any kinda resolution for >>> this >>> > issue? >>> > >>> > Thanks, >>> > Akash >>> >>> >> >> >> > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20229): https://lists.fd.io/g/vpp-dev/message/20229 Mute This Topic: https://lists.fd.io/mt/85920772/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
