Il 1/11/22 7:02 PM, Stephen Hemminger ha scritto: > On Tue, 11 Jan 2022 13:12:24 +0100 > Filip Janiszewski <[email protected]> wrote: > >> Hi, >> >> Is there any specific reason why using rte_pktmbuf_free_bulk seems to be >> much slower than rte_pktmbuf_free in a loop? (DPDK 21.11) >> >> I ran a bunch of tests on a 50GbE link where I'm getting packet drops >> (running with too few RX cores on purpose, to make some performance >> verification) and when the time comes to release the packets, i did a >> quick change like this: >> >> . >> //rte_pktmbuf_free_bulk( data, pkt_cnt ); >> for( idx = 0 ; idx < pkt_cnt ; ++idx ) { >> rte_pktmbuf_free( data[ idx ] ); >> } >> . >> >> And suddenly I'm dropping around 10% less packets (The traffic rate is >> around ~95Mpps). In case that's relevant, RX from the nic is done on a >> separate core than where the pkts are released (processed and released) >> >> I did also the following experiment: Found the MPPs speed value where i >> get around 2-5% drops using rte_pktmbuf_free_bulk, executed a bunch of >> readings where I consistently get drops.. Then switched to the loop with >> rte_pktmbuf_free and executed the same tests again, of a sudden I can't >> drop anymore. >> >> Isn't this strange? I was sure rte_pktmbuf_free_bulk would be kind of >> optimized for bulk releases so people don't have to loop themselves. >> >> Thanks >> > > Is your mbuf pool close to exhausted? How big is your bulk size? > It might be with that with larger bulk sizes, the loop is giving packets > back that instantly get consumed by incoming packets. So either pool is almost > empty or the non-bulk is keeping packets in cache more. >
Well, yes, once it starts dropping the buffer is full, but for quite a while before that event the mem usage is pretty low. In fact, I've added few diagnostics and here they are for the rte_pktmbuf_free_bulk test: . Mem usage: 0.244141%, captured 0 pkts Mem usage: 0.244141%, captured 0 pkts Mem usage: 0.241852%, captured 11,681,034 pkts Mem usage: 0.243807%, captured 44,327,015 pkts Mem usage: 0.243855%, captured 78,834,947 pkts Mem usage: 0.243235%, captured 113,343,787 pkts Mem usage: 0.246191%, captured 147,867,507 pkts Mem usage: 0.264502%, captured 182,367,926 pkts Mem usage: 0.244856%, captured 216,917,982 pkts Mem usage: 0.248837%, captured 251,445,720 pkts Mem usage: 0.257087%, captured 285,985,575 pkts Mem usage: 0.338078%, captured 320,509,279 pkts Mem usage: 0.362778%, captured 355,016,693 pkts Mem usage: 0.692415%, captured 389,521,441 pkts Mem usage: 52.050495%, captured 424,066,179 pkts Mem usage: 99.960041%, captured 456,936,573 pkts // DROPPING FROM HERE Mem usage: 99.962330%, captured 485,568,660 pkts Mem usage: 0.241208%, captured 491,178,294 pkts Mem usage: 0.241208%, captured 491,178,294 pkts . The % value is for the pool usage, it's a 8M items pool. As you can see all of a sudden it sharply gets exhausted and it never recover (the test stops at 500M packets.). Please note the prints have a 500ms interval. Attempting the same test with 1 billion packets lead to a similar result, the pool is exhausted after a while and there are plenty of drops: . Mem usage: 0.244141%, captured 0 pkts Mem usage: 0.244141%, captured 0 pkts Mem usage: 0.242686%, captured 1,994,944 pkts Mem usage: 0.243521%, captured 23,094,546 pkts Mem usage: 0.350094%, captured 57,594,139 pkts Mem usage: 0.245333%, captured 92,103,632 pkts Mem usage: 0.243330%, captured 126,616,534 pkts Mem usage: 0.244308%, captured 161,136,760 pkts Mem usage: 0.244093%, captured 195,633,863 pkts Mem usage: 0.245523%, captured 230,149,916 pkts Mem usage: 0.249910%, captured 264,648,839 pkts Mem usage: 0.258422%, captured 299,165,901 pkts Mem usage: 0.301266%, captured 333,678,228 pkts Mem usage: 0.425720%, captured 368,197,372 pkts Mem usage: 0.542426%, captured 402,699,822 pkts Mem usage: 21.447337%, captured 437,244,879 pkts Mem usage: 86.296201%, captured 471,804,014 pkts Mem usage: 99.958158%, captured 501,730,958 pkts // DROPPING FROM HERE Mem usage: 99.954629%, captured 529,462,253 pkts Mem usage: 99.958587%, captured 556,391,644 pkts Mem usage: 99.932027%, captured 582,999,427 pkts Mem usage: 99.959493%, captured 609,456,194 pkts Mem usage: 99.959779%, captured 635,641,696 pkts Mem usage: 99.958920%, captured 661,792,475 pkts Mem usage: 99.954844%, captured 687,919,194 pkts Mem usage: 99.957728%, captured 713,992,293 pkts Mem usage: 99.960685%, captured 740,042,732 pkts Mem usage: 99.956965%, captured 766,240,304 pkts Mem usage: 99.960780%, captured 792,423,477 pkts Mem usage: 99.960351%, captured 818,629,881 pkts Mem usage: 99.959016%, captured 844,904,955 pkts Mem usage: 99.960637%, captured 871,162,327 pkts Mem usage: 0.241995%, captured 878,826,100 pkts Mem usage: 0.241995%, captured 878,826,100 pkts . I can fix the issue switching from rte_pktmbuf_free_bulk to rte_pktmbuf_free in a loop (Not dropping at all, no matter how many packets I capture).. I would like to understand this issue better, also I'm a little confused on why after a while the performance degrade, you're suggesting there's some packet cache going on, can you elaborate on that? Thanks -- BR, Filip +48 666 369 823
