Hi Jesper, Thanks for the comments. >> I assume this xdpsock code is small and should all fit into the icache. >> However, doing another perf stat on xdpsock l2fwd shows >> >> 13,720,109,581 stalled-cycles-frontend # 60.01% frontend cycles >> idle (23.82%) >> >> <not supported> stalled-cycles-backend >> 7,994,837 branch-misses # 0.16% of all branches >> (23.80%) >> 996,874,424 bus-cycles # 99.679 M/sec (23.80%) >> 18,942,220,445 ref-cycles # 1894.067 M/sec (28.56%) >> 100,983,226 LLC-loads # 10.097 M/sec (23.80%) >> 4,897,089 LLC-load-misses # 4.85% of all LL-cache hits >> (23.80%) >> 66,659,889 LLC-stores # 6.665 M/sec (9.52%) >> 8,373 LLC-store-misses # 0.837 K/sec (9.52%) >> 158,178,410 LLC-prefetches # 15.817 M/sec (9.52%) >> 3,011,180 LLC-prefetch-misses # 0.301 M/sec (9.52%) >> 8,190,383,109 dTLB-loads # 818.971 M/sec (9.52%) >> 20,432,204 dTLB-load-misses # 0.25% of all dTLB cache hits >> (9.52%) >> 3,729,504,674 dTLB-stores # 372.920 M/sec (9.52%) >> 992,231 dTLB-store-misses # 0.099 M/sec (9.52%) >> <not supported> dTLB-prefetches >> <not supported> dTLB-prefetch-misses >> 11,619 iTLB-loads # 0.001 M/sec (9.52%) >> 1,874,756 iTLB-load-misses # 16135.26% of all iTLB cache hits >> (14.28%) > > What was the sample period for this perf stat? > 10 seconds. root@ovs-smartnic:~/net-next/tools/perf# ./perf stat -C 6 sleep 10
>> I have super high iTLB-load-misses. This is probably the cause of high >> frontend stalled. > > It looks very strange that your iTLB-loads are 11,619, while the > iTLB-load-misses are much much higher 1,874,756. > Does it mean cpu try to load the code, then fail, then load again and fail again... So the number of iTLB loads is larger than misses. Maybe it's related to high nmi rate, where the nmi handler clear my iTLB? Let me try to remove the nmi interference first. >> Do you know any way to improve iTLB hit rate? > > The xdpsock code should be small enough to fit in the iCache, but it > might be layout in memory in an unfortunate way. You could play with > rearranging the C-code (look at the objdump alignments). > > If you want to know the details about code alignment issue, and how to > troubleshoot them, you should read this VERY excellent blog post by > Denis Bakhvalov: > https://dendibakh.github.io/blog/2018/01/18/Code_alignment_issues Thanks for the link. William
