Hi Nigel, Are you sure it's connected to cpu 0? Do you mind to share "show pci" and "cat /sys/bus/pci/devices/0000\:0c\:00.0/numa_node" (fix to your address)
On Fri, 1 Aug 2025 at 10:04, Nigel Kukard via lists.fd.io <nkukard= lbsd....@lists.fd.io> wrote: > Hi there, > > I'm trying to figure out a performance issue I'm having with with CX4/5 > NIC's and low throughput on R630's and R640's when I ran into something > that I found a bit odd. > > The NIC is plugged into numa node 0, but the buffers below show that > they're being allocated from numa node 1? > > # vppctl show buffers > Pool Name Index NUMA Size Data Size Total Avail Cached > Used > default-numa-0 0 0 2496 2048 430184 430184 0 > 0 > > *default-numa-1 1 1 2496 2048 430184 411592 124 > 18468 * > > # vppctl show memory main-heap > Thread 0 vpp_main > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 1 vpp_wk_0 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 2 vpp_wk_1 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 3 vpp_wk_2 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 4 vpp_wk_3 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 5 vpp_wk_4 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 6 vpp_wk_5 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 7 vpp_wk_6 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 8 vpp_wk_7 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 9 vpp_wk_8 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 10 vpp_wk_9 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 11 vpp_wk_10 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 12 vpp_wk_11 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 13 vpp_wk_12 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 14 vpp_wk_13 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 15 vpp_wk_14 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 16 vpp_wk_15 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 17 vpp_wk_16 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 18 vpp_wk_17 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > Thread 19 vpp_wk_18 > base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 1G, total 4, mapped 4, not-mapped 0 > numa 0: 4 pages, 4g bytes > total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G > > # vppctl show thread > ID Name Type LWP Sched Policy (Priority) > lcore Core Socket State > 0 vpp_main 8440 other (0) 0 > 0 0 > 1 vpp_wk_0 workers 8444 other (0) 2 > 4 0 > 2 vpp_wk_1 workers 8445 other (0) 4 > 1 0 > 3 vpp_wk_2 workers 8446 other (0) 6 > 3 0 > 4 vpp_wk_3 workers 8447 other (0) 8 > 2 0 > 5 vpp_wk_4 workers 8448 other (0) 10 > 12 0 > 6 vpp_wk_5 workers 8449 other (0) 12 > 8 0 > 7 vpp_wk_6 workers 8450 other (0) 14 > 11 0 > 8 vpp_wk_7 workers 8451 other (0) 16 > 9 0 > 9 vpp_wk_8 workers 8452 other (0) 18 > 10 0 > 10 vpp_wk_9 workers 8453 other (0) 20 > 16 0 > 11 vpp_wk_10 workers 8454 other (0) 22 > 20 0 > 12 vpp_wk_11 workers 8455 other (0) 24 > 17 0 > 13 vpp_wk_12 workers 8456 other (0) 26 > 19 0 > 14 vpp_wk_13 workers 8457 other (0) 28 > 18 0 > 15 vpp_wk_14 workers 8458 other (0) 30 > 28 0 > 16 vpp_wk_15 workers 8459 other (0) 32 > 24 0 > 17 vpp_wk_16 workers 8460 other (0) 34 > 27 0 > 18 vpp_wk_17 workers 8461 other (0) 36 > 25 0 > 19 vpp_wk_18 workers 8462 other (0) 38 > 26 0 > > # lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Address sizes: 46 bits physical, 48 bits virtual > Byte Order: Little Endian > CPU(s): 80 > On-line CPU(s) list: 0-39 > Off-line CPU(s) list: 40-79 > Vendor ID: GenuineIntel > Model name: Intel(R) Xeon(R) Gold 6122 CPU @ 1.80GHz > CPU family: 6 > Model: 85 > Thread(s) per core: 1 > Core(s) per socket: 20 > Socket(s): 2 > Stepping: 4 > CPU(s) scaling MHz: 99% > CPU max MHz: 3700.0000 > CPU min MHz: 1000.0000 > BogoMIPS: 3601.00 > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep > mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe > syscall nx pdpe1gb rdtscp lm constant_t > sc art arch_perfmon pebs bts rep_good nopl > xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl > vmx smx est tm2 ssse3 sdbg fma cx16 xtpr > pdcm pcid dca sse4_1 sse4_2 x2apic movbe > popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm > 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin > ssbd mba ibrs ibpb stibp tpr_shadow > flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 > erms invpcid rtm cqm mpx rdt_a avx512f avx512dq r > dseed adx smap clflushopt clwb intel_pt > avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc > cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida > arat pln pts vnmi pku ospke md_clear flush_l1d > arch_capabilities > Virtualization features: > Virtualization: VT-x > Caches (sum of all): > L1d: 1.3 MiB (40 instances) > L1i: 1.3 MiB (40 instances) > L2: 40 MiB (40 instances) > L3: 55 MiB (2 instances) > NUMA: > NUMA node(s): 2 > NUMA node0 CPU(s): > 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 > NUMA node1 CPU(s): > 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 > Vulnerabilities: > Gather data sampling: Vulnerable > Indirect target selection: Not affected > Itlb multihit: KVM: Vulnerable > L1tf: Mitigation; PTE Inversion; VMX vulnerable, SMT > disabled > Mds: Vulnerable; SMT disabled > Meltdown: Vulnerable > Mmio stale data: Vulnerable > Reg file data sampling: Not affected > Retbleed: Vulnerable > Spec rstack overflow: Not affected > Spec store bypass: Vulnerable > Spectre v1: Vulnerable: __user pointer sanitization and > usercopy barriers only; no swapgs barriers > Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; > PBRSB-eIBRS: Not affected; BHI: Not affected > Srbds: Not affected > Tsa: Not affected > Tsx async abort: Vulnerable > > > I was wondering if this is correct, or if I missed something? > > Regards > Nigel > > > > > > > > -- Best regards Stanislav Zaikin
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#26235): https://lists.fd.io/g/vpp-dev/message/26235 Mute This Topic: https://lists.fd.io/mt/114480912/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
