Hi there,I'm trying to figure out a performance issue I'm having with with CX4/5 NIC's and low throughput on R630's and R640's when I ran into something that I found a bit odd.
The NIC is plugged into numa node 0, but the buffers below show that they're being allocated from numa node 1?
# vppctl show buffersPool Name Index NUMA Size Data Size Total Avail Cached Used
default-numa-0 0 0 2496 2048 430184 430184 0 0*default-numa-1 1 1 2496 2048 430184 411592 124 18468
* # vppctl show memory main-heap Thread 0 vpp_main base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 1 vpp_wk_0 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 2 vpp_wk_1 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 3 vpp_wk_2 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 4 vpp_wk_3 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 5 vpp_wk_4 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 6 vpp_wk_5 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 7 vpp_wk_6 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 8 vpp_wk_7 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 9 vpp_wk_8 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 10 vpp_wk_9 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 11 vpp_wk_10 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 12 vpp_wk_11 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 13 vpp_wk_12 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 14 vpp_wk_13 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 15 vpp_wk_14 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 16 vpp_wk_15 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 17 vpp_wk_16 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 18 vpp_wk_17 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G Thread 19 vpp_wk_18 base 0x7831c0000000, size 4g, locked, unmap-on-destroy, name 'main heap' page stats: page-size 1G, total 4, mapped 4, not-mapped 0 numa 0: 4 pages, 4g bytes total: 4.00G, used: 227.76M, free: 3.78G, trimmable: 3.77G # vppctl show threadID Name Type LWP Sched Policy (Priority) lcore Core Socket State 0 vpp_main 8440 other (0) 0 0 0 1 vpp_wk_0 workers 8444 other (0) 2 4 0 2 vpp_wk_1 workers 8445 other (0) 4 1 0 3 vpp_wk_2 workers 8446 other (0) 6 3 0 4 vpp_wk_3 workers 8447 other (0) 8 2 0 5 vpp_wk_4 workers 8448 other (0) 10 12 0 6 vpp_wk_5 workers 8449 other (0) 12 8 0 7 vpp_wk_6 workers 8450 other (0) 14 11 0 8 vpp_wk_7 workers 8451 other (0) 16 9 0 9 vpp_wk_8 workers 8452 other (0) 18 10 0 10 vpp_wk_9 workers 8453 other (0) 20 16 0 11 vpp_wk_10 workers 8454 other (0) 22 20 0 12 vpp_wk_11 workers 8455 other (0) 24 17 0 13 vpp_wk_12 workers 8456 other (0) 26 19 0 14 vpp_wk_13 workers 8457 other (0) 28 18 0 15 vpp_wk_14 workers 8458 other (0) 30 28 0 16 vpp_wk_15 workers 8459 other (0) 32 24 0 17 vpp_wk_16 workers 8460 other (0) 34 27 0 18 vpp_wk_17 workers 8461 other (0) 36 25 0 19 vpp_wk_18 workers 8462 other (0) 38 26 0
# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-39 Off-line CPU(s) list: 40-79 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 6122 CPU @ 1.80GHz CPU family: 6 Model: 85 Thread(s) per core: 1 Core(s) per socket: 20 Socket(s): 2 Stepping: 4 CPU(s) scaling MHz: 99% CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 3601.00Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_t sc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq r dseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts vnmi pku ospke md_clear flush_l1d arch_capabilities
Virtualization features: Virtualization: VT-x Caches (sum of all): L1d: 1.3 MiB (40 instances) L1i: 1.3 MiB (40 instances) L2: 40 MiB (40 instances) L3: 55 MiB (2 instances) NUMA: NUMA node(s): 2NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
Vulnerabilities: Gather data sampling: Vulnerable Indirect target selection: Not affected Itlb multihit: KVM: VulnerableL1tf: Mitigation; PTE Inversion; VMX vulnerable, SMT disabled
Mds: Vulnerable; SMT disabled Meltdown: Vulnerable Mmio stale data: Vulnerable Reg file data sampling: Not affected Retbleed: Vulnerable Spec rstack overflow: Not affected Spec store bypass: VulnerableSpectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Not affected; BHI: Not affected
Srbds: Not affected Tsa: Not affected Tsx async abort: Vulnerable I was wondering if this is correct, or if I missed something? Regards Nigel
OpenPGP_signature.asc
Description: OpenPGP digital signature
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#26234): https://lists.fd.io/g/vpp-dev/message/26234 Mute This Topic: https://lists.fd.io/mt/114480912/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
