Current code assumes that all numa nodes have memory present, which is reasonable assumption for any production use. It is on my todo list to improve handling of such cases, but it is not 5-minute task, so I cannot promise anything soon. Simply changing from "online" to "has_memory" will not help, as still crash will happen 1st time buffer is allocated on numa1 thread...
> On 28 Feb 2019, at 06:31, Lijian Zhang <lijian.zh...@arm.com> wrote: > > Hi Maintainers, > I have two supermicro x86 servers with unusual numa configs. The first server > has two physical CPUs, but only one numa node. The other server has two > physical CPUs, two numa nodes, but only numa node 0 has attached memories. > The detailed CPU info are dumped as below. > > VPP crashed on both server with below call-stack. VPP gets numa nodes ‘0-1’ > from ‘/sys/devices/system/node/possible’, and initialize buffers for all of > numa nodes, while numa node 1 has no memories, so below function failed with > numa node. > > If ‘/sys/devices/system/node/has_memories’ is used to detect the numa nodes, > on which VPP initialize buffers, there will be no such crash issue. > > Could you please evaluate this issue, and evaluate if this is a reasonable > change requirement, or should we just skip the issue because of the abnormal > numa configs in my servers? > > #0 vlib_physmem_shared_map_create (vm=0x7ffff6853240 <vlib_global_main>, > name=0x7fffb786621c "buffers-numa-0", size=1073741824, log2_page_sz=30, > numa_node=0, map_index=0x7fffb51ffd84) at > /home/lijian/tasks/origin/src/vlib/physmem.c:45 > #1 0x00007ffff657b279 in vlib_buffer_main_init_numa_node (vm=0x7ffff6853240 > <vlib_global_main>, numa_node=0) at > /home/lijian/tasks/origin/src/vlib/buffer.c:672 > #2 0x00007ffff657b8f9 in vlib_buffer_main_init (vm=0x7ffff6853240 > <vlib_global_main>) at /home/lijian/tasks/origin/src/vlib/buffer.c:801 > #3 0x00007ffff65c7bb2 in vlib_main (vm=0x7ffff6853240 <vlib_global_main>, > input=0x7fffb51fffb0) at /home/lijian/tasks/origin/src/vlib/main.c:1988 > #4 0x00007ffff662570a in thread0 (arg=140737329312320) at > /home/lijian/tasks/origin/src/vlib/unix/main.c:606 > #5 0x00007ffff6079544 in clib_calljmp () from > /home/lijian/tasks/origin/build-root/build-vpp_debug-native/vpp/lib/libvppinfra.so.19.04 > #6 0x00007fffffffd2b0 in ?? () > #7 0x00007ffff6625bd7 in vlib_unix_main (argc=36, argv=0x55555586f3b0) at > /home/lijian/tasks/origin/src/vlib/unix/main.c:675 > #8 0x000055555555bedc in main (argc=36, argv=0x55555586f3b0) at > /home/lijian/tasks/origin/src/vpp/vnet/main.c:274 > > > lijian@net-x86-supermicro-01:~/tasks/origin$ sudo numactl -H > available: 1 nodes (0) > node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 > node 0 size: 64383 MB > node 0 free: 33048 MB > node distances: > node 0 > 0: 10 > lijian@net-x86-supermicro-01:~/tasks/origin$ lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 20 > On-line CPU(s) list: 0-19 > Thread(s) per core: 1 > Core(s) per socket: 10 > Socket(s): 2 > NUMA node(s): 1 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 63 > Model name: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz > Stepping: 2 > CPU MHz: 2300.179 > CPU max MHz: 3000.0000 > CPU min MHz: 1200.0000 > BogoMIPS: 4601.05 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 25600K > NUMA node0 CPU(s): 0-19 > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx > pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology > nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 > ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt > tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single > kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 > smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts > > > lijian@net-x86-supermicro-02:~/tasks/origin$ sudo numactl -H > available: 2 nodes (0-1) > node 0 cpus: 0 1 2 3 4 5 6 7 8 9 > node 0 size: 64311 MB > node 0 free: 30865 MB > node 1 cpus: 10 11 12 13 14 15 16 17 18 19 > node 1 size: 0 MB > node 1 free: 0 MB > node distances: > node 0 1 > 0: 10 20 > 1: 20 10 > lijian@net-x86-supermicro-02:~/tasks/origin$ lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 20 > On-line CPU(s) list: 0-19 > Thread(s) per core: 1 > Core(s) per socket: 10 > Socket(s): 2 > NUMA node(s): 2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 63 > Model name: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz > Stepping: 2 > CPU MHz: 1200.176 > CPU max MHz: 2300.0000 > CPU min MHz: 1200.0000 > BogoMIPS: 4599.71 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 25600K > NUMA node0 CPU(s): 0-9 > NUMA node1 CPU(s): 10-19 > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx > pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology > nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est > tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt > tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb > invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi > flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid > cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts flush_l1d > lijian@net-x86-supermicro-02:~/tasks/origin$ > > Thanks. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#12381): https://lists.fd.io/g/vpp-dev/message/12381 > <https://lists.fd.io/g/vpp-dev/message/12381> > Mute This Topic: https://lists.fd.io/mt/30160902/675642 > <https://lists.fd.io/mt/30160902/675642> > Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [dmar...@me.com > <mailto:dmar...@me.com>] > -=-=-=-=-=-=-=-=-=-=-=- -- Damjan
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12387): https://lists.fd.io/g/vpp-dev/message/12387 Mute This Topic: https://lists.fd.io/mt/30160902/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
