Re: [gentoo-user] Do I need NUMA set up in my kernel?
On 24/9/21 7:25 pm, William Kenworthy wrote: > On 24/9/21 5:38 pm, Michael wrote: >> On Friday, 24 September 2021 10:06:49 BST Peter Humphrey wrote: >>> On Thursday, 23 September 2021 19:20:52 BST Michael wrote: Out of interest, have you tried booting a NUMA enabled kernel to see what dmesg reports? >>> Yes, it's been enabled ever since I had a dual-socket motherboard, years >>> ago. I didn't understand why I did or didn't need it until I read Miles's >>> post yesterday (thanks, Miles). I don't know why it hadn't been made clear >>> in any websites I've visited. >>> On an old laptop, which definitely has only a single AMD APU, I get: $ dmesg | grep -i NUMA -A2 [0.002078] No NUMA configuration found [0.002080] Faking a node at [mem >>> 0x-0x00042eff] >>> [0.002085] NODE_DATA(0) allocated [mem 0x42effc000-0x42eff] >>> I had something similar. Oddly, with NUMA configured I get "not found" and >>> without it I get "pci_bus :00: on NUMA node 0". The system seems to run >>> happily either way. >> Sorry I should have made it clear - the above "No NUMA configuration found" >> message was obtained *with* NUMA enabled in my kernel. >> >> I suppose "NUMA on node 0" is the default first socket, which the kernel >> sets >> up. If the kernel can't find a second CPU it will be 'faking' a multi-CPU >> memory allocation setup, when it comes to allocate memory to the only CPU >> available. If the kernel does not have NUMA enabled then it doesn't need to >> fake anything. It will treat the hardware as a single socket MoBo and no >> further tests would be undertaken. All suppositions of course, I haven't >> looked at the code. ;-) > Try "numactl --hardware" (from the numactl package) > > rattus ~ # numactl --hardware > available: 1 nodes (0) > node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 > node 0 size: 31942 MB > node 0 free: 7210 MB > node distances: > node 0 > 0: 10 > rattus ~ # > > (Intel 6 core - NUMA emulation in the kernel.) > > I can only find testing NUMA code and hardware as a reason to have > emulation enabled on a non-NUMA system? > > BillK > > Actually Iam using "numactl -C 4,5 /etc/init.d/amavisd start" to lock processes to particular cpu's (on an arm big.LITTLE architecture. I will need to compile a new kernel without NUMA emulation to see if it still works. BillK
Re: [gentoo-user] Do I need NUMA set up in my kernel?
On 24/9/21 5:38 pm, Michael wrote: > On Friday, 24 September 2021 10:06:49 BST Peter Humphrey wrote: >> On Thursday, 23 September 2021 19:20:52 BST Michael wrote: >>> Out of interest, have you tried booting a NUMA enabled kernel to see what >>> dmesg reports? >> Yes, it's been enabled ever since I had a dual-socket motherboard, years >> ago. I didn't understand why I did or didn't need it until I read Miles's >> post yesterday (thanks, Miles). I don't know why it hadn't been made clear >> in any websites I've visited. >> >>> On an old laptop, which definitely has only a single AMD >>> APU, I get: >>> >>> $ dmesg | grep -i NUMA -A2 >>> [0.002078] No NUMA configuration found >>> [0.002080] Faking a node at [mem >> 0x-0x00042eff] >> >>> [0.002085] NODE_DATA(0) allocated [mem 0x42effc000-0x42eff] >> I had something similar. Oddly, with NUMA configured I get "not found" and >> without it I get "pci_bus :00: on NUMA node 0". The system seems to run >> happily either way. > Sorry I should have made it clear - the above "No NUMA configuration found" > message was obtained *with* NUMA enabled in my kernel. > > I suppose "NUMA on node 0" is the default first socket, which the kernel sets > up. If the kernel can't find a second CPU it will be 'faking' a multi-CPU > memory allocation setup, when it comes to allocate memory to the only CPU > available. If the kernel does not have NUMA enabled then it doesn't need to > fake anything. It will treat the hardware as a single socket MoBo and no > further tests would be undertaken. All suppositions of course, I haven't > looked at the code. ;-) Try "numactl --hardware" (from the numactl package) rattus ~ # numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 node 0 size: 31942 MB node 0 free: 7210 MB node distances: node 0 0: 10 rattus ~ # (Intel 6 core - NUMA emulation in the kernel.) I can only find testing NUMA code and hardware as a reason to have emulation enabled on a non-NUMA system? BillK
Re: [gentoo-user] Do I need NUMA set up in my kernel?
On Friday, 24 September 2021 10:06:49 BST Peter Humphrey wrote: > On Thursday, 23 September 2021 19:20:52 BST Michael wrote: > > Out of interest, have you tried booting a NUMA enabled kernel to see what > > dmesg reports? > > Yes, it's been enabled ever since I had a dual-socket motherboard, years > ago. I didn't understand why I did or didn't need it until I read Miles's > post yesterday (thanks, Miles). I don't know why it hadn't been made clear > in any websites I've visited. > > > On an old laptop, which definitely has only a single AMD > > APU, I get: > > > > $ dmesg | grep -i NUMA -A2 > > [0.002078] No NUMA configuration found > > [0.002080] Faking a node at [mem > > 0x-0x00042eff] > > > [0.002085] NODE_DATA(0) allocated [mem 0x42effc000-0x42eff] > > I had something similar. Oddly, with NUMA configured I get "not found" and > without it I get "pci_bus :00: on NUMA node 0". The system seems to run > happily either way. Sorry I should have made it clear - the above "No NUMA configuration found" message was obtained *with* NUMA enabled in my kernel. I suppose "NUMA on node 0" is the default first socket, which the kernel sets up. If the kernel can't find a second CPU it will be 'faking' a multi-CPU memory allocation setup, when it comes to allocate memory to the only CPU available. If the kernel does not have NUMA enabled then it doesn't need to fake anything. It will treat the hardware as a single socket MoBo and no further tests would be undertaken. All suppositions of course, I haven't looked at the code. ;-) signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Do I need NUMA set up in my kernel?
On Thursday, 23 September 2021 19:20:52 BST Michael wrote: > Out of interest, have you tried booting a NUMA enabled kernel to see what > dmesg reports? Yes, it's been enabled ever since I had a dual-socket motherboard, years ago. I didn't understand why I did or didn't need it until I read Miles's post yesterday (thanks, Miles). I don't know why it hadn't been made clear in any websites I've visited. > On an old laptop, which definitely has only a single AMD > APU, I get: > > $ dmesg | grep -i NUMA -A2 > [0.002078] No NUMA configuration found > [0.002080] Faking a node at [mem 0x-0x00042eff] > [0.002085] NODE_DATA(0) allocated [mem 0x42effc000-0x42eff] I had something similar. Oddly, with NUMA configured I get "not found" and without it I get "pci_bus :00: on NUMA node 0". The system seems to run happily either way. -- Regards, Peter.
Re: [gentoo-user] Do I need NUMA set up in my kernel?
On Thursday, 23 September 2021 17:38:15 BST Peter Humphrey wrote: > I've booted a kernel with no NUMA config, and it seems to run fine on this > single-socket Ryzen motherboard. I just get the one entry in dmesg: > > $ dmesg | grep -i numa > [0.297998] pci_bus :00: on NUMA node 0 > > That's it. I'm running five BOINC projects, some of which run on vbox, so on > this motherboard it seems clear that I don't need NUMA. Out of interest, have you tried booting a NUMA enabled kernel to see what dmesg reports? On an old laptop, which definitely has only a single AMD APU, I get: $ dmesg | grep -i NUMA -A2 [0.002078] No NUMA configuration found [0.002080] Faking a node at [mem 0x-0x00042eff] [0.002085] NODE_DATA(0) allocated [mem 0x42effc000-0x42eff] signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Do I need NUMA set up in my kernel?
On Thursday, 23 September 2021 17:32:46 BST Charlotte Delenk wrote: > On 9/23/21 18:30, Grant Taylor wrote: > > > On 9/23/21 4:39 AM, Miles Malone wrote: > > > >> You'd need NUMA if you had a NUMA machine. In current context, that > >> would be either a) a dual socket system, b) an amd threadripper, or > >> c) some of the really high core xeons. If your motherboard doesnt > >> have certain memory banks allocated to certain processors or cores, > >> you're probably not running a NUMA machine. > > > > > > > > Will a kernel without NUMA support boot and run on a system that has a > > NUMA architecture? > > > > > > > > If it will boot and run, does it simply do so in a sub-optimal way? > > Pure speculation for this one but if it works it's probably only going > to detect part of the memory and some of the devices and one cpu, since > the other resources are physically connected to the other cpus. > > > > > > > Flipping the coin on the other side, is there any negative effect > > (other than kernel size / lines of code / attack surface) for having > > NUMA support enabled on a non-NUMA system? > > There is no meaningful downside to leaving it enabled, it's enabled in > many distribution kernels for a reason. I've booted a kernel with no NUMA config, and it seems to run fine on this single-socket Ryzen motherboard. I just get the one entry in dmesg: $ dmesg | grep -i numa [0.297998] pci_bus :00: on NUMA node 0 That's it. I'm running five BOINC projects, some of which run on vbox, so on this motherboard it seems clear that I don't need NUMA. -- Regards, Peter.
Re: [gentoo-user] Do I need NUMA set up in my kernel?
On 9/23/21 18:30, Grant Taylor wrote: On 9/23/21 4:39 AM, Miles Malone wrote: You'd need NUMA if you had a NUMA machine. In current context, that would be either a) a dual socket system, b) an amd threadripper, or c) some of the really high core xeons. If your motherboard doesnt have certain memory banks allocated to certain processors or cores, you're probably not running a NUMA machine. Will a kernel without NUMA support boot and run on a system that has a NUMA architecture? If it will boot and run, does it simply do so in a sub-optimal way? Pure speculation for this one but if it works it's probably only going to detect part of the memory and some of the devices and one cpu, since the other resources are physically connected to the other cpus. Flipping the coin on the other side, is there any negative effect (other than kernel size / lines of code / attack surface) for having NUMA support enabled on a non-NUMA system? There is no meaningful downside to leaving it enabled, it's enabled in many distribution kernels for a reason. OpenPGP_0x3CEF5DDA915AECB0.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-user] Do I need NUMA set up in my kernel?
On 9/23/21 4:39 AM, Miles Malone wrote: You'd need NUMA if you had a NUMA machine. In current context, that would be either a) a dual socket system, b) an amd threadripper, or c) some of the really high core xeons. If your motherboard doesnt have certain memory banks allocated to certain processors or cores, you're probably not running a NUMA machine. Will a kernel without NUMA support boot and run on a system that has a NUMA architecture? If it will boot and run, does it simply do so in a sub-optimal way? Flipping the coin on the other side, is there any negative effect (other than kernel size / lines of code / attack surface) for having NUMA support enabled on a non-NUMA system? -- Grant. . . . unix || die
Re: [gentoo-user] Do I need NUMA set up in my kernel?
On Thursday, 23 September 2021 11:39:39 BST Miles Malone wrote: > You'd need NUMA if you had a NUMA machine. In current context, that > would be either a) a dual socket system, b) an amd threadripper, or c) > some of the really high core xeons. If your motherboard doesnt have > certain memory banks allocated to certain processors or cores, you're > probably not running a NUMA machine. > > NUMA stands for non-uniform memory access, it means that certain > processor cores have more direct access to certain parts of memory > than others do (e.g. to access the other memory they need the other > cpu core to pass it through) > > On Thu, 23 Sept 2021 at 19:39, Charlotte Delenk wrote: > > Hi Peter, > > > > On 9/23/21 10:59, Peter Humphrey wrote: > > > Hello list, > > > > > > I see "[0.003162] No NUMA configuration found" in dmesg. Does that > > > mean I should, or can, remove the NUMA settings from the kernel? This > > > is a Ryzen M9 5900X machine. > > > > I have CONFIG_NUMA unset on both of my AMD Ryzen machines (Zen+ and > > Zen2) with no issues Thank you both. I'll try removing it and see what happens. -- Regards, Peter.
Re: [gentoo-user] Do I need NUMA set up in my kernel?
You'd need NUMA if you had a NUMA machine. In current context, that would be either a) a dual socket system, b) an amd threadripper, or c) some of the really high core xeons. If your motherboard doesnt have certain memory banks allocated to certain processors or cores, you're probably not running a NUMA machine. NUMA stands for non-uniform memory access, it means that certain processor cores have more direct access to certain parts of memory than others do (e.g. to access the other memory they need the other cpu core to pass it through) On Thu, 23 Sept 2021 at 19:39, Charlotte Delenk wrote: > > Hi Peter, > > On 9/23/21 10:59, Peter Humphrey wrote: > > Hello list, > > > > I see "[0.003162] No NUMA configuration found" in dmesg. Does that mean > > I > > should, or can, remove the NUMA settings from the kernel? This is a Ryzen M9 > > 5900X machine. > > I have CONFIG_NUMA unset on both of my AMD Ryzen machines (Zen+ and > Zen2) with no issues > >
Re: [gentoo-user] Do I need NUMA set up in my kernel?
Hi Peter, On 9/23/21 10:59, Peter Humphrey wrote: Hello list, I see "[0.003162] No NUMA configuration found" in dmesg. Does that mean I should, or can, remove the NUMA settings from the kernel? This is a Ryzen M9 5900X machine. I have CONFIG_NUMA unset on both of my AMD Ryzen machines (Zen+ and Zen2) with no issues OpenPGP_0x3CEF5DDA915AECB0.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
[gentoo-user] Do I need NUMA set up in my kernel?
Hello list, I see "[0.003162] No NUMA configuration found" in dmesg. Does that mean I should, or can, remove the NUMA settings from the kernel? This is a Ryzen M9 5900X machine. I have this at the moment: $ grep NUMA /usr/src/linux/.config CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y # CONFIG_NUMA_BALANCING is not set CONFIG_NUMA=y CONFIG_AMD_NUMA=y CONFIG_X86_64_ACPI_NUMA=y # CONFIG_NUMA_EMU is not set CONFIG_USE_PERCPU_NUMA_NODE_ID=y CONFIG_ACPI_NUMA=y CONFIG_NUMA_KEEP_MEMINFO=y -- Regards, Peter.
