Re: [slurm-users] slurmd -C showing incorrect core count
From what I know of how this works, no, it’s not getting it from a local file or the master node. I don’t believe it even makes a network connection, nor requires a slurm.conf in order to run. If you can run it fresh on a node with no config and that’s what it comes up with, it’s probably getting it from the VM somehow. -- || \\UTGERS, |---*O*--- ||_// the State | Ryan Novosielski - novos...@rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\of NJ | Office of Advanced Research Computing - MSB C630, Newark `' > On Mar 11, 2020, at 10:26 AM, mike tie wrote: > > > Yep, slurmd -C is obviously getting the data from somewhere, either a local > file or from the master node. hence my email to the group; I was hoping > that someone would just say: "yeah, modify file ". But oh well. I'll > start playing with strace and gdb later this week; looking through the > source might also be helpful. > > I'm not cloning existing virtual machines with slurm. I have access to a > vmware system that from time to time isn't running at full capacity; usage > is stable for blocks of a month or two at a time, so my thought/plan was to > spin up a slurm compute node on it, and resize it appropriately every few > months (why not put it to work). I started with 10 cores, and it looks like > I can up it to 16 cores for a while, and that's when I ran into the problem. > > -mike > > > > Michael Tie > Technical Director > Mathematics, Statistics, and Computer Science > > One North College Street phn: 507-222-4067 > Northfield, MN 55057 cel:952-212-8933 > m...@carleton.edufax:507-222-4312 > > > > On Wed, Mar 11, 2020 at 1:15 AM Kirill 'kkm' Katsnelson > wrote: > On Tue, Mar 10, 2020 at 1:41 PM mike tie wrote: > Here is the output of lstopo > > $ lstopo -p > Machine (63GB) > Package P#0 + L3 (16MB) > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#0 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#1 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#2 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#3 > Package P#1 + L3 (16MB) > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#4 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#5 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#6 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#7 > Package P#2 + L3 (16MB) > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#8 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#9 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#10 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#11 > Package P#3 + L3 (16MB) > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#12 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#13 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#14 > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#15 > > There is no sane way to derive the number 10 from this topology. obviously: > it has a prime factor of 5, but everything in the lstopo output is sized in > powers of 2 (4 packages, a.k.a. sockets, 4 single-threaded CPU cores per). > > I responded yesterday but somehow managed to plop my signature into the > middle of it, so maybe you have missed inline replies? > > It's very, very likely that the number is stored *somewhere*. First to > eliminate is the hypothesis that the number is acquired from the control > daemon. That's the simplest step and the largest landgrab in the > divide-and-conquer analysis plan. Then just look where it comes from on the > VM. strace(1) will reveal all files slurmd reads. > > You are not rolling out the VMs from an image, ain't you? I'm wondering why > do you need to tweak an existing VM that is already in a weird state. Is > simply setting its snapshot aside and creating a new one from an image > hard/impossible? I did not touch VMWare for more than 10 years, so I may be a > bit naive; in the platform I'm working now (GCE), create-use-drop pattern of > VM use is much more common and simpler than create and maintain it to either > *ad infinitum* or *ad nauseam*, whichever will have been reached the > earliest. But I don't know anything about VMWare; maybe it's not possible or > feasible with it. > > -kkm >
Re: [slurm-users] slurmd -C showing incorrect core count
On 3/12/20 9:37 PM, Kirill 'kkm' Katsnelson wrote: Aaah, that's a cool find! I never really looked inside my nodes for more than a year since I debugged all my stuff so it "just works". They are conjured out of nothing and dissolve back into nothing after 10 minutes of inactivity. But good to know! In the cloud, changing the amount of RAM and the number and even type of CPUs is all too easy. Also on some architectures doing that discovery can take time, so having it cached can be useful (slurmd will just read it once on startup). For us that's on a ramdisk filesystem (as Cray XC nodes have no local disk) so it vanishes every time the node reboots. My bet is that Mike's nodes have persistent storage and have an old copy of this file, hence the weird discrepancy he's seeing. All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
Re: [slurm-users] slurmd -C showing incorrect core count
On Wed, Mar 11, 2020 at 9:57 PM Chris Samuel wrote: > If so move it out of the way somewhere safe (just in case) and try again. > Aaah, that's a cool find! I never really looked inside my nodes for more than a year since I debugged all my stuff so it "just works". They are conjured out of nothing and dissolve back into nothing after 10 minutes of inactivity. But good to know! In the cloud, changing the amount of RAM and the number and even type of CPUs is all too easy. Mike, if I were you, I'd probably move out *all* files out of that directory. Who knows what other surprising mismatches due to the changed hardware it contains. I will eat my hat if there is anything of value there--I never had to prepopulate any directories under /var on a compute node. /etc/slurm/* is the only Slurm-related thing my cloud rig nodes pulls down from the common config storage upon boot--the only boot in their lifetime. That was a very educational problem-solving session, thank you both guys! -kkm
Re: [slurm-users] slurmd -C showing incorrect core count
On 10/3/20 1:40 pm, mike tie wrote:
Here is the output of lstopo
Hmm, well I believe Slurm should be using hwloc (which provides lstopo)
to get its information (at least it calls the xcpuinfo_hwloc_topo_get()
function for that), so if lstopo works then slurmd should too.
Ah, looking a bit deeper I see in src/slurmd/common/xcpuinfo.c:
if (!hwloc_xml_whole)
hwloc_xml_whole = xstrdup_printf("%s/hwloc_topo_whole.xml",
conf->spooldir);
Do you happen to have a file called "hwloc_topo_whole.xml" in your spool
directory on that node? I'm wondering if it's cached old config there.
If so move it out of the way somewhere safe (just in case) and try again.
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
Re: [slurm-users] slurmd -C showing incorrect core count
Yup, I think if you get stuck so badly, the first thing is to make sure the node does not get the number 10 from the controller, and the second just reimage the VM fresh. It maybe not the quickest way, but at least predictable in the sense of time spent. Good luck! -kkm On Wed, Mar 11, 2020 at 7:28 AM mike tie wrote: > > Yep, slurmd -C is obviously getting the data from somewhere, either a > local file or from the master node. hence my email to the group; I was > hoping that someone would just say: "yeah, modify file ". But oh > well. I'll start playing with strace and gdb later this week; looking > through the source might also be helpful. > > I'm not cloning existing virtual machines with slurm. I have access to a > vmware system that from time to time isn't running at full capacity; usage > is stable for blocks of a month or two at a time, so my thought/plan was to > spin up a slurm compute node on it, and resize it appropriately every few > months (why not put it to work). I started with 10 cores, and it looks > like I can up it to 16 cores for a while, and that's when I ran into the > problem. > > -mike > > > > *Michael Tie*Technical Director > Mathematics, Statistics, and Computer Science > > One North College Street phn: 507-222-4067 > Northfield, MN 55057 cel:952-212-8933 > m...@carleton.edufax:507-222-4312 > > > On Wed, Mar 11, 2020 at 1:15 AM Kirill 'kkm' Katsnelson > wrote: > >> On Tue, Mar 10, 2020 at 1:41 PM mike tie wrote: >> >>> Here is the output of lstopo >>> >> >>> *$* lstopo -p >>> >>> Machine (63GB) >>> >>> Package P#0 + L3 (16MB) >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#0 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#1 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#2 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#3 >>> >>> Package P#1 + L3 (16MB) >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#4 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#5 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#6 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#7 >>> >>> Package P#2 + L3 (16MB) >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#8 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#9 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#10 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#11 >>> >>> Package P#3 + L3 (16MB) >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#12 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#13 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#14 >>> >>> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#15 >>> >> >> There is no sane way to derive the number 10 from this topology. >> obviously: it has a prime factor of 5, but everything in the lstopo output >> is sized in powers of 2 (4 packages, a.k.a. sockets, 4 single-threaded CPU >> cores per). >> >> I responded yesterday but somehow managed to plop my signature into the >> middle of it, so maybe you have missed inline replies? >> >> It's very, very likely that the number is stored *somewhere*. First to >> eliminate is the hypothesis that the number is acquired from the control >> daemon. That's the simplest step and the largest landgrab in the >> divide-and-conquer analysis plan. Then just look where it comes from on the >> VM. strace(1) will reveal all files slurmd reads. >> >> You are not rolling out the VMs from an image, ain't you? I'm wondering >> why do you need to tweak an existing VM that is already in a weird state. >> Is simply setting its snapshot aside and creating a new one from an image >> hard/impossible? I did not touch VMWare for more than 10 years, so I may be >> a bit naive; in the platform I'm working now (GCE), create-use-drop pattern >> of VM use is much more common and simpler than create and maintain it to >> either *ad infinitum* or *ad nauseam*, whichever will have been reached the >> earliest. But I don't know anything about VMWare; maybe it's not possible >> or feasible with it. >> >> -kkm >> >>
Re: [slurm-users] slurmd -C showing incorrect core count
Yep, slurmd -C is obviously getting the data from somewhere, either a local file or from the master node. hence my email to the group; I was hoping that someone would just say: "yeah, modify file ". But oh well. I'll start playing with strace and gdb later this week; looking through the source might also be helpful. I'm not cloning existing virtual machines with slurm. I have access to a vmware system that from time to time isn't running at full capacity; usage is stable for blocks of a month or two at a time, so my thought/plan was to spin up a slurm compute node on it, and resize it appropriately every few months (why not put it to work). I started with 10 cores, and it looks like I can up it to 16 cores for a while, and that's when I ran into the problem. -mike *Michael Tie*Technical Director Mathematics, Statistics, and Computer Science One North College Street phn: 507-222-4067 Northfield, MN 55057 cel:952-212-8933 m...@carleton.edufax:507-222-4312 On Wed, Mar 11, 2020 at 1:15 AM Kirill 'kkm' Katsnelson wrote: > On Tue, Mar 10, 2020 at 1:41 PM mike tie wrote: > >> Here is the output of lstopo >> > >> *$* lstopo -p >> >> Machine (63GB) >> >> Package P#0 + L3 (16MB) >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#0 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#1 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#2 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#3 >> >> Package P#1 + L3 (16MB) >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#4 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#5 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#6 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#7 >> >> Package P#2 + L3 (16MB) >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#8 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#9 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#10 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#11 >> >> Package P#3 + L3 (16MB) >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#12 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#13 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#14 >> >> L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#15 >> > > There is no sane way to derive the number 10 from this topology. > obviously: it has a prime factor of 5, but everything in the lstopo output > is sized in powers of 2 (4 packages, a.k.a. sockets, 4 single-threaded CPU > cores per). > > I responded yesterday but somehow managed to plop my signature into the > middle of it, so maybe you have missed inline replies? > > It's very, very likely that the number is stored *somewhere*. First to > eliminate is the hypothesis that the number is acquired from the control > daemon. That's the simplest step and the largest landgrab in the > divide-and-conquer analysis plan. Then just look where it comes from on the > VM. strace(1) will reveal all files slurmd reads. > > You are not rolling out the VMs from an image, ain't you? I'm wondering > why do you need to tweak an existing VM that is already in a weird state. > Is simply setting its snapshot aside and creating a new one from an image > hard/impossible? I did not touch VMWare for more than 10 years, so I may be > a bit naive; in the platform I'm working now (GCE), create-use-drop pattern > of VM use is much more common and simpler than create and maintain it to > either *ad infinitum* or *ad nauseam*, whichever will have been reached the > earliest. But I don't know anything about VMWare; maybe it's not possible > or feasible with it. > > -kkm > >
Re: [slurm-users] slurmd -C showing incorrect core count
On Tue, Mar 10, 2020 at 1:41 PM mike tie wrote: > Here is the output of lstopo > > *$* lstopo -p > > Machine (63GB) > > Package P#0 + L3 (16MB) > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#0 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#1 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#2 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#3 > > Package P#1 + L3 (16MB) > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#4 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#5 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#6 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#7 > > Package P#2 + L3 (16MB) > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#8 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#9 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#10 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#11 > > Package P#3 + L3 (16MB) > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#12 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#13 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#14 > > L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#15 > There is no sane way to derive the number 10 from this topology. obviously: it has a prime factor of 5, but everything in the lstopo output is sized in powers of 2 (4 packages, a.k.a. sockets, 4 single-threaded CPU cores per). I responded yesterday but somehow managed to plop my signature into the middle of it, so maybe you have missed inline replies? It's very, very likely that the number is stored *somewhere*. First to eliminate is the hypothesis that the number is acquired from the control daemon. That's the simplest step and the largest landgrab in the divide-and-conquer analysis plan. Then just look where it comes from on the VM. strace(1) will reveal all files slurmd reads. You are not rolling out the VMs from an image, ain't you? I'm wondering why do you need to tweak an existing VM that is already in a weird state. Is simply setting its snapshot aside and creating a new one from an image hard/impossible? I did not touch VMWare for more than 10 years, so I may be a bit naive; in the platform I'm working now (GCE), create-use-drop pattern of VM use is much more common and simpler than create and maintain it to either *ad infinitum* or *ad nauseam*, whichever will have been reached the earliest. But I don't know anything about VMWare; maybe it's not possible or feasible with it. -kkm
Re: [slurm-users] slurmd -C showing incorrect core count
Here is the output of lstopo *$* lstopo -p Machine (63GB) Package P#0 + L3 (16MB) L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#0 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#1 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#2 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#3 Package P#1 + L3 (16MB) L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#4 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#5 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#6 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#7 Package P#2 + L3 (16MB) L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#8 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#9 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#10 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#11 Package P#3 + L3 (16MB) L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#0 + PU P#12 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#1 + PU P#13 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#2 + PU P#14 L2 (4096KB) + L1d (32KB) + L1i (32KB) + Core P#3 + PU P#15 HostBridge P#0 PCI 8086:7010 Block(Removable Media Device) "sr0" PCI 1234: GPU "card0" GPU "controlD64" PCI 1af4:1004 PCI 1af4:1000 *Michael Tie*Technical Director Mathematics, Statistics, and Computer Science One North College Street phn: 507-222-4067 Northfield, MN 55057 cel:952-212-8933 m...@carleton.edufax:507-222-4312 On Tue, Mar 10, 2020 at 12:21 AM Chris Samuel wrote: > On 9/3/20 7:44 am, mike tie wrote: > > > Specifically, how is slurmd -C getting that info? Maybe this is a > > kernel issue, but other than lscpu and /proc/cpuinfo, I don't know where > > to look. Maybe I should be looking at the slurmd source? > > It would be worth looking at what something like "lstopo" from the hwloc > package says about your VM. > > All the best, > Chris > -- > Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA > >
Re: [slurm-users] slurmd -C showing incorrect core count
Yes, it's odd. -kkm On Mon, Mar 9, 2020 at 7:44 AM mike tie wrote: > > Interesting. I'm still confused by the where slurmd -C is getting the > data. When I think of where the kernel stores info about the processor, I > normally think of /proc/cpuinfo. (by the way, I am running centos 7 in the > vm. The vm hypervisor is VMware). /proc/cpuinfo does show 16 cores. > AFAIK, the topology can be queried from /sys/devices/system/node/node*/ < https://www.kernel.org/doc/html/latest/admin-guide/mm/numaperf.html> and /sys/devices/system/cpu/cpu*/topology. Whether or not Slurm in fact gets the topology from there, I do not know. The build has dependencies on both libhwloc and libnuma--that's a clue. > I understand your concern over the processor speed. So I tried a > different vm where I see the following specs: > It's not even so much its speed per se, rather the way the hypervisor has finely chopped the 16 virtual CPUs into 4 sockets without hyperthreads. It makes no sense at all. I have a hunch that the other VM (the one that reports the correct CPU) should rather put them into a single socket, at least by default. But yeah, it does not answer the question where the number 10 is popping up from. > When I increase the core count on that vm, reboot, and run slurm -C it too > continues to show the lower original core count. > Most likely it's stored somewhere on disk. > Specifically, how is slurmd -C getting that info? Maybe this is a kernel > issue, but other than lscpu and /proc/cpuinfo, I don't know where to look. > I would not bet 1 to 100 on a kernel bug. The number is most likely to come from either some stray config file, or a cache on disk. I don't know if slurmd stores any cache, never had to look (all my nodes are virtual and created and deleted on demand, thus always start fresh), but if it does, it's somewhere under /var/lib/slurm*. I thought (possibly incorrectly) that the switch -C reports the node size and CPU configuration without even looking at config files. I would check first if it talks to the controller at all (tweak e.g. the port number in slurm.conf), and, if it does, what is the current slurmctld's idea about this node (scontrol show node=, IIRC, or something very much like that). > Maybe I should be looking at the slurmd source? > slurmd should be much simpler than slurmctld, and the -C query must be a straightforward, very synchronous operation. But reading sources is quite time-consuming, so I would venture into it only as a last resort. Since -C is not forking, it should be easy to run it under gdb. YMMV, of course. -kkm
Re: [slurm-users] slurmd -C showing incorrect core count
On 9/3/20 7:44 am, mike tie wrote: Specifically, how is slurmd -C getting that info? Maybe this is a kernel issue, but other than lscpu and /proc/cpuinfo, I don't know where to look. Maybe I should be looking at the slurmd source? It would be worth looking at what something like "lstopo" from the hwloc package says about your VM. All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
Re: [slurm-users] slurmd -C showing incorrect core count
Interesting. I'm still confused by the where slurmd -C is getting the data. When I think of where the kernel stores info about the processor, I normally think of /proc/cpuinfo. (by the way, I am running centos 7 in the vm. The vm hypervisor is VMware). /proc/cpuinfo does show 16 cores. I understand your concern over the processor speed. So I tried a different vm where I see the following specs: vendor_id : GenuineIntel cpu family : 6 model : 85 model name : Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz When I increase the core count on that vm, reboot, and run slurm -C it too continues to show the lower original core count. Specifically, how is slurmd -C getting that info? Maybe this is a kernel issue, but other than lscpu and /proc/cpuinfo, I don't know where to look. Maybe I should be looking at the slurmd source? -Mike *Michael Tie*Technical Director Mathematics, Statistics, and Computer Science One North College Street phn: 507-222-4067 Northfield, MN 55057 cel:952-212-8933 m...@carleton.edufax:507-222-4312 On Sun, Mar 8, 2020 at 7:32 PM Kirill 'kkm' Katsnelson wrote: > To answer your direct question, the ground truth of 'slurmctld -C' is what > the kernel thinks the hardware is (what you see in lscpu, except it > probably employs some tricks for VMs with an odd topology). And it got > severely confused by what the kernel reported to it. I know from experience > that certain odd cloud VM shapes throw it off balance. > > I do not really like the output of lscpu. I have never seen such a strange > shape of a VM. CPU family 15 is in the Pentium 4 line < > https://software.intel.com/en-us/articles/intel-architecture-and-processor-identification-with-cpuid-model-and-family-numbers>, > and model 6 was the last breath of this unsuccessful NetBurst > architecture--such a rarity that Linux kernel does not even have in its > database: "Common KVM processor" is a slug for "everything else that one > of these soul-sapping KVMs may return". Flags show that the processor > supports SSE2 and 3, but not 4.1, 4.2 or AVX, which is consistent with a > Pentium 4, but 16M of L3 cache is about an average total RAM in a desktop > at the time P4 was a thing. And the CPU is a NUMA (no real Pentium 4 had > the NUMA, only SMP)¹. > > Any advice? >> > > My best advice would be to either use a different hypervisor or tune > correctly the one you have. Sometimes a hypervisor is tuned for live VM > migration, when it is frozen on one hardware type and thawed on another, > and may tweak the CPUID in advance to hide features from the guest OS so > that it would be able to continue if migrated to less capable hardware; but > still, using the P4 as the least common denominator is way too extreme. > Something is seriously wrong on the KVM host. > > The VM itself is braindead. Even if you will have got it up and running, > the absence of SSE4.1 and 4.2, AVX, AXV2, and AVX512² would make it about > as efficient a computing node as a brick. Unless the host CPU is really a > Presler Pentium 4, in which case you are way too long overdue for a > hardware upgrade :))) > > -kkm > > ¹ It's not impossible that lscpu shows an SMP machine as if containing a > single NUMA node, but I have a recollection that this is not the case. I > haven't seen a non-NUMA CPU in quite a while. > ² Intel had gone besides-itself-creative this time. It was even bigger a > naming leap than switching from Roman to decimal between Pentium III to > Pentium *drum roll* 4 *cymbal crash*. > > > On Sun, Mar 8, 2020 at 1:20 PM mike tie wrote: > >> >> I am running a slurm client on a virtual machine. The virtual machine >> originally had a core count of 10. But I have now increased the cores to >> 16, but "slurmd -C" continues to show 10. I have increased the core count >> in the slurm.conf file. and that is being seen correctly. The state of the >> node is stuck in a Drain state because of this conflict. How do I get >> slurmd -C to see the new number of cores? >> >> I'm running slurm 18.08. I have tried running "scontrol reconfigure" on >> the head node. I have restarted slurmd on all the client nodes, and I have >> restarted slurmctld on the master node. >> >> Where is the data about compute note CPUs stored? I can't seem to find a >> config or setting file on the compute node. >> >> The compute node that I am working on is "liverpool" >> >> *mtie@liverpool** ~ $* slurmd -C >> >> NodeName=liverpool CPUs=10 Boards=1 SocketsPerBoard=10 CoresPerSocket=1 >> ThreadsPerCore=1 RealMemory=64263 >> >> UpTime=1-21:55:36 >> >> >> *mtie@liverpool** ~ $* lscpu >> >> Architecture: x86_64 >> >> CPU op-mode(s):32-bit, 64-bit >> >> Byte Order:Little Endian >> >> CPU(s):16 >> >> On-line CPU(s) list: 0-15 >> >> Thread(s) per core:1 >> >> Core(s) per socket:4 >> >> Socket(s): 4 >> >> NUMA node(s): 1 >> >>
Re: [slurm-users] slurmd -C showing incorrect core count
To answer your direct question, the ground truth of 'slurmctld -C' is what the kernel thinks the hardware is (what you see in lscpu, except it probably employs some tricks for VMs with an odd topology). And it got severely confused by what the kernel reported to it. I know from experience that certain odd cloud VM shapes throw it off balance. I do not really like the output of lscpu. I have never seen such a strange shape of a VM. CPU family 15 is in the Pentium 4 line < https://software.intel.com/en-us/articles/intel-architecture-and-processor-identification-with-cpuid-model-and-family-numbers>, and model 6 was the last breath of this unsuccessful NetBurst architecture--such a rarity that Linux kernel does not even have in its database: "Common KVM processor" is a slug for "everything else that one of these soul-sapping KVMs may return". Flags show that the processor supports SSE2 and 3, but not 4.1, 4.2 or AVX, which is consistent with a Pentium 4, but 16M of L3 cache is about an average total RAM in a desktop at the time P4 was a thing. And the CPU is a NUMA (no real Pentium 4 had the NUMA, only SMP)¹. Any advice? > My best advice would be to either use a different hypervisor or tune correctly the one you have. Sometimes a hypervisor is tuned for live VM migration, when it is frozen on one hardware type and thawed on another, and may tweak the CPUID in advance to hide features from the guest OS so that it would be able to continue if migrated to less capable hardware; but still, using the P4 as the least common denominator is way too extreme. Something is seriously wrong on the KVM host. The VM itself is braindead. Even if you will have got it up and running, the absence of SSE4.1 and 4.2, AVX, AXV2, and AVX512² would make it about as efficient a computing node as a brick. Unless the host CPU is really a Presler Pentium 4, in which case you are way too long overdue for a hardware upgrade :))) -kkm ¹ It's not impossible that lscpu shows an SMP machine as if containing a single NUMA node, but I have a recollection that this is not the case. I haven't seen a non-NUMA CPU in quite a while. ² Intel had gone besides-itself-creative this time. It was even bigger a naming leap than switching from Roman to decimal between Pentium III to Pentium *drum roll* 4 *cymbal crash*. On Sun, Mar 8, 2020 at 1:20 PM mike tie wrote: > > I am running a slurm client on a virtual machine. The virtual machine > originally had a core count of 10. But I have now increased the cores to > 16, but "slurmd -C" continues to show 10. I have increased the core count > in the slurm.conf file. and that is being seen correctly. The state of the > node is stuck in a Drain state because of this conflict. How do I get > slurmd -C to see the new number of cores? > > I'm running slurm 18.08. I have tried running "scontrol reconfigure" on > the head node. I have restarted slurmd on all the client nodes, and I have > restarted slurmctld on the master node. > > Where is the data about compute note CPUs stored? I can't seem to find a > config or setting file on the compute node. > > The compute node that I am working on is "liverpool" > > *mtie@liverpool** ~ $* slurmd -C > > NodeName=liverpool CPUs=10 Boards=1 SocketsPerBoard=10 CoresPerSocket=1 > ThreadsPerCore=1 RealMemory=64263 > > UpTime=1-21:55:36 > > > *mtie@liverpool** ~ $* lscpu > > Architecture: x86_64 > > CPU op-mode(s):32-bit, 64-bit > > Byte Order:Little Endian > > CPU(s):16 > > On-line CPU(s) list: 0-15 > > Thread(s) per core:1 > > Core(s) per socket:4 > > Socket(s): 4 > > NUMA node(s): 1 > > Vendor ID: GenuineIntel > > CPU family:15 > > Model: 6 > > Model name:Common KVM processor > > Stepping: 1 > > CPU MHz: 2600.028 > > BogoMIPS: 5200.05 > > Hypervisor vendor: KVM > > Virtualization type: full > > L1d cache: 32K > > L1i cache: 32K > > L2 cache: 4096K > > L3 cache: 16384K > > NUMA node0 CPU(s): 0-15 > > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr > pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm > constant_tsc nopl xtopology eagerfpu pni cx16 x2apic hypervisor lahf_lm > > > *mtie@liverpool** ~ $* more /etc/slurm/slurm.conf | grep liverpool > > NodeName=*liverpool* NodeAddr=137.22.10.202 CPUs=16 State=UNKNOWN > > PartitionName=BioSlurm Nodes=*liverpool* Default=YES MaxTime=INFINITE > State=UP > > > *mtie@liverpool** ~ $* sinfo -n liverpool -o %c > > CPUS > > 16 > > *mtie@liverpool** ~ $* sinfo -n liverpool -o %E > > REASON > > Low socket*core*thread count, Low CPUs > > > > Any advice? > > >
[slurm-users] slurmd -C showing incorrect core count
I am running a slurm client on a virtual machine. The virtual machine originally had a core count of 10. But I have now increased the cores to 16, but "slurmd -C" continues to show 10. I have increased the core count in the slurm.conf file. and that is being seen correctly. The state of the node is stuck in a Drain state because of this conflict. How do I get slurmd -C to see the new number of cores? I'm running slurm 18.08. I have tried running "scontrol reconfigure" on the head node. I have restarted slurmd on all the client nodes, and I have restarted slurmctld on the master node. Where is the data about compute note CPUs stored? I can't seem to find a config or setting file on the compute node. The compute node that I am working on is "liverpool" *mtie@liverpool** ~ $* slurmd -C NodeName=liverpool CPUs=10 Boards=1 SocketsPerBoard=10 CoresPerSocket=1 ThreadsPerCore=1 RealMemory=64263 UpTime=1-21:55:36 *mtie@liverpool** ~ $* lscpu Architecture: x86_64 CPU op-mode(s):32-bit, 64-bit Byte Order:Little Endian CPU(s):16 On-line CPU(s) list: 0-15 Thread(s) per core:1 Core(s) per socket:4 Socket(s): 4 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family:15 Model: 6 Model name:Common KVM processor Stepping: 1 CPU MHz: 2600.028 BogoMIPS: 5200.05 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0-15 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology eagerfpu pni cx16 x2apic hypervisor lahf_lm *mtie@liverpool** ~ $* more /etc/slurm/slurm.conf | grep liverpool NodeName=*liverpool* NodeAddr=137.22.10.202 CPUs=16 State=UNKNOWN PartitionName=BioSlurm Nodes=*liverpool* Default=YES MaxTime=INFINITE State=UP *mtie@liverpool** ~ $* sinfo -n liverpool -o %c CPUS 16 *mtie@liverpool** ~ $* sinfo -n liverpool -o %E REASON Low socket*core*thread count, Low CPUs Any advice?
