Maybe you could use a Gres;
128GB machines could have Gres=M64,M128 512GB machines Gres=M64,M128,M512 When a user submits a job that only requires 64GB (--gres=M64), it will go to any machine - using partition priority to assign smaller nodes first. Another 64GB job cannot go to the same nodes. A 512GB job can only go to the bigger memory machines. On Sun, 2014-12-21 at 21:11 -0800, Morris Jette wrote: > If you set RealMemory high then you will also need to configure > FastSchedule=2, but that also prevents Slurm from downing nodes with > down CPUs or other hardware. > > On December 21, 2014 8:22:43 PM PST, [email protected] wrote: > > Hi, > > We are having trouble with scheduling a cluster with nodes with two > memory sizes. Most of our nodes nominally have 128GB and a few have 512GB. > Unsurprisingly, slurm detects that slightly less memory is available to > schedule. We want: > 1) users to be able to make simple choices to get what they need > 2) the scarce large memory nodes to be available when needed > > The strategy we are pursuing has SelectType=select/cons_res and a > partition scheme to separate out the larger memory nodes, with overlapping > partitions so those nodes can run either short jobs or jobs that explicitly > request the partition with larger nodes. > > General comments on the strategy are welcome, but our immediate > problem is that if a user requests --mem=131072 (128*1024) then slurm will > not assign a 128GB node. Also 2 "256GB" jobs cannot co-exist on a 512GB node > (or 2x64GB on 128GB or ...). > > We've tried setting RealMemory explicitl > y but > it did not seem to take and reading the mailing list it is probably > not the right setting (it seems to be an assertion that this node is expected > to have at least this amount of memory else consider it to be in a bad > state). FWIW we altered slurm.conf and ran 'scontrol reconfigure'. > > Can we override the amount of memory that is scheduled (explicitly or > with an overcommit factor) or do we have to tell our users to request > slightly less memory? > > I'd prefer not to have to write an explicit plugin to tweak users > requests or to have to put our own fudge factor in the existing scheduler. > > Thanks, > > Gareth Williams Ph.D. > CSIRO > www.csiro.au | https://wiki.csiro.au/display/ASC/ > PLEASE NOTE > The information contained in this email may be confidential or > privileged. Any unauthorised use or disclosure is prohibited. If you have > received this email in > error, > please delete it immediately and notify the sender by return email. > Thank you. To the extent permitted by law, CSIRO does not represent, warrant > and/or guarantee that the integrity of this communication has been maintained > or that the communication is free of errors, virus, interception or > interference. > Please consider the environment before printing this email. >
