If you set RealMemory high then you will also need to configure FastSchedule=2, but that also prevents Slurm from downing nodes with down CPUs or other hardware.
On December 21, 2014 8:22:43 PM PST, [email protected] wrote: > >Hi, > >We are having trouble with scheduling a cluster with nodes with two >memory sizes. Most of our nodes nominally have 128GB and a few have >512GB. Unsurprisingly, slurm detects that slightly less memory is >available to schedule. We want: > 1) users to be able to make simple choices to get what they need > 2) the scarce large memory nodes to be available when needed > >The strategy we are pursuing has SelectType=select/cons_res and a >partition scheme to separate out the larger memory nodes, with >overlapping partitions so those nodes can run either short jobs or jobs >that explicitly request the partition with larger nodes. > >General comments on the strategy are welcome, but our immediate problem >is that if a user requests --mem=131072 (128*1024) then slurm will not >assign a 128GB node. Also 2 "256GB" jobs cannot co-exist on a 512GB >node (or 2x64GB on 128GB or ...). > >We've tried setting RealMemory explicitly but it did not seem to take >and reading the mailing list it is probably not the right setting (it >seems to be an assertion that this node is expected to have at least >this amount of memory else consider it to be in a bad state). FWIW we >altered slurm.conf and ran 'scontrol reconfigure'. > >Can we override the amount of memory that is scheduled (explicitly or >with an overcommit factor) or do we have to tell our users to request >slightly less memory? > >I'd prefer not to have to write an explicit plugin to tweak users >requests or to have to put our own fudge factor in the existing >scheduler. > >Thanks, > >Gareth Williams Ph.D. >CSIRO >www.csiro.au | https://wiki.csiro.au/display/ASC/ >PLEASE NOTE >The information contained in this email may be confidential or >privileged. Any unauthorised use or disclosure is prohibited. If you >have received this email in error, please delete it immediately and >notify the sender by return email. Thank you. To the extent permitted >by law, CSIRO does not represent, warrant and/or guarantee that the >integrity of this communication has been maintained or that the >communication is free of errors, virus, interception or interference. >Please consider the environment before printing this email. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
