Hi, We are having trouble with scheduling a cluster with nodes with two memory sizes. Most of our nodes nominally have 128GB and a few have 512GB. Unsurprisingly, slurm detects that slightly less memory is available to schedule. We want: 1) users to be able to make simple choices to get what they need 2) the scarce large memory nodes to be available when needed
The strategy we are pursuing has SelectType=select/cons_res and a partition scheme to separate out the larger memory nodes, with overlapping partitions so those nodes can run either short jobs or jobs that explicitly request the partition with larger nodes. General comments on the strategy are welcome, but our immediate problem is that if a user requests --mem=131072 (128*1024) then slurm will not assign a 128GB node. Also 2 "256GB" jobs cannot co-exist on a 512GB node (or 2x64GB on 128GB or ...). We've tried setting RealMemory explicitly but it did not seem to take and reading the mailing list it is probably not the right setting (it seems to be an assertion that this node is expected to have at least this amount of memory else consider it to be in a bad state). FWIW we altered slurm.conf and ran 'scontrol reconfigure'. Can we override the amount of memory that is scheduled (explicitly or with an overcommit factor) or do we have to tell our users to request slightly less memory? I'd prefer not to have to write an explicit plugin to tweak users requests or to have to put our own fudge factor in the existing scheduler. Thanks, Gareth Williams Ph.D. CSIRO www.csiro.au | https://wiki.csiro.au/display/ASC/ PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email.
