Quoting Janne Blomqvist <[email protected]>: > I'm thinking that one could do something different in the case when > all the nodes for a job disappear. E.g. for serial or small parallel > jobs that fit within a single node, or somebody tripping over the > network cable connecting the slurmctld node to the rest of the > cluster, or some supposedly quick maintenance task taking longer > than expected, etc. In such a case, couldn't slurmctld wait with > killing the job until 1) a node returns to service and slurmd (or is > it slurmstepd?) reports that the job is no longer present, in which > case the job can be immediately killed 2) a node returns to service > and slurmd/slurmstepd reports that the job is still there, start the > timer and wait until SlurmdTimeout for the other nodes to return, > and if they don't, kill the job. > > Janne Blomqvist
That is a very good idea. If all of the nodes associated with a job are not responding, they will not be used for any other jobs, so delaying the killing of a job has no negative effect. It should also be relatively simple to do using a simple bitmap operation. Moe
