On 2012-12-28T22:21:04 EET, Moe Jette wrote: > > Sorry for the lost jobs. There were two new RPCs that needed to be > added to a table, but were not. This prevented slurmd daemons running > version 2.4. from communicating with the slurmctld daemon running > version 2.5. The fix is here at the location shown below and will be > in version 2.5.1: > > https://github.com/SchedMD/slurm/commit/844f70a2c233e57f55948494bbd9c377163813fb > > I have already changed the code for version 2.6 to only list the RPCs > that can not work between releases (job step creation and task > launch). That is a much smaller list and will rarely change, which > should prevent this problem in future releases.
Hi, thanks for fixing this! I wonder, would it be possible to make the job killing logic a bit less aggressive? In both cases where we have lost jobs during a slurm upgrade, it has been due to communication failure between slurmctld and slurmd's causing jobs to be killed after the timeout, even though in both situations the jobs were actually running just fine. Currently it seems that once slurmctld<->slurmd communication fails, a timer starts, and after SlurmdTimeout seconds jobs running on the node are killed. I understand that this kind of timeout-based killing logic is necessary to handle the case when one node in a multi-node job dies, and the job itself hangs rather than dies. Otherwise the hung job would occupy the healthy nodes until the job was killed by the job time limit. I'm thinking that one could do something different in the case when all the nodes for a job disappear. E.g. for serial or small parallel jobs that fit within a single node, or somebody tripping over the network cable connecting the slurmctld node to the rest of the cluster, or some supposedly quick maintenance task taking longer than expected, etc. In such a case, couldn't slurmctld wait with killing the job until 1) a node returns to service and slurmd (or is it slurmstepd?) reports that the job is no longer present, in which case the job can be immediately killed 2) a node returns to service and slurmd/slurmstepd reports that the job is still there, start the timer and wait until SlurmdTimeout for the other nodes to return, and if they don't, kill the job. > > Moe > > > Quoting Janne Blomqvist <[email protected]>: > >> >> Hi, >> >> just a few quick notes about our experience upgrading from 2.4.x (2.4.3 >> IIRC) to 2.5.0 yesterday. >> >> First, the bad news. Some, but not all, jobs were lost. Luckily the >> cluster was mostly idle after the Christmas holidays, but still. First, >> slurmdbd was upgraded, that went fine although it took a few minutes to >> update the DB tables. Next the slurmctld master, followed by the >> slurmctld backup, and then the compute nodes. AFAICT the reason for the >> job loss was that after the master slurmctld was updated it was unable >> to communicate with the compute nodes, and after the timeout expired it >> killed the jobs. After starting slurmctld 2.5.0 in the log file we had >> lots of messages like >> >> [2012-12-27T10:22:19+02:00] error: Invalid Protocol Version 6144 from >> uid=0 at 10.10.253.52:6818 >> [2012-12-27T10:22:19+02:00] error: slurm_receive_msgs: Protocol version >> has changed, re-link your code >> >> followed by >> >> [2012-12-27T10:27:16+02:00] error: Nodes >> cn[01-18,20-64,68-224],fn[01-02],gpu[001-008],tb[003,005-008] not responding >> >> and then finally lots of messages like >> >> [2012-12-27T10:28:59+02:00] Killing job_id 2722550 on failed node cn29 >> >> I suspect that the jobs that were not lost happened to run on nodes >> where the slurmd upgrade finished before the timeout. Previously we have >> successfully done on-the-fly upgrades between slurm major versions, and >> also the release notes said it should work, so it was a bit surprising >> that it would fail now. Oh well. >> >> >> On a more positive note, we took the new multifactor2 plugin into use, >> and so far it seems to work as designed, although due to the holidays >> there's not much action in the queue so it's still too early for further >> conclusions. >> >> -- >> Janne Blomqvist >> > -- Janne Blomqvist
