Sorry for the lost jobs. There were two new RPCs that needed to be added to a table, but were not. This prevented slurmd daemons running version 2.4. from communicating with the slurmctld daemon running version 2.5. The fix is here at the location shown below and will be in version 2.5.1:
https://github.com/SchedMD/slurm/commit/844f70a2c233e57f55948494bbd9c377163813fb I have already changed the code for version 2.6 to only list the RPCs that can not work between releases (job step creation and task launch). That is a much smaller list and will rarely change, which should prevent this problem in future releases. Moe Quoting Janne Blomqvist <[email protected]>: > > Hi, > > just a few quick notes about our experience upgrading from 2.4.x (2.4.3 > IIRC) to 2.5.0 yesterday. > > First, the bad news. Some, but not all, jobs were lost. Luckily the > cluster was mostly idle after the Christmas holidays, but still. First, > slurmdbd was upgraded, that went fine although it took a few minutes to > update the DB tables. Next the slurmctld master, followed by the > slurmctld backup, and then the compute nodes. AFAICT the reason for the > job loss was that after the master slurmctld was updated it was unable > to communicate with the compute nodes, and after the timeout expired it > killed the jobs. After starting slurmctld 2.5.0 in the log file we had > lots of messages like > > [2012-12-27T10:22:19+02:00] error: Invalid Protocol Version 6144 from > uid=0 at 10.10.253.52:6818 > [2012-12-27T10:22:19+02:00] error: slurm_receive_msgs: Protocol version > has changed, re-link your code > > followed by > > [2012-12-27T10:27:16+02:00] error: Nodes > cn[01-18,20-64,68-224],fn[01-02],gpu[001-008],tb[003,005-008] not responding > > and then finally lots of messages like > > [2012-12-27T10:28:59+02:00] Killing job_id 2722550 on failed node cn29 > > I suspect that the jobs that were not lost happened to run on nodes > where the slurmd upgrade finished before the timeout. Previously we have > successfully done on-the-fly upgrades between slurm major versions, and > also the release notes said it should work, so it was a bit surprising > that it would fail now. Oh well. > > > On a more positive note, we took the new multifactor2 plugin into use, > and so far it seems to work as designed, although due to the holidays > there's not much action in the queue so it's still too early for further > conclusions. > > -- > Janne Blomqvist >
