Lindsay,
Sorry about that. There is a one line patch for this below which will
be in Slurm v2.5.2. The bug was introduced with new advanced
reservation logic that can reserve specific cores in a node rather
than entire nodes. The core reservation logic does not work for
Bluegene or Cray systems, but Bluegene support should be in Slurm v2.6.
Moe
diff --git a/src/slurmctld/reservation.c b/src/slurmctld/reservation.c
index 6fdc5f1..d2cfa6e 100644
--- a/src/slurmctld/reservation.c
+++ b/src/slurmctld/reservation.c
@@ -1314,6 +1314,7 @@ slurmctld_resv_t *_load_reservation_state(Buf buffer,
safe_unpack_time(&resv_ptr->start_time, buffer);
safe_unpack8((uint8_t *)&resv_ptr->user_not, buffer);
} else {
+ resv_ptr->full_nodes = 1; /* Added in v2.5 */
safe_unpackstr_xmalloc(&resv_ptr->accounts,
&uint32_tmp, buffer);
safe_unpack_time(&resv_ptr->end_time, buffer);
Quoting Lindsay Todd <[email protected]>:
> We are updating to slurm 2.5.1 from 2.4.5 on our SL5 x86 cluster
> today. We have a slurm maintenance reservation in place for this
> cluster.
>
> Starting with a scontrol shutdown, I made a backup copy of our mysql
> slurm database and our slurm spool. Then I updated the slurm database
> successfully -- slurmdbd runs and accepts connections.
>
> Now to the slurm controller and backup controller -- here things did
> not go so well. The connection is made to the slurm database, then we
> have an assertion error. I ran this under gdb:
>
> (gdb) run -D
> Starting program: /usr/sbin/slurmctld -D
> warning: no loadable sections found in added symbol-file
> system-supplied DSO at
> 0x2aaaaaaab000
> [Thread debugging using libthread_db enabled]
> slurmctld: Job accounting information stored, but details not gathered
> [New Thread 0x40100940 (LWP 2371)]
> [New Thread 0x40201940 (LWP 2372)]
> [New Thread 0x40302940 (LWP 2373)]
> slurmctld: slurmctld version 2.5.1 started on cluster ccnix86
> [New Thread 0x40403940 (LWP 2375)]
> slurmctld: error: read_slurm_conf: default partition not set.
> slurmctld: Recovered state of 408 nodes
> slurmctld: Recovered job 1090751 2511
> slurmctld: Recovered job 1090828 2511
> slurmctld: Recovered job 1090836 2511
> slurmctld: Recovered job 1090837 2511
> slurmctld: Recovered job 1090838 2511
> slurmctld: Recovered job 1090839 2511
> slurmctld: Recovered job 1090840 2511
> slurmctld: Recovered job 1090841 2511
> slurmctld: Recovered job 1090842 2511
> slurmctld: Recovered job 1090843 2511
> slurmctld: Recovered job 1090844 2511
> slurmctld: Recovered job 1090845 2511
> slurmctld: Recovered job 1090846 2511
> slurmctld: Recovered job 1090847 2511
> slurmctld: Recovered job 1090848 2511
> slurmctld: Recovered job 1090849 2511
> slurmctld: Recovered job 1090850 2511
> slurmctld: Recovered job 1090851 2511
> slurmctld: Recovered job 1090852 2511
> slurmctld: Recovered job 1090853 2511
> slurmctld: Recovered job 1090854 2511
> slurmctld: Recovered job 1090855 2511
> slurmctld: Recovered job 1090856 2511
> slurmctld: Recovered job 1090857 2511
> slurmctld: Recovered job 1090858 2511
> slurmctld: Recovered job 1090859 2511
> slurmctld: Recovered job 1090860 2511
> slurmctld: Recovered job 1090861 2511
> slurmctld: Recovered job 1090862 2511
> slurmctld: Recovered job 1090863 2511
> slurmctld: Recovered job 1090864 2511
> slurmctld: Recovered job 1090865 2511
> slurmctld: Recovered job 1090866 2511
> slurmctld: Recovered job 1090902 2596
> slurmctld: Recovered job 1090903 715
> slurmctld: Recovered job 1090904 715
> slurmctld: Recovered job 1090905 715
> slurmctld: Recovered job 1090906 715
> slurmctld: Recovered job 1091152 2838
> slurmctld: Recovered job 1091162 2511
> slurmctld: Recovered job 1091163 2511
> slurmctld: Recovered job 1091164 2511
> slurmctld: Recovered job 1091165 2511
> slurmctld: Recovered job 1091166 2511
> slurmctld: Recovered job 1091167 2511
> slurmctld: Recovered job 1091168 2511
> slurmctld: Recovered job 1091169 2511
> slurmctld: Recovered job 1091170 2511
> slurmctld: Recovered job 1091171 2511
> slurmctld: Recovered job 1091172 2511
> slurmctld: Recovered job 1091173 2511
> slurmctld: Recovered job 1091174 2511
> slurmctld: Recovered job 1091175 2511
> slurmctld: Recovered job 1091176 2511
> slurmctld: Recovered job 1091177 2511
> slurmctld: Recovered job 1091178 2511
> slurmctld: Recovered job 1091179 2511
> slurmctld: Recovered job 1091180 2511
> slurmctld: Recovered job 1091181 2511
> slurmctld: Recovered job 1091182 2511
> slurmctld: Recovered job 1091183 2511
> slurmctld: Recovered job 1091184 2511
> slurmctld: Recovered job 1091185 2511
> slurmctld: Recovered job 1091186 2511
> slurmctld: Recovered job 1091187 2511
> slurmctld: Recovered job 1091188 2511
> slurmctld: Recovered job 1091189 2511
> slurmctld: Recovered job 1091190 2511
> slurmctld: Recovered job 1091191 2511
> slurmctld: Recovered job 1091192 2511
> slurmctld: Recovered job 1091814 240
> slurmctld: Recovered job 1091815 240
> slurmctld: Recovered job 1091816 240
> slurmctld: Recovered job 1091817 240
> slurmctld: Recovered job 1091818 240
> slurmctld: Recovered job 1091819 240
> slurmctld: Recovered job 1091820 240
> slurmctld: Recovered job 1091821 240
> slurmctld: Recovered job 1091822 240
> slurmctld: Recovered job 1091823 240
> slurmctld: Recovered job 1091824 240
> slurmctld: Recovered job 1091825 240
> slurmctld: Recovered job 1091826 240
> slurmctld: Recovered job 1091827 240
> slurmctld: Recovered job 1091828 240
> slurmctld: Recovered job 1091829 240
> slurmctld: Recovered job 1091830 240
> slurmctld: Recovered job 1091831 240
> slurmctld: Recovered job 1091832 240
> slurmctld: Recovered job 1091833 240
> slurmctld: Recovered job 1091834 240
> slurmctld: Recovered job 1091835 240
> slurmctld: Recovered job 1091836 240
> slurmctld: Recovered job 1091837 240
> slurmctld: Recovered job 1091838 240
> slurmctld: Recovered job 1091839 240
> slurmctld: Recovered job 1091840 240
> slurmctld: Recovered job 1091841 240
> slurmctld: Recovered job 1091842 240
> slurmctld: Recovered job 1091843 240
> slurmctld: Recovered job 1091844 240
> slurmctld: Recovered job 1091845 240
> slurmctld: Recovered job 1091846 240
> slurmctld: Recovered job 1091847 240
> slurmctld: Recovered job 1091848 240
> slurmctld: Recovered job 1091849 240
> slurmctld: Recovered job 1091850 240
> slurmctld: Recovered job 1091851 240
> slurmctld: Recovered job 1091852 240
> slurmctld: Recovered job 1091853 240
> slurmctld: Recovered job 1091854 240
> slurmctld: Recovered job 1091855 240
> slurmctld: Recovered job 1091856 240
> slurmctld: Recovered job 1091857 240
> slurmctld: Recovered job 1091858 240
> slurmctld: Recovered job 1091859 240
> slurmctld: Recovered job 1091860 240
> slurmctld: Recovered job 1091861 240
> slurmctld: Recovered job 1091862 240
> slurmctld: Recovered job 1094137 737
> slurmctld: Recovered job 1094138 731
> slurmctld: Recovered job 1094241 2601
> slurmctld: Recovered job 1094242 2601
> slurmctld: Recovered job 1094243 2601
> slurmctld: Recovered job 1094244 2601
> slurmctld: Recovered job 1094245 2601
> slurmctld: Recovered job 1094246 2601
> slurmctld: Recovered job 1094247 2601
> slurmctld: Recovered job 1094248 2601
> slurmctld: Recovered job 1094249 2601
> slurmctld: Recovered job 1094250 1962
> slurmctld: Recovered job 1094257 178
> slurmctld: Recovered job 1094258 2646
> slurmctld: Recovered job 1094259 2646
> slurmctld: Recovered job 1094260 2646
> slurmctld: Recovered job 1094261 2646
> slurmctld: Recovered job 1094278 737
> slurmctld: Recovered job 1094279 737
> slurmctld: Recovered job 1094280 737
> slurmctld: Recovered job 1094281 737
> slurmctld: Recovered job 1094282 737
> slurmctld: Recovered job 1094283 737
> slurmctld: Recovered job 1094284 737
> slurmctld: Recovered job 1094285 737
> slurmctld: Recovered job 1094286 737
> slurmctld: Recovered information about 145 jobs
> slurmctld: Purging files for defunct batch job 545881
> slurmctld: Purging files for defunct batch job 545885
> slurmctld: Purging files for defunct batch job 545889
> slurmctld: Purging files for defunct batch job 545199
> slurmctld: Purging files for defunct batch job 545991
> slurmctld: Purging files for defunct batch job 545891
> slurmctld: Purging files for defunct batch job 545752
> slurmctld: Purging files for defunct batch job 545852
> slurmctld: Purging files for defunct batch job 545846
> slurmctld: Purging files for defunct batch job 545943
> slurmctld: Purging files for defunct batch job 545947
> slurmctld: Purging files for defunct batch job 545753
> slurmctld: Purging files for defunct batch job 546063
> slurmctld: Purging files for defunct batch job 545198
> slurmctld: Purging files for defunct batch job 545990
> slurmctld: Purging files for defunct batch job 545884
> slurmctld: Purging files for defunct batch job 546065
> slurmctld: Purging files for defunct batch job 545896
> slurmctld: Purging files for defunct batch job 545879
> slurmctld: Purging files for defunct batch job 545920
> slurmctld: Purging files for defunct batch job 545869
> slurmctld: Purging files for defunct batch job 545949
> slurmctld: Purging files for defunct batch job 546004
> slurmctld: Purging files for defunct batch job 546008
> slurmctld: Purging files for defunct batch job 546041
> slurmctld: Purging files for defunct batch job 545200
> slurmctld: Purging files for defunct batch job 545955
> slurmctld: Purging files for defunct batch job 545850
> slurmctld: Purging files for defunct batch job 545158
> slurmctld: Purging files for defunct batch job 545954
> slurmctld: Purging files for defunct batch job 545201
> slurmctld: Purging files for defunct batch job 545750
> slurmctld: Purging files for defunct batch job 545911
> slurmctld: Purging files for defunct batch job 546001
> slurmctld: Purging files for defunct batch job 545948
> slurmctld: Purging files for defunct batch job 545848
> slurmctld: Purging files for defunct batch job 545915
> slurmctld: Purging files for defunct batch job 545940
> slurmctld: Purging files for defunct batch job 545944
> slurmctld: Purging files for defunct batch job 545748
> slurmctld: Purging files for defunct batch job 545919
> slurmctld: Purging files for defunct batch job 545868
> slurmctld: Purging files for defunct batch job 545897
> slurmctld: Purging files for defunct batch job 546064
> slurmctld: Purging files for defunct batch job 545878
> slurmctld: Recovered state of reservation root_162
> slurmctld: Recovered state of 1 reservations
> slurmctld: Running as primary controller
> slurmctld: Registering slurmctld at port 6817 with slurmdbd.
> slurmctld: job_test_resv: root_162 reservation uses partial nodes
> slurmctld: bitstring.c:588: bit_copy: Assertion `(b) != ((void *)0)' failed.
>
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x40403940 (LWP 2375)]
> 0x0000003297230285 in raise () from /lib64/libc.so.6
> (gdb) where
> #0 0x0000003297230285 in raise () from /lib64/libc.so.6
> #1 0x0000003297231d30 in abort () from /lib64/libc.so.6
> #2 0x0000003297229706 in __assert_fail () from /lib64/libc.so.6
> #3 0x000000000049933f in bit_copy (b=0x0) at bitstring.c:588
> #4 0x000000000046b0d3 in job_test_resv (job_ptr=0x9656c8, when=0x40403050,
> move_time=true, node_bitmap=0x40403060, exc_core_bitmap=0x40403058)
> at reservation.c:3669
> #5 0x00002aaaabfbad08 in _attempt_backfill (args=<value optimized out>)
> at backfill.c:708
> #6 backfill_agent (args=<value optimized out>) at backfill.c:476
> #7 0x0000003297a0677d in start_thread () from /lib64/libpthread.so.0
> #8 0x00000032972d3c1d in clone () from /lib64/libc.so.6
> (gdb)
>
> I've attached the resv_state file. If I move this out of
> /var/spool/slurm, I can successfully restart slurmctld. At that point
> I can recreate the reservations, and this seems to be working even
> when I restart slurmctld. (Which means I am all set for now, but
> still someone else may run into this problem.)
>
> /Lindsay
> --
> R. Lindsay Todd, PhD email: [email protected]
> Senior Systems Programmer phone: 518-276-2605
> Rensselaer Polytechnic Institute fax: 518-276-2809
> Troy, NY 12180-3590 WWW: http://www.rpi.edu/~toddr
>
> The views, opinions, and judgments expressed in this message are
> solely those of the author. The message contents have not been
> reviewed or approved by Rensselaer.
>
