Hi, slurmctld 2.3.5 crashed and we got a core file. Is this gdb back trace meaningful? Is there other info that would be helpful or would anyone like a copy (~900MB)?
I had thought maybe it was related to --resv-ports which we just starting looking at ... module add openmpi/1.6-23/bin srun -p drdb-swdev -t 120 -N 4 --resv-ports --exclusive -- ... but upon trying that again several times it didn't crash, so maybe it was probably something else. drdlogin0079:~$ scontrol -V slurm 2.3.5 drdlogin0079:all$ file /var/spool/slurmcl2/all/drdlogin0079.en.desres.deshaw.com-1476.core /var/spool/slurmcl2/all/drdlogin0079.en.desres.deshaw.com-1476.core: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'slurmctld drdlogin0079:all$ gdb /opt/slurmcl2/sbin/slurmctld /var/spool/slurmcl2/all/drdlogin0079.en.desres.deshaw.com-1476.core GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /opt/slurmcl2/sbin/slurmctld...done. [New Thread 14119] [New Thread 14116] [New Thread 14115] [New Thread 1745] [New Thread 1744] [New Thread 1743] [New Thread 1742] [New Thread 1741] [New Thread 1589] [New Thread 1484] [New Thread 1479] [New Thread 1478] [New Thread 1476] Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /opt/slurmcl2/lib64/slurm/accounting_storage_slurmdbd.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/accounting_storage_slurmdbd.so Reading symbols from /opt/slurmcl2/lib64/slurm/auth_munge.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/auth_munge.so Reading symbols from /usr/lib64/libmunge.so.2...done. Loaded symbols for /usr/lib64/libmunge.so.2 Reading symbols from /opt/slurmcl2/lib64/slurm/crypto_munge.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/crypto_munge.so Reading symbols from /opt/slurmcl2/lib64/slurm/select_cons_res.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/select_cons_res.so Reading symbols from /opt/slurmcl2/lib64/slurm/preempt_none.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/preempt_none.so Reading symbols from /opt/slurmcl2/lib64/slurm/checkpoint_none.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/checkpoint_none.so Reading symbols from /opt/slurmcl2/lib64/slurm/jobacct_gather_linux.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/jobacct_gather_linux.so Reading symbols from /opt/slurmcl2/lib64/slurm/switch_none.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/switch_none.so Reading symbols from /opt/slurmcl2/lib64/slurm/topology_none.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/topology_none.so Reading symbols from /opt/slurmcl2/lib64/slurm/sched_backfill.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/sched_backfill.so Reading symbols from /opt/slurmcl2/lib64/slurm/priority_multifactor.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/priority_multifactor.so Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/slurmcl2/lib64/slurm/jobcomp_none.so...done. Loaded symbols for /opt/slurmcl2/lib64/slurm/jobcomp_none.so warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff522f5000 Core was generated by `/opt/slurmcl2/sbin/slurmctld -n -18'. Program terminated with signal 6, Aborted. #0 0x000000396f030265 in raise () from /lib64/libc.so.6 (gdb) where #0 0x000000396f030265 in raise () from /lib64/libc.so.6 #1 0x000000396f031d10 in abort () from /lib64/libc.so.6 #2 0x000000396f0296e6 in __assert_fail () from /lib64/libc.so.6 #3 0x000000000048f511 in bit_overlap (b1=0x11419230, b2=0x1146b5b0) at bitstring.c:690 #4 0x0000000000456b7f in resv_port_alloc (step_ptr=0x120006b8) at port_mgr.c:240 #5 0x0000000000471f83 in step_create (step_specs=0x12224238, new_step_record=0x451a0b38, batch_step=4) at step_mgr.c:2007 #6 0x000000000045a00a in _slurm_rpc_job_step_create (msg=0x11aa0768) at proc_req.c:1663 #7 0x000000000045f89e in slurmctld_req (msg=0x11aa0768) at proc_req.c:236 #8 0x000000000042d9ab in _service_connection (arg=0x2aaab800bdb8) at controller.c:1018 #9 0x000000396fc0673d in start_thread () from /lib64/libpthread.so.0 #10 0x000000396f0d44bd in clone () from /lib64/libc.so.6 Thanks in advance, Chris
