Sorry for the flood of messages, but I know 2.5.5 is about to be released, so I want to provide all available info about bugs ASAP...
I discovered that slurmctld only crashes if I use "CPUs=2" in slurm.conf. If I replace it with "CPUs=1" or "Sockets=1 CoresPerSocket=2 ThreadsPerCore=1", everything is fine. On 03/29/13 07:17, Jason Bacon wrote: > > Quick followup... > > I recompiled with -g -O0 to get more debug info: > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 800e0aac0 (LWP 100538/slurmctld)] > 0x00007fffff6f9418 in ?? () > (gdb) where > #0 0x00007fffff6f9418 in ?? () > #1 0x0000000800bf616f in qsort () from /lib/libc.so.7 > #2 0x000000080131b06e in _block_sync_core_bitmap (job_ptr=0x802033a08, > cr_type=1) at dist_tasks.c:489 > #3 0x000000080131c2cd in cr_dist (job_ptr=0x802033a08, cr_type=1) > at dist_tasks.c:896 > #4 0x00000008013223f3 in cr_job_test (job_ptr=0x802033a08, > bitmap=0x8020413e0, min_nodes=1, max_nodes=500000, req_nodes=1, > mode=0, > cr_type=1, job_node_req=NODE_CR_ONE_ROW, cr_node_cnt=1, > cr_part_ptr=0x800e0cd68, node_usage=0x800eac668, exc_core_bitmap=0x0) > at job_test.c:2530 > #5 0x0000000801316a42 in _run_now (job_ptr=0x802033a08, > bitmap=0x8020413e0, > min_nodes=1, max_nodes=500000, req_nodes=1, job_node_req=1, > preemptee_candidates=0x0, preemptee_job_list=0x7fffff6f9a90, > exc_core_bitmap=0x0) at select_cons_res.c:1498 > #6 0x0000000801317e63 in select_p_job_test (job_ptr=0x802033a08, > bitmap=0x8020413e0, min_nodes=1, max_nodes=500000, req_nodes=1, > mode=0, > preemptee_candidates=0x0, preemptee_job_list=0x7fffff6f9a90, > exc_core_bitmap=0x0) at select_cons_res.c:2002 > #7 0x00000000004cb6d3 in select_g_job_test () > #8 0x000000000046160f in deallocate_nodes () > #9 0x0000000000460b0d in deallocate_nodes () > #10 0x00000000004621cd in select_nodes () > ---Type<return> to continue, or q<return> to quit--- > #11 0x000000000043e1d8 in rehash_jobs () > #12 0x000000000043e66e in job_allocate () > #13 0x000000000046d439 in validate_operator () > #14 0x000000000046b978 in slurmctld_req () > #15 0x000000000042fec9 in main () > > On 03/28/13 20:33, Jason bacon wrote: >> I just ported slurm to FreeBSD, with very few issues. >> >> I've set up two simple test clusters, one on old P4 systems and one on >> AMD64 systems. >> >> I have not run into any issues on the P4 systems. >> >> On the AMD64 systems, everything seems to work fine except srun, which >> causes slurmctld to crash. >> Has anyone seen something like this before? >> >> Thanks, >> >> Jason >> >> Controller or compute node: >> >> <<<ROOT@porpoise>>> /home/bacon 850 # sinfo >> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST >> default-partitio* up infinite 1 idle porpoise >> <<<ROOT@porpoise>>> /home/bacon 851 # scontrol show partitions >> PartitionName=default-partition >> AllocNodes=ALL AllowGroups=ALL Default=YES >> DefaultTime=NONE DisableRootJobs=NO GraceTime=0 Hidden=NO >> MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 >> Nodes=porpoise >> Priority=1 RootOnly=NO ReqResv=NO Shared=NO PreemptMode=OFF >> State=UP TotalCPUs=2 TotalNodes=1 DefMemPerNode=UNLIMITED >> MaxMemPerNode=UNLIMITED >> >> <<<ROOT@porpoise>>> /home/bacon 852 # srun -n1 hostname >> srun: error: slurm_receive_msg: Zero Bytes were transmitted or received >> srun: error: Unable to allocate resources: Zero Bytes were transmitted >> or received >> >> Controller: >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "amd64-marcel-freebsd"...(no debugging >> symbols found)... >> (gdb) run -D >> Starting program: /usr/local/sbin/slurmctld -D >> [New LWP 100380] >> [New Thread 800e041c0 (LWP 100380/initial thread)] >> slurmctld: pidfile not locked, assuming no running daemon >> slurmctld: Accounting storage NOT INVOKED plugin loaded >> slurmctld: slurmctld version 2.5.4 started on cluster cluster >> slurmctld: Munge cryptographic signature plugin loaded >> slurmctld: Consumable Resources (CR) Node Selection plugin loaded with >> argument 1 >> slurmctld: preempt/none loaded >> slurmctld: Checkpoint plugin loaded: checkpoint/none >> slurmctld: Job accounting gather NOT_INVOKED plugin loaded >> slurmctld: debug: No backup controller to shutdown >> slurmctld: switch NONE plugin loaded >> slurmctld: debug: Reading slurm.conf file: /usr/local/etc/slurm.conf >> slurmctld: topology NONE plugin loaded >> slurmctld: debug: No DownNodes >> slurmctld: sched: Backfill scheduler plugin loaded >> [New Thread 800e77540 (LWP 100245/slurmctld)] >> slurmctld: Recovered state of 1 nodes >> slurmctld: Recovered information about 0 jobs >> slurmctld: cons_res: select_p_node_init >> slurmctld: cons_res: preparing for 1 partitions >> slurmctld: debug: Updating partition uid access list >> slurmctld: Recovered state of 0 reservations >> slurmctld: State of 0 triggers recovered >> slurmctld: read_slurm_conf: backup_controller not specified. >> slurmctld: cons_res: select_p_reconfigure >> slurmctld: cons_res: select_p_node_init >> slurmctld: cons_res: preparing for 1 partitions >> slurmctld: Running as primary controller >> slurmctld: debug: Priority BASIC plugin loaded >> [New Thread 800e77380 (LWP 100322/slurmctld)] >> [New Thread 800e771c0 (LWP 100361/slurmctld)] >> [New Thread 800e0ae40 (LWP 100397/slurmctld)] >> [New Thread 800e0ac80 (LWP 100414/slurmctld)] >> slurmctld: debug: power_save module disabled, SuspendTime< 0 >> [Thread 800e0ac80 (LWP 100414/slurmctld) exited] >> slurmctld: debug: Spawning registration agent for porpoise 1 hosts >> [New Thread 800e0aac0 (LWP 100414/slurmctld)] >> [New Thread 80201fc80 (LWP 100509/slurmctld)] >> [New Thread 80201fac0 (LWP 100524/slurmctld)] >> [New Thread 800eb9380 (LWP 100534/slurmctld)] >> slurmctld: auth plugin for Munge (http://code.google.com/p/munge/) loaded >> [New Thread 801c54c80 (LWP 100535/slurmctld)] >> [Thread 800eb9380 (LWP 100534/slurmctld) exited] >> [Thread 80201fac0 (LWP 100524/slurmctld) exited] >> slurmctld: debug: validate_node_specs: node porpoise registered with 0 jobs >> [Thread 801c54c80 (LWP 100535/slurmctld) exited] >> [Thread 80201fc80 (LWP 100509/slurmctld) exited] >> [Thread 800e0aac0 (LWP 100414/slurmctld) exited] >> [New Thread 800eb9380 (LWP 100414/slurmctld)] >> >> Program received signal SIGSEGV, Segmentation fault. >> [Switching to Thread 800eb9380 (LWP 100414/slurmctld)] >> 0x00007fffff3f5ee8 in ?? () >> (gdb) where >> #0 0x00007fffff3f5ee8 in ?? () >> #1 0x0000000800b9f16f in qsort () from /lib/libc.so.7 >> #2 0x000000080131876b in _block_sync_core_bitmap () >> from /usr/local/lib/slurm/select_cons_res.so >> #3 0x00000008013198b7 in cr_dist () >> from /usr/local/lib/slurm/select_cons_res.so >> #4 0x000000080131e76b in cr_job_test () >> from /usr/local/lib/slurm/select_cons_res.so >> #5 0x0000000801316c6d in select_p_job_test () >> from /usr/local/lib/slurm/select_cons_res.so >> #6 0x00000000004536f8 in list_find_feature () >> #7 0x00000000004540cf in list_find_feature () >> #8 0x0000000000456cd5 in select_nodes () >> #9 0x000000000043d80f in job_allocate () >> #10 0x000000000045c3a0 in validate_super_user () >> #11 0x000000000046475d in slurmctld_req () >> #12 0x000000000042faa5 in main () >> > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Jason W. Bacon [email protected] http://personalpages.tds.net/~jwbacon ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
