I have never seen this before. Would yo do anything to remove or delete node records?
We use Ubuntu for most Slurm development and testing. We use valgrind to test for memory leaks, corruption, etc. I'm not sure how difficult this is to reproduce and valgrind will slow the slurmctld down a lot, but if you want to try it, an execute line like this should identify the problem: valgrind --tool=memcheck --leak-check=yes --num-callers=8 --leak-resolution=med ./slurmctld -Dc >valg.ctld.out 2>&1 The valgrind messages will intersperse with slurmctld messages in the file valg.ctld.out Quoting Chris Read <[email protected]>: > Greetings... > > We have had 2 similar segfaults on our controller recently, once with 2.5.1 > and again now with 2.5.6. Both blow out with an identical stack trace > (except for the value of the node record it's searching for). Here is the > latest stack: > > #0 0x00007f224b194e26 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x0000000000515e18 in find_node_record (name=0x7f2134699918 "c15") at > node_conf.c:822 > #2 0x000000000042d14e in _batch_launch_defer > (queued_req_ptr=0x7f213568d718) at agent.c:1610 > #3 0x000000000042e8da in agent_retry (min_wait=60, mail_too=true) at > agent.c:1210 > #4 0x000000000042f28b in agent (args=0x7f21372193b8) at agent.c:348 > #5 0x00007f224b429e9a in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #6 0x00007f224b156cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > #7 0x0000000000000000 in ?? () > > We are running on Ubuntu 12.04.1 > > Anyone seen this before? > > Chris >
