On Thu, May 16, 2013 at 2:43 PM, Moe Jette <[email protected]> wrote:
> > I have never seen this before. Would yo do anything to remove or > delete node records? > No. Last node change was done a week before, removed it from the config file and restarted slurmctld. We use Ubuntu for most Slurm development and testing. We use valgrind > to test for memory leaks, corruption, etc. I'm not sure how difficult > this is to reproduce and valgrind will slow the slurmctld down a lot, > but if you want to try it, an execute line like this should identify > the problem: > valgrind --tool=memcheck --leak-check=yes --num-callers=8 > --leak-resolution=med ./slurmctld -Dc >valg.ctld.out 2>&1 > > This problem is not common, but we have seen a few rare cases that appear to imply there is a thread safety issue going on somewhere. We've not been able to collect enough data yet on the problem though. I've run valgrind on the controller in our test environment (identical everything except size and load), here are the interesting bits after running 1000 short jobs and doing some common queries we use: ==17779== Memcheck, a memory error detector ==17779== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. ==17779== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info ==17779== Command: /usr/sbin/slurmctld -Dc ==17779== ==17779== Thread 24: ==17779== Use of uninitialised value of size 8 ==17779== at 0x54C17EB: _itoa_word (_itoa.c:195) ==17779== by 0x54C3837: vfprintf (vfprintf.c:1629) ==17779== by 0x54EA813: vsprintf (iovsprintf.c:43) ==17779== by 0x54CCA06: sprintf (sprintf.c:34) ==17779== by 0x557B463: inet_ntop (inet_ntop.c:99) ==17779== by 0x4D2073: _slurm_print_slurm_addr (slurm_protocol_socket_implementation.c:912) ==17779== by 0x4B7F0E: slurm_send_node_msg (slurm_protocol_api.c:2830) ==17779== by 0x4B9480: slurm_send_only_node_msg (slurm_protocol_api.c:3439) ==17779== ==17779== Conditional jump or move depends on uninitialised value(s) ==17779== at 0x54C17F5: _itoa_word (_itoa.c:195) ==17779== by 0x54C3837: vfprintf (vfprintf.c:1629) ==17779== by 0x54EA813: vsprintf (iovsprintf.c:43) ==17779== by 0x54CCA06: sprintf (sprintf.c:34) ==17779== by 0x557B463: inet_ntop (inet_ntop.c:99) ==17779== by 0x4D2073: _slurm_print_slurm_addr (slurm_protocol_socket_implementation.c:912) ==17779== by 0x4B7F0E: slurm_send_node_msg (slurm_protocol_api.c:2830) ==17779== by 0x4B9480: slurm_send_only_node_msg (slurm_protocol_api.c:3439) ==17779== ==17779== Conditional jump or move depends on uninitialised value(s) ==17779== at 0x54C34F1: vfprintf (vfprintf.c:1629) ==17779== by 0x5581D8F: __vsnprintf_chk (vsnprintf_chk.c:65) ==17779== by 0x5581CD7: __snprintf_chk (snprintf_chk.c:36) ==17779== by 0x4D20A8: _slurm_print_slurm_addr (stdio2.h:65) ==17779== by 0x4B7F0E: slurm_send_node_msg (slurm_protocol_api.c:2830) ==17779== by 0x4B9480: slurm_send_only_node_msg (slurm_protocol_api.c:3439) ==17779== by 0x42D762: _thread_per_group_rpc (agent.c:887) ==17779== by 0x5263E99: start_thread (pthread_create.c:308) ==17779== ==17779== Use of uninitialised value of size 8 ==17779== at 0x54C17EB: _itoa_word (_itoa.c:195) ==17779== by 0x54C3837: vfprintf (vfprintf.c:1629) ==17779== by 0x5581D8F: __vsnprintf_chk (vsnprintf_chk.c:65) ==17779== by 0x5581CD7: __snprintf_chk (snprintf_chk.c:36) ==17779== by 0x4D20A8: _slurm_print_slurm_addr (stdio2.h:65) ==17779== by 0x4B7F0E: slurm_send_node_msg (slurm_protocol_api.c:2830) ==17779== by 0x4B9480: slurm_send_only_node_msg (slurm_protocol_api.c:3439) ==17779== by 0x42D762: _thread_per_group_rpc (agent.c:887) ==17779== ==17779== Conditional jump or move depends on uninitialised value(s) ==17779== at 0x54C17F5: _itoa_word (_itoa.c:195) ==17779== by 0x54C3837: vfprintf (vfprintf.c:1629) ==17779== by 0x5581D8F: __vsnprintf_chk (vsnprintf_chk.c:65) ==17779== by 0x5581CD7: __snprintf_chk (snprintf_chk.c:36) ==17779== by 0x4D20A8: _slurm_print_slurm_addr (stdio2.h:65) ==17779== by 0x4B7F0E: slurm_send_node_msg (slurm_protocol_api.c:2830) ==17779== by 0x4B9480: slurm_send_only_node_msg (slurm_protocol_api.c:3439) ==17779== by 0x42D762: _thread_per_group_rpc (agent.c:887) ==17779== ==17779== Thread 12: ==17779== Syscall param socketcall.sendto(msg) points to uninitialised byte(s) ==17779== at 0x526B2CC: send (send.c:33) ==17779== by 0x4D1B1A: _slurm_send_timeout (slurm_protocol_socket_implementation.c:310) ==17779== by 0x4D1D75: _slurm_msg_sendto_timeout (slurm_protocol_socket_implementation.c:232) ==17779== by 0x4B7E5F: slurm_send_node_msg (slurm_protocol_api.c:2818) ==17779== by 0x46535C: slurmctld_req (proc_req.c:1058) ==17779== by 0x43135F: _service_connection (controller.c:1022) ==17779== by 0x5263E99: start_thread (pthread_create.c:308) ==17779== Address 0xb15b081 is 11,281 bytes inside a block of size 16,392 alloc'd ==17779== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==17779== by 0x48F25C: slurm_xmalloc (xmalloc.c:94) ==17779== by 0x49C982: init_buf (pack.c:158) ==17779== by 0x4B7DEA: slurm_send_node_msg (slurm_protocol_api.c:2792) ==17779== by 0x46535C: slurmctld_req (proc_req.c:1058) ==17779== by 0x43135F: _service_connection (controller.c:1022) ==17779== by 0x5263E99: start_thread (pthread_create.c:308) ==17779== ==17779== Thread 2: ==17779== Invalid read of size 1 ==17779== at 0x4C2BFA2: strlen (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==17779== by 0x4F12F1: slurmdbd_pack_job_start_msg (slurmdbd_defs.c:3185) ==17779== by 0x4F427B: slurmdbd_pack_list_msg (slurmdbd_defs.c:3447) ==17779== by 0x4F5C0A: pack_slurmdbd_msg (slurmdbd_defs.c:531) ==17779== by 0x4F7C46: slurm_send_recv_slurmdbd_msg (slurmdbd_defs.c:326) ==17779== by 0x6ADDCEC: _set_db_inx_thread (accounting_storage_slurmdbd.c:306) ==17779== by 0x5263E99: start_thread (pthread_create.c:308) ==17779== Address 0xa2e84c8 is 8 bytes inside a block of size 9 free'd ==17779== at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==17779== by 0x48F53B: slurm_xfree (xmalloc.c:270) ==17779== by 0x454F9D: _fill_in_gres_fields (node_scheduler.c:1721) ==17779== by 0x45775D: select_nodes (node_scheduler.c:1683) ==17779== by 0x44CF85: schedule (job_scheduler.c:952) ==17779== by 0x461277: slurmctld_req (proc_req.c:2715) ==17779== by 0x43135F: _service_connection (controller.c:1022) ==17779== by 0x5263E99: start_thread (pthread_create.c:308) ==17779== ==17779== Invalid read of size 1 ==17779== at 0x4C2D1A0: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==17779== by 0x49D116: packmem (string3.h:52) ==17779== by 0x4F12FF: slurmdbd_pack_job_start_msg (slurmdbd_defs.c:3185) ==17779== by 0x4F427B: slurmdbd_pack_list_msg (slurmdbd_defs.c:3447) ==17779== by 0x4F5C0A: pack_slurmdbd_msg (slurmdbd_defs.c:531) ==17779== by 0x4F7C46: slurm_send_recv_slurmdbd_msg (slurmdbd_defs.c:326) ==17779== by 0x6ADDCEC: _set_db_inx_thread (accounting_storage_slurmdbd.c:306) ==17779== by 0x5263E99: start_thread (pthread_create.c:308) ==17779== Address 0xa2e84c8 is 8 bytes inside a block of size 9 free'd ==17779== at 0x4C2A82E: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==17779== by 0x48F53B: slurm_xfree (xmalloc.c:270) ==17779== by 0x454F9D: _fill_in_gres_fields (node_scheduler.c:1721) ==17779== by 0x45775D: select_nodes (node_scheduler.c:1683) ==17779== by 0x44CF85: schedule (job_scheduler.c:952) ==17779== by 0x461277: slurmctld_req (proc_req.c:2715) ==17779== by 0x43135F: _service_connection (controller.c:1022) ==17779== by 0x5263E99: start_thread (pthread_create.c:308) ==17779== So there are some issues to be looked at... Chris The valgrind messages will intersperse with slurmctld messages in the > file valg.ctld.out > > > Quoting Chris Read <[email protected]>: > > > Greetings... > > > > We have had 2 similar segfaults on our controller recently, once with > 2.5.1 > > and again now with 2.5.6. Both blow out with an identical stack trace > > (except for the value of the node record it's searching for). Here is the > > latest stack: > > > > #0 0x00007f224b194e26 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > > #1 0x0000000000515e18 in find_node_record (name=0x7f2134699918 "c15") at > > node_conf.c:822 > > #2 0x000000000042d14e in _batch_launch_defer > > (queued_req_ptr=0x7f213568d718) at agent.c:1610 > > #3 0x000000000042e8da in agent_retry (min_wait=60, mail_too=true) at > > agent.c:1210 > > #4 0x000000000042f28b in agent (args=0x7f21372193b8) at agent.c:348 > > #5 0x00007f224b429e9a in start_thread () from > > /lib/x86_64-linux-gnu/libpthread.so.0 > > #6 0x00007f224b156cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > #7 0x0000000000000000 in ?? () > > > > We are running on Ubuntu 12.04.1 > > > > Anyone seen this before? > > > > Chris > > > >
