I'm not sure that's going to help me get to the root of this problem. I don't seem to be able to pinpoint freeswitch or just a freeswitch configuration issue, but surely do not have the proper information to open an informative JIRA issue yet. I do seem to see it occurring in less than 12 hours after a fresh reboot though. Thanks.
On Wed, Jan 20, 2010 at 6:44 PM, Eric Varsanyi <[email protected]> wrote: > FWIW we have this issue supporting our appliance product (which has nothing > to do with sipx or telephony), I wrote a little script that gets a stack > backtrace from every thread in the errant process. When its looping you can > run it a few times in a row and get a good feel for where the loop is. This > does not kill the process in question, but it does 'pause' it while you're > running the trace. > > Here's an example with a freeswitch that is burning the usual CPU (about 5% > of 1 CPU) while idle. I used 'top' with the 'H' option to show threads and > discovered these 4 active a lot: > > 20780 sipxchan 20 0 313m 24m 6384 S 0.2 0.1 2:14.41 freeswitch > > 20890 sipxchan 20 0 313m 24m 6384 S 0.2 0.1 1:34.97 freeswitch > > 20931 sipxchan 20 0 313m 24m 6384 S 0.2 0.1 1:41.75 freeswitch > > 20932 sipxchan 20 0 313m 24m 6384 S 0.2 0.1 1:43.66 freeswitch > > > If you look down below in the thread traces you can see they're sleeping > and in some sort of poll loop (I don't know the guts of freeswitch). If you > run this a few times you might catch a thread doing some real work that's > indicative of your problem. > > -Eric > > > 20555 ? Sl 9:00 /usr/local/freeswitch/bin/freeswitch -conf > /etc/sipxpbx/sip1/INSTALL/etc/sipxpbx/freeswitch/conf -db > /etc/sipxpbx/sip1/INSTALL/var/sipxdata/tmp/freeswitch -log > /etc/sipxpbx/sip1/INSTALL/var/log/sipxpbx -htdocs > > /etc/sipxpbx/sip1/INSTALL/etc/sipxpbx/freeswitch/conf/htdocs -nc -nf -nonat > TP:0 > d3 ~/wrkl/dbfx1/bin$ .su > d3 /wrkl/ev/dbfx1/bin# gdb /proc/20555/exe 20555 > GNU gdb Fedora (6.8-32.fc10) > ... lots of status messages about loading shared library symbols... > Missing separate debuginfos, use: debuginfo-install > alsa-lib-1.0.21-2.fc10.x86_64 cyrus-sasl-lib-2.1.22-19.fc10.x86_64 > e2fsprogs-libs-1.41.4-6.fc10.x86_64 glibc-2.9-3.x86_64 > keyutils-libs-1.2-3.fc9.x86_64 krb5-libs-1.6.3-18.fc10.x86_64 > libgcc-4.3.2-7.x86_64 libselinux-2.0.78-1.fc10.x86_64 > libstdc++-4.3.2-7.x86_64 ncurses-libs-5.6-20.20080927.fc10.x86_64 > openldap-2.4.12-1.fc10.x86_64 openssl-0.9.8g-14.fc10.x86_64 > postgresql-libs-8.3.8-2.fc10.x86_64 unixODBC-2.2.12-9.fc10.x86_64 > zlib-1.2.3-18.fc9.x86_64 > (gdb) thread apply all bt > > Thread 18 (Thread 0x7fbf90ac9950 (LWP 20577)): > #0 0x0000003016604a6f in clock_nanosleep () from /lib64/librt.so.1 > #1 0x00007fbf90c4b3ae in do_sleep (t=<value optimized out>) at > src/switch_time.c:147 > #2 0x00007fbf90bf04fa in pool_thread (thread=<value optimized out>, > obj=<value optimized out>) at src/switch_core_memory.c:530 > #3 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 17 (Thread 0x7fbf9083e950 (LWP 20581)): > #0 0x000000301520b309 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fbf90c5f1d1 in apr_queue_pop (queue=0x7fbf909025a8, > data=0x7fbf9083dfd0) at misc/apr_queue.c:276 > #2 0x00007fbf90c14866 in switch_event_dispatch_thread (thread=<value > optimized out>, obj=<value optimized out>) > at src/switch_event.c:252 > #3 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 16 (Thread 0x7fbf90802950 (LWP 20587)): > #0 0x000000301520b309 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fbf90c5f1d1 in apr_queue_pop (queue=0x10454d8, > data=0x7fbf90801fc0) at misc/apr_queue.c:276 > #2 0x00007fbf90c13bc4 in switch_event_thread (thread=<value optimized > out>, obj=<value optimized out>) at src/switch_event.c:295 > #3 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 15 (Thread 0x7fbf907c6950 (LWP 20588)): > #0 0x000000301520b309 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fbf90c5f1d1 in apr_queue_pop (queue=0x7fbf90a8c5a8, > data=0x7fbf907c5fc0) at misc/apr_queue.c:276 > #2 0x00007fbf90c13bc4 in switch_event_thread (thread=<value optimized > out>, obj=<value optimized out>) at src/switch_event.c:295 > #3 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 14 (Thread 0x7fbf9078a950 (LWP 20589)): > #0 0x000000301520b309 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fbf90c5f1d1 in apr_queue_pop (queue=0x7fbf909c75a8, > data=0x7fbf90789fc0) at misc/apr_queue.c:276 > #2 0x00007fbf90c13bc4 in switch_event_thread (thread=<value optimized > out>, obj=<value optimized out>) at src/switch_event.c:295 > #3 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 13 (Thread 0x7fbf90689950 (LWP 20697)): > #0 0x000000301520b309 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fbf90c5f1d1 in apr_queue_pop (queue=0x1054178, > data=0x7fbf90688fd8) at misc/apr_queue.c:276 > #2 0x00007fbf90c43271 in log_thread (t=<value optimized out>, obj=<value > optimized out>) at src/switch_log.c:288 > #3 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 12 (Thread 0x7fbf90611950 (LWP 20780)): > #0 0x0000003016604a6f in clock_nanosleep () from /lib64/librt.so.1 > #1 0x00007fbf90c4b3ae in do_sleep (t=<value optimized out>) at > src/switch_time.c:147 > #2 0x00007fbf90bf813d in switch_core_sql_thread (thread=<value optimized > out>, obj=<value optimized out>) > ---Type <return> to continue, or q <return> to quit--- > at src/switch_core_sqldb.c:881 > #3 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 11 (Thread 0x7fbf9064d950 (LWP 20784)): > #0 0x0000003014adeaa2 in select () from /lib64/libc.so.6 > #1 0x00007fbf90c6faa5 in apr_sleep (t=<value optimized out>) at > time/unix/time.c:246 > #2 0x00007fbf90c4b34e in do_sleep (t=0) at src/switch_time.c:138 > #3 0x00007fbf90c07d8a in switch_scheduler_task_thread (thread=<value > optimized out>, obj=<value optimized out>) > at src/switch_scheduler.c:171 > #4 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #5 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 10 (Thread 0x7fbf90081950 (LWP 20884)): > #0 0x0000003014ae6888 in epoll_wait () from /lib64/libc.so.6 > #1 0x00007fbf8add4a74 in su_epoll_port_wait_events (self=0x10f94e0, > tout=<value optimized out>) at su_epoll_port.c:495 > #2 0x00007fbf8add45a1 in su_base_port_step (self=0x10f94e0, tout=1000) at > su_base_port.c:467 > #3 0x00007fbf8ad1325a in sofia_profile_thread_run (thread=<value optimized > out>, obj=<value optimized out>) at sofia.c:1238 > #4 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #5 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 9 (Thread 0x7fbf90045950 (LWP 20888)): > #0 0x0000003014ae6888 in epoll_wait () from /lib64/libc.so.6 > #1 0x00007fbf8add4a74 in su_epoll_port_wait_events (self=0x10fab00, > tout=<value optimized out>) at su_epoll_port.c:495 > #2 0x00007fbf8add47ab in su_base_port_run (self=0x10fab00) at > su_base_port.c:349 > #3 0x00007fbf8add65f7 in su_pthread_port_clone_main (varg=<value optimized > out>) at su_pthread_port.c:324 > #4 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #5 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 8 (Thread 0x7fbf83fff950 (LWP 20890)): > #0 0x0000003016604a6f in clock_nanosleep () from /lib64/librt.so.1 > #1 0x00007fbf90c4b3ae in do_sleep (t=<value optimized out>) at > src/switch_time.c:147 > #2 0x00007fbf8ad13db9 in sofia_profile_worker_thread_run (thread=<value > optimized out>, obj=<value optimized out>) at sofia.c:1021 > #3 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 7 (Thread 0x7fbf8984d950 (LWP 20929)): > #0 0x0000003014adeaa2 in select () from /lib64/libc.so.6 > #1 0x00007fbf90c6faa5 in apr_sleep (t=<value optimized out>) at > time/unix/time.c:246 > #2 0x00007fbf90c4b34e in do_sleep (t=0) at src/switch_time.c:138 > #3 0x00007fbf8a004e3e in node_thread_run (thread=<value optimized out>, > obj=<value optimized out>) > at > /etc/sipxpbx/sip1/DEP_RPM_TOPDIR/BUILD/sipx-freeswitch-1.0.5/src/mod/applications/mod_fifo/mod_fifo.c:610 > #4 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #5 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 6 (Thread 0x7fbf881cb950 (LWP 20931)): > #0 0x0000003016604a6f in clock_nanosleep () from /lib64/librt.so.1 > #1 0x00007fbf90c4b3ae in do_sleep (t=<value optimized out>) at > src/switch_time.c:147 > ---Type <return> to continue, or q <return> to quit--- > #2 0x00007fbf90c4b6ca in timer_next (timer=0x7fbf881c9ce0) at > src/switch_time.c:477 > #3 0x00007fbf90beddb8 in switch_core_timer_next (timer=0x1) at > src/switch_core_timer.c:74 > #4 0x00007fbf82cfc582 in read_stream_thread (thread=<value optimized out>, > obj=0x115d9e8) > at > /etc/sipxpbx/sip1/DEP_RPM_TOPDIR/BUILD/sipx-freeswitch-1.0.5/src/mod/formats/mod_local_stream/mod_local_stream.c:226 > #5 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #6 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 5 (Thread 0x7fbf8818f950 (LWP 20932)): > #0 0x0000003016604a6f in clock_nanosleep () from /lib64/librt.so.1 > #1 0x00007fbf90c4b3ae in do_sleep (t=<value optimized out>) at > src/switch_time.c:147 > #2 0x00007fbf90c4b6ca in timer_next (timer=0x7fbf8818dce0) at > src/switch_time.c:477 > #3 0x00007fbf90beddb8 in switch_core_timer_next (timer=0x1) at > src/switch_core_timer.c:74 > #4 0x00007fbf82cfc582 in read_stream_thread (thread=<value optimized out>, > obj=0x1161a08) > at > /etc/sipxpbx/sip1/DEP_RPM_TOPDIR/BUILD/sipx-freeswitch-1.0.5/src/mod/formats/mod_local_stream/mod_local_stream.c:226 > #5 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #6 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 4 (Thread 0x7fbf88153950 (LWP 20937)): > #0 0x0000003016604a6f in clock_nanosleep () from /lib64/librt.so.1 > #1 0x00007fbf90c4b3ae in do_sleep (t=<value optimized out>) at > src/switch_time.c:147 > #2 0x00007fbf90c4ce7c in softtimer_runtime () at src/switch_time.c:612 > #3 0x00007fbf90c0dc83 in switch_loadable_module_exec (thread=0x1, obj=0x0) > at src/switch_loadable_module.c:95 > #4 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #5 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 3 (Thread 0x7fbf88117950 (LWP 20938)): > #0 0x000000301520e1cb in accept () from /lib64/libpthread.so.0 > #1 0x00007fbf90c6d104 in apr_socket_accept (new=0x7fbf88116f68, > sock=0x119de30, connection_context=0x11734c8) > at network_io/unix/sockets.c:187 > #2 0x00007fbf8b795dd7 in mod_event_socket_runtime () > at > /etc/sipxpbx/sip1/DEP_RPM_TOPDIR/BUILD/sipx-freeswitch-1.0.5/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:2349 > #3 0x00007fbf90c0dc83 in switch_loadable_module_exec (thread=0x19, > obj=0x1173760) at src/switch_loadable_module.c:95 > #4 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #5 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 2 (Thread 0x7fbf880db950 (LWP 20939)): > #0 0x0000003014adc886 in poll () from /lib64/libc.so.6 > #1 0x00007fbf8b9b16e4 in waitForConnection () at > ../../../../libs/xmlrpc-c/lib/abyss/src/socket_unix.c:635 > #2 chanSwitchAccept (chanSwitchP=<value optimized out>, > channelPP=0x7fbf880dacf0, channelInfoPP=0x7fbf880dace8, > errorP=0x7fbf880dacf8) > at ../../../../libs/xmlrpc-c/lib/abyss/src/socket_unix.c:729 > #3 0x00007fbf8b9a4d6d in ChanSwitchAccept (chanSwitchP=0x7fbf84009740, > channelPP=0x7fbf880dacf0, channelInfoPP=0x7fbf880dace8, > errorP=0x7fbf880dacf8) at > ../../../../libs/xmlrpc-c/lib/abyss/src/chanswitch.c:149 > #4 0x00007fbf8b9b000b in acceptAndProcessNextConnection () at > ../../../../libs/xmlrpc-c/lib/abyss/src/server.c:908 > #5 serverRun2 () at ../../../../libs/xmlrpc-c/lib/abyss/src/server.c:956 > #6 ServerRun (serverP=0x7fbf8bbee6b0) at > ../../../../libs/xmlrpc-c/lib/abyss/src/server.c:975 > #7 0x00007fbf8b9a21f7 in mod_xml_rpc_runtime () > at > /etc/sipxpbx/sip1/DEP_RPM_TOPDIR/BUILD/sipx-freeswitch-1.0.5/src/mod/xml_int/mod_xml_rpc/mod_xml_rpc.c:959 > #8 0x00007fbf90c0dc83 in switch_loadable_module_exec > (thread=0x7fbf880dabd0, obj=0x2) at src/switch_loadable_module.c:95 > ---Type <return> to continue, or q <return> to quit--- > #9 0x00000030152073da in start_thread () from /lib64/libpthread.so.0 > #10 0x0000003014ae62bd in clone () from /lib64/libc.so.6 > > Thread 1 (Thread 0x7fbf90b91810 (LWP 20555)): > #0 0x0000003014adeaa2 in select () from /lib64/libc.so.6 > #1 0x00007fbf90c6faa5 in apr_sleep (t=<value optimized out>) at > time/unix/time.c:246 > #2 0x00007fbf90c4b34e in do_sleep (t=0) at src/switch_time.c:138 > #3 0x00007fbf90c0403a in switch_core_runtime_loop (bg=<value optimized > out>) at src/switch_core.c:695 > #4 0x00000000004027da in main () > (gdb) quit > The program is running. Quit anyway (and detach it)? (y or n) y > Detaching from program: /proc/20555/exe, process 20555 > d3 /wrkl/ev/dbfx1/bin# > > -Eric > > On Jan 20, 2010, at 2:23 PM, Tony Graziano wrote: > > OK, that told me something. I see that the command I used earlier to > generate the coredump from the PID actually put the core file in > /var/log/sipxpbx by default, but it is not included in the snapshot that I > took. > > So that independent command did work, but is the core file helpful even > though the snapshot was taken just after the core file was generated? > > I looked at the parameters for sipx-snapshot and can tell there is some > core functionality, but it is not built into sipxconfig in 4.0.4. > > I see in 4.0 there is an option from the cli to specify "-core". Is the > core file only useful if generated with sipx-snapshot? > > Am I correct in assuming sipx-snapshot does not do a > On Wed, Jan 20, 2010 at 3:02 PM, Dale Worley <[email protected]> wrote: > >> On Wed, 2010-01-20 at 14:49 -0500, Tony Graziano wrote: >> > OK, I saw it again today using 99.9 CPU, I ran >> > >> > kill -s SIGSEGV <pid> >> > >> > I then created and downloaded a snapshot file . >> > >> > I did not see the core file. I expected a "core" file to be generated >> > from my working directory and I did not observe it. Is there something >> > extra to an ISO install to lace in a config file to get the coredump >> > file to be created? >> >> I think we looked over this problem, and there is no way to force a >> process to terminate and generate a core file from the outside. >> >> In recent versions, "sipx-snapshot --generate-core-files" will generate >> simulated core files of all the sipXecs processes and include them in >> the snapshot. >> >> Dale >> >> >> > > > -- > ====================== > Tony Graziano, Manager > Telephone: 434.984.8430 > Fax: 434.984.8431 > > Email: [email protected] > > LAN/Telephony/Security and Control Systems Helpdesk: > Telephone: 434.984.8426 > Fax: 434.984.8427 > > Helpdesk Contract Customers: > http://www.myitdepartment.net/gethelp/ > > Why do mathematicians always confuse Halloween and Christmas? > Because 31 Oct = 25 Dec. > > _______________________________________________ > sipx-dev mailing list [email protected] > List Archive: http://list.sipfoundry.org/archive/sipx-dev > Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev > sipXecs IP PBX -- http://www.sipfoundry.org/ > > > -- ====================== Tony Graziano, Manager Telephone: 434.984.8430 Fax: 434.984.8431 Email: [email protected] LAN/Telephony/Security and Control Systems Helpdesk: Telephone: 434.984.8426 Fax: 434.984.8427 Helpdesk Contract Customers: http://www.myitdepartment.net/gethelp/ Why do mathematicians always confuse Halloween and Christmas? Because 31 Oct = 25 Dec.
_______________________________________________ sipx-dev mailing list [email protected] List Archive: http://list.sipfoundry.org/archive/sipx-dev Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev sipXecs IP PBX -- http://www.sipfoundry.org/
