:D Getting somewhere now. I think basically attach gdb to the process and then run 'info threads' and switch to the second thread.
:- Ok the below output was run whilst dbmail-pop3d was actually in an OK state but does the below actually look more like what is needed? [root@mail4-db-4a ~]# gdb762 GNU gdb (GDB) 7.6.2 [GDB v7.6.2 for FreeBSD] Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd9.2". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. (gdb) attach 75963 Attaching to process 75963 Reading symbols from /usr/local/sbin/dbmail-pop3d...done. Reading symbols from /usr/local/lib/dbmail/libdbmail.so.0...done. [etc etc etc] (gdb) info threads Id Target Id Frame 2 Thread 805007400 (LWP 100907) 0x00000008041d3d8c in kevent () from /lib/libc.so.7 * 1 Thread 80500a400 (LWP 102174) 0x0000000803ed64dc in ?? () from /lib/libthr.so.3 (gdb) thread 2 [Switching to thread 2 (Thread 805007400 (LWP 100907))] (gdb) bt full #0 0x00000008041d3d8c in kevent () from /lib/libc.so.7 No symbol table info available. #1 0x000000080297247c in kq_dispatch () from /usr/local/lib/event2/libevent-2.0.so.6 No symbol table info available. #2 0x00000008029618b3 in event_base_loop () from /usr/local/lib/event2/libevent-2.0.so.6 No symbol table info available. #3 0x00000008008628a1 in server_run (conf=0x7fffffffa310) at server.c:842 i = 1 __func__ = "server_run" #4 0x0000000800862d6d in server_mainloop (config=0x7fffffffa310, service=0x403b3a "POP", servicename=<optimized out>) at server.c:976 __func__ = "server_mainloop" #5 0x0000000000403a37 in main (argc=1, argv=<optimized out>) at pop3d.c:52 config = {no_daemonize = 0, log_verbose = 0, pidFile = 0x805017d80 "/var/run/dbmail/dbmail-pop3d.pid", timeout = 120, login_timeout = 30, iplist = 0x8050dd920, port = "110", '\000' <repeats 1020 times>, ssl_port = '\000' <repeats 1023 times>, ipcount = 1, socketcount = 1, ssl_socketcount = 0, listenSockets = 0x80500a800, ssl_listenSockets = 0x80500ac00, service_before_smtp = 0, authlog = 0, ssl = 0, backlog = 512, resolveIP = 0, evh = 0x0, service_name = "POP", '\000' <repeats 1020 times>, process_name = "dbmail-pop3d", '\000' <repeats 1011 times>, serverUser = "nobody", '\000' <repeats 1017 times>, serverGroup = "nogroup", '\000' <repeats 1016 times>, socket = '\000' <repeats 1023 times>, log = "/var/log/dbmail.log", '\000' <repeats 1004 times>, error_log = "/var/log/dbmail.err", '\000' <repeats 1004 times>, pid_dir = "/var/run/dbmail", '\000' <repeats 1008 times>, tls_cafile = '\000' <repeats 1023 times>, tls_cert = '\000' <repeats 1023 times>, tls_key = '\000' <repeats 1023 times>, tls_ciphers = '\000' <repeats 1023 times>, ClientHandler = 0x402060 <pop3_handle_connection>, cb = 0x0} result = 0 __func__ = "main" (gdb) bt full #0 0x00000008041d3d8c in kevent () from /lib/libc.so.7 No symbol table info available. #1 0x000000080297247c in kq_dispatch () from /usr/local/lib/event2/libevent-2.0.so.6 No symbol table info available. #2 0x00000008029618b3 in event_base_loop () from /usr/local/lib/event2/libevent-2.0.so.6 No symbol table info available. #3 0x00000008008628a1 in server_run (conf=0x7fffffffa310) at server.c:842 i = 1 __func__ = "server_run" #4 0x0000000800862d6d in server_mainloop (config=0x7fffffffa310, service=0x403b3a "POP", servicename=<optimized out>) at server.c:976 __func__ = "server_mainloop" #5 0x0000000000403a37 in main (argc=1, argv=<optimized out>) at pop3d.c:52 config = {no_daemonize = 0, log_verbose = 0, pidFile = 0x805017d80 "/var/run/dbmail/dbmail-pop3d.pid", timeout = 120, login_timeout = 30, iplist = 0x8050dd920, port = "110", '\000' <repeats 1020 times>, ssl_port = '\000' <repeats 1023 times>, ipcount = 1, socketcount = 1, ssl_socketcount = 0, listenSockets = 0x80500a800, ssl_listenSockets = 0x80500ac00, service_before_smtp = 0, authlog = 0, ssl = 0, backlog = 512, resolveIP = 0, evh = 0x0, service_name = "POP", '\000' <repeats 1020 times>, process_name = "dbmail-pop3d", '\000' <repeats 1011 times>, serverUser = "nobody", '\000' <repeats 1017 times>, serverGroup = "nogroup", '\000' <repeats 1016 times>, socket = '\000' <repeats 1023 times>, log = "/var/log/dbmail.log", '\000' <repeats 1004 times>, error_log = "/var/log/dbmail.err", '\000' <repeats 1004 times>, pid_dir = "/var/run/dbmail", '\000' <repeats 1008 times>, tls_cafile = '\000' <repeats 1023 times>, tls_cert = '\000' <repeats 1023 times>, tls_key = '\000' <repeats 1023 times>, tls_ciphers = '\000' <repeats 1023 times>, ClientHandler = 0x402060 <pop3_handle_connection>, cb = 0x0} result = 0 __func__ = "main" (gdb) Best regards, James Greig -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Reindl Harald Sent: 16 March 2014 14:44 To: [email protected] Subject: Re: [Dbmail] Iphone HTML emails well, and now it get's interesting if GDB likes that ID two choices - it accepts it as -p (PID) or the second guess maybe the manpage or --help output talks about thread-ID's on your system tmpfile=`mktemp` echo 'bt' > $tmpfile gdb --batch -x $tmpfile -p 101826 > dbmail-debug-backtrace.txt rm -f $tmpfile Am 16.03.2014 15:39, schrieb James Greig: > The joy :) > > Well seems you can get a 'thread ID' using procstat but that's as far as I've > got thus far. > > [root@mail4-db-4a ~]# !ps > ps Hlauxc -p `pidof dbmail-pop3d` > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > USER %CPU %MEM STARTED > 65534 75130 1 0 103 0 139116 49136 - R ?? 24:54.17 > dbmail-pop3d nobody 100.0 0.2 1:59PM > 65534 75130 1 0 20 0 139116 49136 uwait S ?? 0:00.00 > dbmail-pop3d nobody 0.0 0.2 1:59PM > > [root@mail4-db-4a ~]# procstat -t 75130 > PID TID COMM TDNAME CPU PRI STATE WCHAN > 75130 101826 dbmail-pop3d - 10 203 run - > 75130 102171 dbmail-pop3d - 7 120 sleep uwait > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of Reindl Harald > Sent: 16 March 2014 14:36 > To: [email protected] > Subject: Re: [Dbmail] Iphone HTML emails > > > Am 16.03.2014 15:21, schrieb Paul J Stevens: >> On 16-03-14 15:02, James Greig wrote: >>> Completely appreciate that by me just saying it "won't work" isn't >>> helpful :) >>> >>> Ok. Top in freebsd doesn't have -n1 and -p so the next thing I did >>> was move to just using ps >>> >>> [root@mail4-db-4a ~]# ps Hlauxc -p `pidof dbmail-pop3d` >>> UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND >>> USER %CPU %MEM STARTED >>> 65534 74473 1 0 103 0 139116 57488 - R ?? 1:33.86 >>> dbmail-pop3d nobody 100.0 0.2 1:17PM >>> 65534 74473 1 0 20 0 139116 57488 uwait I ?? 0:00.00 >>> dbmail-pop3d nobody 0.0 0.2 1:17PM >>> >>> As you can see that's the output and you can also see 100.0% CPU at the >>> moment. >>> >>> Am I just awk'in out the normal PID there? >> >> The idea is to attach the gdb to the busy thread/pid, generate a bt >> and get an idea of where in the code the thread is looping. >> >> On linux each thread/fork gets it's own PID, but apparently on FBSD >> things are different. >> >> I just generalised the backtrace script to work for both imapd and >> other processes. >> >> By default it will look for busy threads on imapd, but you can tweak >> the >> behavior: >> >> dumpbt.sh dbmail-pop3d 60 >> >> will look for pop3d processes eating up more than 60% cpu and >> generate a bt for them >> >> Alas it only works for linux. >> >> http://git.dbmail.eu/paul/dbmail/tree/test-scripts/dumpbt.sh >> >> If Harald suffers from the same problem on linux, maybe he can run >> this and get a handle on the problem > > hah i generalized mine last night too :-) > > i fear the timeouts i showed recently with pop3d growing over a longer > run are a different problem, AFAIk i never seen dbmail-pop3d consume > that much CPU, no idea what that is, why that grows over time and how > top debug that :-( > > [root@mail:~]$ cat /Volumes/dune/debug/debug.sh #!/usr/bin/bash if [ "$1" != > "" ]; then PROTOCOL="$1" > else > PROTOCOL="imap" > fi > DAEMON="dbmail-${PROTOCOL}d" > cd /Volumes/dune/debug/ > rm -f dbmail-debug-*.txt > top -H -b -n1 -p `pidof $DAEMON` > dbmail-debug-top.txt > MAXCPU=20 > # get the PID > PID=`pidof $DAEMON` > # find busy threads > BUSY=`top -H -b -n1 -p $PID|grep 'pool'|awk '{if ($9 > $MAXCPU) print > $1}'|head -1` if [ -n "$BUSY" ]; then # generate a gdb commandfile > tmpfile=`mktemp` echo 'bt' > $tmpfile # get a backtrace gdb --batch -x > $tmpfile -p $BUSY > dbmail-debug-backtrace.$BUSY.txt rm -f $tmpfile fi cat > dbmail-debug-top.txt /usr/bin/ls -l -h --color=tty -X > --group-directories-first --time-style=long-iso _______________________________________________ DBmail mailing list [email protected] http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
