It's certainly acting like the flag isn't being defined. That's what I wanted you to check version.c... It needs to be a preprocessor flag, so that we #ifdef in the locking code.
traceEvent() is what calls syslog(). See how threads 1 and 8 are deadlocked? That's what the patch is supposed to (and does) fix. -----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rik Schneider Sent: Wednesday, March 30, 2005 5:50 PM To: [email protected] Subject: RE: [Ntop] Ntop 3.1.1 or cvs on RHEL2.1 The LINUXredhat8.0 had it listed as a preprocessor flag. I have changed it to a CFLAG and rebuilt it. I am testing now. Just turning on the "-W 3001" switch causes a hang in thread 8. Here is the stack trace for you. (gdb) thread 8 [Switching to thread 8 (Thread 49159 (LWP 24899))]#0 0x4135cbe5 in __sigsuspend (set=0x44b293dc) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0x44b293dc) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x44b2abe0) at pthread.c:1027 #2 0x412b3029 in __pthread_alt_lock (lock=0x40152c7c, self=0x0) at restart.h:34 #3 0x412afd36 in __pthread_mutex_lock (mutex=0x40152c6c) at mutex.c:120 #4 0x4011d835 in _accessMutex (mutexId=0x40152c6c, where=0x40149b5d "message", fileName=0x40149978 "util.c", fileLine=2639) at util.c:1806 #5 0x4011ef9e in traceEvent (eventTraceLevel=2, file=0x4014649f "leaks.c", line=609, format=0x401464c3 "free of NULL pointer @ %s:%d") at util.c:2639 #6 0x40101329 in ntop_safefree (ptr=0x44b2a1ac, file=0x400c0296 "http.c", line=3750) at leaks.c:609 #7 0x40064266 in handleHTTPrequest (from= {hostFamily = 2, addr = {_hostIp4Address = {s_addr = 2886869916}, _hostIp6Address = {in6_u = {u6_addr8 = "[EMAIL PROTECTED]@�83A", u6_addr16 = {9116, 44050, 5896, 16398, 13422, 16387, 14548, 16691}, u6_addr32 = {2886869916, 1074665224, 1073951854, 1093875924}}}}}) at http.c:3750 #8 0x40099e54 in handleSingleWebConnection (fdmask=0x44b2aa4c) at webInterface.c:9069 #9 0x40099aef in handleWebConnections (notUsed=0x0) at webInterface.c:8912 #10 0x412aec6f in pthread_start_thread (arg=0x44b2abe0) at manager.c:279 #11 0x4141bcea in thread_start () from /lib/i686/libc.so.6 A full trace of all of the threads follows. (gdb) thread 1 [Switching to thread 1 (Thread 8192 (LWP 24890))]#0 0x4135cbe5 in __sigsuspend (set=0xbffea210) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0xbffea210) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x412b6020) at pthread.c:1027 #2 0x412b3029 in __pthread_alt_lock (lock=0x40152c7c, self=0x0) at restart.h:34 #3 0x412afd36 in __pthread_mutex_lock (mutex=0x40152c6c) at mutex.c:120 #4 0x4011d835 in _accessMutex (mutexId=0x40152c6c, where=0x40149b5d "message", fileName=0x40149978 "util.c", fileLine=2639) at util.c:1806 #5 0x4011ef9e in traceEvent (eventTraceLevel=3, file=0x400c89df "report.c", line=108, format=0x400c15e0 "Note: Reporting device initally set to %d [%s]%s") at util.c:2639 #6 0x40064933 in initReports () at report.c:108 #7 0x0804ae2d in main () (gdb) thread 2 [Switching to thread 2 (Thread 16385 (LWP 24891))]#0 0x41414487 in __poll (fds=0x810572c, nfds=1, timeout=2000) at ../sysdeps/unix/sysv/linux/poll.c:63 63 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory. in ../sysdeps/unix/sysv/linux/poll.c (gdb) info stack #0 0x41414487 in __poll (fds=0x810572c, nfds=1, timeout=2000) at ../sysdeps/unix/sysv/linux/poll.c:63 #1 0x412ae920 in __pthread_manager (arg=0x0) at manager.c:135 #2 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 3 [Switching to thread 3 (Thread 8194 (LWP 24892))]#0 0x4135cbe5 in __sigsuspend (set=0x4232894c) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 ../sysdeps/unix/sysv/linux/sigsuspend.c: No such file or directory. in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0x4232894c) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x4232abe0) at pthread.c:1027 #2 0x412b230c in __new_sem_wait (sem=0x40152554) at restart.h:34 #3 0x4011df34 in waitSem (semId=0x40152554) at util.c:2061 #4 0x4010995d in dequeuePacket (notUsed=0x0) at pbuf.c:2346 #5 0x412aec6f in pthread_start_thread (arg=0x4232abe0) at manager.c:279 #6 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 4 [Switching to thread 4 (Thread 16387 (LWP 24893))]#0 0x413e9681 in __libc_nanosleep () from /lib/i686/libc.so.6 (gdb) info stack #0 0x413e9681 in __libc_nanosleep () from /lib/i686/libc.so.6 #1 0x413e95bb in __sleep (seconds=145) at ../sysdeps/unix/sysv/linux/sleep.c:70 #2 0x4012101d in ntop_sleep (secs=150) at util.c:3874 #3 0x401028ae in scanFingerprintLoop (notUsed=0x0) at ntop.c:703 #4 0x412aec6f in pthread_start_thread (arg=0x42b2abe0) at manager.c:279 #5 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 5 [Switching to thread 5 (Thread 24580 (LWP 24894))]#0 0x413e9681 in __libc_nanosleep () from /lib/i686/libc.so.6 (gdb) info stack #0 0x413e9681 in __libc_nanosleep () from /lib/i686/libc.so.6 #1 0x413e95bb in __sleep (seconds=60) at ../sysdeps/unix/sysv/linux/sleep.c:70 #2 0x4012101d in ntop_sleep (secs=60) at util.c:3874 #3 0x401026e4 in scanIdleLoop (notUsed=0x0) at ntop.c:654 #4 0x412aec6f in pthread_start_thread (arg=0x4332abe0) at manager.c:279 #5 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 6 [Switching to thread 6 (Thread 32773 (LWP 24895))]#0 0x4135cbe5 in __sigsuspend (set=0x43b2a95c) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0x43b2a95c) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x43b2abe0) at pthread.c:1027 #2 0x412b230c in __new_sem_wait (sem=0x40152564) at restart.h:34 #3 0x4011df34 in waitSem (semId=0x40152564) at util.c:2061 #4 0x400f3bf3 in dequeueAddress (_i=0x0) at address.c:617 #5 0x412aec6f in pthread_start_thread (arg=0x43b2abe0) at manager.c:279 #6 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 7 [Switching to thread 7 (Thread 40966 (LWP 24896))]#0 0x4135cbe5 in __sigsuspend (set=0x44328eec) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0x44328eec) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x4432abe0) at pthread.c:1027 #2 0x412b3029 in __pthread_alt_lock (lock=0x40152c7c, self=0x0) at restart.h:34 #3 0x412afd36 in __pthread_mutex_lock (mutex=0x40152c6c) at mutex.c:120 #4 0x4011d835 in _accessMutex (mutexId=0x40152c6c, where=0x40149b5d "message", fileName=0x40149978 "util.c", fileLine=2639) at util.c:1806 #5 0x4011ef9e in traceEvent (eventTraceLevel=3, file=0x40149978 "util.c", line=5756, format=0x4014bb40 "CHKVER: Version file is from '%s'") at util.c:2639 #6 0x4012401e in processVersionFile (buf=0x44329acc "HTTP/1.1 200 OK", bufLen=1572) at util.c:5756 #7 0x4012431e in checkVersion (notUsed=0x0) at util.c:5812 #8 0x412aec6f in pthread_start_thread (arg=0x4432abe0) at manager.c:279 #9 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 8 [Switching to thread 8 (Thread 49159 (LWP 24899))]#0 0x4135cbe5 in __sigsuspend (set=0x44b293dc) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0x44b293dc) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x44b2abe0) at pthread.c:1027 #2 0x412b3029 in __pthread_alt_lock (lock=0x40152c7c, self=0x0) at restart.h:34 #3 0x412afd36 in __pthread_mutex_lock (mutex=0x40152c6c) at mutex.c:120 #4 0x4011d835 in _accessMutex (mutexId=0x40152c6c, where=0x40149b5d "message", fileName=0x40149978 "util.c", fileLine=2639) at util.c:1806 #5 0x4011ef9e in traceEvent (eventTraceLevel=2, file=0x4014649f "leaks.c", line=609, format=0x401464c3 "free of NULL pointer @ %s:%d") at util.c:2639 #6 0x40101329 in ntop_safefree (ptr=0x44b2a1ac, file=0x400c0296 "http.c", line=3750) at leaks.c:609 #7 0x40064266 in handleHTTPrequest (from= {hostFamily = 2, addr = {_hostIp4Address = {s_addr = 2886869916}, _hostIp6Address = {in6_u = {u6_addr8 = "[EMAIL PROTECTED]@�83A", u6_addr16 = {9116, 44050, 5896, 16398, 13422, 16387, 14548, 16691}, u6_addr32 = {2886869916, 1074665224, 1073951854, 1093875924}}}}}) at http.c:3750 #8 0x40099e54 in handleSingleWebConnection (fdmask=0x44b2aa4c) at webInterface.c:9069 #9 0x40099aef in handleWebConnections (notUsed=0x0) at webInterface.c:8912 #10 0x412aec6f in pthread_start_thread (arg=0x44b2abe0) at manager.c:279 #11 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 9 [Switching to thread 9 (Thread 57352 (LWP 24900))]#0 0x413e9681 in __libc_nanosleep () from /lib/i686/libc.so.6 (gdb) info stack #0 0x413e9681 in __libc_nanosleep () from /lib/i686/libc.so.6 #1 0x413e95bb in __sleep (seconds=166) at ../sysdeps/unix/sysv/linux/sleep.c:70 #2 0x4012101d in ntop_sleep (secs=300) at util.c:3874 #3 0x44b50114 in rrdMainLoop (notUsed=0x0) at rrdPlugin.c:2753 #4 0x412aec6f in pthread_start_thread (arg=0x45370be0) at manager.c:279 #5 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 10 Thread ID 10 not known. Shutting it down and starting it with "/usr/bin/ntop --no-fc -u ntop -d -P /var/ntop -i eth1 -m 172.18.32.0/21 -w3000 -W 0 --disable-schedyield -t4" gives the following locked threads: (gdb) thread 7 [Switching to thread 7 (Thread 49159 (LWP 30654))]#0 0x4135cbe5 in __sigsuspend (set=0x44b28eec) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0x44b28eec) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x44b2abe0) at pthread.c:1027 #2 0x412b3029 in __pthread_alt_lock (lock=0x414672d4, self=0x0) at restart.h:34 #3 0x412afd36 in __pthread_mutex_lock (mutex=0x414672c4) at mutex.c:120 #4 0x413de8ee in __tz_convert (timer=0x44b290d8, use_localtime=1, tp=0x44b29a0c) at tzset.c:561 #5 0x413da981 in __localtime_r (t=0x44b290d8, tp=0x44b29a0c) at localtime.c:33 #6 0x4011eb87 in traceEvent (eventTraceLevel=4, file=0x400c0296 "http.c", line=3290, format=0x400c1138 "SECURITY: Loading items table") at util.c:2521 #7 0x400634e3 in checkHTTPpassword (theRequestedURL=0x44b2a4fc "/", theRequestedURLLen=512, thePw=0x44b2a4bc "", thePwLen=64) at http.c:3290 #8 0x4006412a in handleHTTPrequest (from= {hostFamily = 2, addr = {_hostIp4Address = {s_addr = 2886869916}, _hostIp6Address = {in6_u = {u6_addr8 = "[EMAIL PROTECTED]@�83A", u6_addr16 = {9116, 44050, 5896, 16398, 13422, 16387, 14548, 16691}, u6_addr32 = {2886869916, 1074665224, 1073951854, 1093875924}}}}}) at http.c:3710 #9 0x40099e54 in handleSingleWebConnection (fdmask=0x44b2aa4c) at webInterface.c:9069 #10 0x40099aef in handleWebConnections (notUsed=0x0) at webInterface.c:8912 #11 0x412aec6f in pthread_start_thread (arg=0x44b2abe0) at manager.c:279 #12 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 8 [Switching to thread 8 (Thread 57352 (LWP 30655))]#0 0x4135cbe5 in __sigsuspend (set=0x4536fefc) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0x4536fefc) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x45370be0) at pthread.c:1027 #2 0x412b3029 in __pthread_alt_lock (lock=0x414672d4, self=0x0) at restart.h:34 #3 0x412afd36 in __pthread_mutex_lock (mutex=0x414672c4) at mutex.c:120 #4 0x413de8ee in __tz_convert (timer=0x44b581a4, use_localtime=1, tp=0x453703fc) at tzset.c:561 #5 0x413da981 in __localtime_r (t=0x44b581a4, tp=0x453703fc) at localtime.c:33 #6 0x44b500f4 in rrdMainLoop (notUsed=0x0) at rrdPlugin.c:2746 #7 0x412aec6f in pthread_start_thread (arg=0x45370be0) at manager.c:279 #8 0x4141bcea in thread_start () from /lib/i686/libc.so.6 (gdb) thread 9 [Switching to thread 9 (Thread 65545 (LWP 30656))]#0 0x4135cbe5 in __sigsuspend (set=0x45b6ff0c) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 45 in ../sysdeps/unix/sysv/linux/sigsuspend.c (gdb) info stack #0 0x4135cbe5 in __sigsuspend (set=0x45b6ff0c) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45 #1 0x412b12b9 in __pthread_wait_for_restart_signal (self=0x45b70be0) at pthread.c:1027 #2 0x412b3029 in __pthread_alt_lock (lock=0x414672d4, self=0x0) at restart.h:34 #3 0x412afd36 in __pthread_mutex_lock (mutex=0x414672c4) at mutex.c:120 #4 0x413de8ee in __tz_convert (timer=0x45b700f8, use_localtime=1, tp=0x45b70a2c) at tzset.c:561 #5 0x413da981 in __localtime_r (t=0x45b700f8, tp=0x45b70a2c) at localtime.c:33 #6 0x4011eb87 in traceEvent (eventTraceLevel=3, file=0x401464f2 "ntop.c", line=87, format=0x401467c0 "THREADMGMT: pcapDispatch(%s) thread running [p%d, t%lu]...") at util.c:2521 #7 0x401015a5 in pcapDispatch (_i=0x0) at ntop.c:87 #8 0x412aec6f in pthread_start_thread (arg=0x45b70be0) at manager.c:279 #9 0x4141bcea in thread_start () from /lib/i686/libc.so.6 -- Rik Schneider Weathernews Americas Inc. (405) 310-2840 - Office (405) 310-2900 - Main (405) 388-1318 - Mobile [EMAIL PROTECTED] No trees were harmed during the creation of this message; however some electrons were inconvenienced. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Burton Strauss Sent: Wednesday, March 30, 2005 4:08 PM To: [email protected] Subject: RE: [Ntop] Ntop 3.1.1 or cvs on RHEL2.1 -t 4 causes a lot more messages, which makes it more likely to have multiple threads syslog()ing simultaneously which is what the fix is for. (RH7.2 is definitely pre NPTL and thus will be running an older glibc which isn't thread safe in syslog.) So: 1) Make sure that the -DFORPRENPTL really took - check version.c or in the About | Configuration page. Look for: Compile Time: ./configure and then the "compiler (CFLAGS)". 2) If it hangs up, be sure and walk through EACH thread and do the info stack, e.g. (gdb) thread 1 (gdb) info stack (gdb) thread 2 (gdb) info stack Etc. That may be the only way to see what was going on when it deadlocks. -----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rik Schneider Sent: Wednesday, March 30, 2005 2:53 PM To: [email protected] Subject: RE: [Ntop] Ntop 3.1.1 or cvs on RHEL2.1 It is recognized as redhat2.1. RHEL2.1 is based on RH 7.2 and uses pthreads. Using a source tree from cvs (CVS checkout at about 2245Z Mar 29, 2005) I created configureextra/LINUXredhat2.1 based on the contents of configureextra/LINUXredhat8.0. --- begin configureextra/LINUXredhat2.1 --- #!/bin/sh echo " Setting RHEL 2.1 specific flag values" CPPFLAGS="${CPPFLAGS} -DFORPRENPTL" --- end configureextra/LINUXredhat2.1 --- After building it I experimented with the -t option. If I give a value > 3 the web service does not work. I seem to have the same problem if I start the SSL server. If I use "--no-fc -u ntop -d -P /var/ntop -i eth1 -m 172.18.32.0/21 -w3000 -W 0 --disable-schedyield -t3" everything works fine for a while. I have it running now and will do a gdb traceback the next time I get it hung up. -- Rik Schneider Weathernews Americas Inc. (405) 310-2840 - Office (405) 310-2900 - Main (405) 388-1318 - Mobile [EMAIL PROTECTED] No trees were harmed during the creation of this message; however some electrons were inconvenienced. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Burton Strauss Sent: Monday, March 28, 2005 6:15 AM To: [email protected] Subject: RE: [Ntop] Ntop 3.1 or cvs on RHEL2.1 2.1 is probably pre NPTL and so you need the work-around for the syslog() hang. Figure out what the configureextra file should be and copy the lines from LINUXredhat8.0 (I've posted instructions on this within the last 2 weeks). -----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rik Schneider Sent: Friday, March 25, 2005 3:49 PM To: [email protected] Subject: [Ntop] Ntop 3.1 or cvs on RHEL2.1 I don't know if anyone has succeeded in this, I haven't. Here is what I have tried so far. Building from source using Red Hat's libpcap fails during configure with: ******************************************************************* * * ERROR: pcap header, bpf or library routines are missing * (yes means it was found, no means it was not found) * * pcap.h...yes * net/bpf.h...yes * pcap-bpf.h...no * pcap_open_live() in -lpcap...yes * pcap_findalldevs() in -lpcap...no * * (One of the bpf files is required, but either will do) * *>>> No way to proceed. * *??? 1. Install libpcap (http://www.tcpdump.org/) *??? and Rerun ./configure *??? or 2. Use the --with-pcap-root= option * ******************************************************************* I grabbed and installed a newer libpcap and tcpdump. I built them as RPMs. tcpdump sees the interface fine and gives lots of data. I them built ntop-3.1 against the new libpcap library and install it. I then edit the ntop.conf file setup the password and then start ntop. It starts up several threads. So far, so good. When I attempt to open the web interface at port 3000 it hung my browser (Firefox on RHEL4). I stopped all the treads and restarted everything by hand under strace. strace -ff -x -s 80 -o /tmp/ntop.strace /usr/bin/ntop -u ntop -d -P /var/ntop -i eth1 -m 172.18.32.0/21\ -t7 -w3000 -W 0 --disable-schedyield --use-syslog=local6 The web thread gets hung at rt_sigsuspend. The log entries for this thread are: 14:40:22 mon2 ntop[1462]: [MSGID0316203] [webInterface:8869] Note: SIGPIPE handler set (ignore) 14:40:22 mon2 ntop[1462]: [MSGID0986275] [webInterface:8890] WEB: ntop's web server is now processing requests) for web server The only thread that seems to increase in size appears to be a polling thread. I built the dev version from CVS with the same results. Am I running into a known issue or... _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
