I think it's pretty clear that the large # of processes, each having open files, is what's causing the too many files message. So that's a red herring. If you have it, the truss output when the 1st Zombie is being created is the part of interest.
1. Search the back traffic for zombies. We've had this problem in the past, don't remember the solution. 2. It might be the SIGCHILD stuff - look in the code (maybe the behavior in FreeBSD 5 is different and the ifdef needs to be changed). Might be that we need to reap zombies where we didn't before or vice versa. -----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Tim Brigley Sent: Wednesday, November 12, 2003 9:00 AM To: [EMAIL PROTECTED] It Subject: [Ntop-dev] stability problem with -d in 2.2.95 on freebsd 5.1 Finally had time to look at this again, and i'm liking the minor html menu changes since 2.2 BTW. However the stability on freebsd 5.1 remains shakey. if i run it with ntop -i fxp0 -w3000 -W3001 -a /netflow/log/ntop.access.log -u ntop -P /netflow/db It works great, and seems to have solved my earlier stability problems with 2.2 on freebsd going zombie after a few hours. However if i add a -d to daemonize, it opens http ports, kicks of various threads and stuff like normal, seems to be working fine, but if you load the html it renders and delivers a bit of the html page, but it freezes up and never generates the full html page. After the browser times out. your left with a Error 501 The requested method is not implemented by this server. Received request: "DET /sortDataReceivedProtos.html HTTP/1.1" --- If i truss the process, i get the following output. poll(0x8647800,0x2,0x0) = 1 (0x1) accept(0x10,0xbfaa9e40,0xbfaa9d3c) ERR#24 'Too many open files' gettimeofday(0xbfaa93e8,0x0) = 0 (0x0) write(1,0x806b000,87) ERR#5 'Input/output error' poll(0x8647800,0x2,0x0) = 1 (0x1) accept(0x10,0xbfaa9e40,0xbfaa9d3c) ERR#24 'Too many open files' gettimeofday(0xbfaa93e8,0x0) = 0 (0x0) write(1,0x806b000,87) ERR#5 'Input/output error' poll(0x8647800,0x2,0x0) = 1 (0x1) accept(0x10,0xbfaa9e40,0xbfaa9d3c) ERR#24 'Too many open files' gettimeofday(0xbfaa93e8,0x0) = 0 (0x0) write(1,0x806b000,87) ERR#5 'Input/output error' lsof on the pid gives the following. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ntop 74263 ntop cwd VDIR 4,19 6656 259072 /usr/bin ntop 74263 ntop rtd VDIR 4,14 512 2 / ntop 74263 ntop txt VREG 4,19 217171 828107 /usr/local/bin/ntop ntop 74263 ntop txt VREG 4,19 109240 730155 /usr/libexec/ld-elf.so.1 ntop 74263 ntop txt VREG 4,19 1170302 829060 /usr/local/lib/libntopreport-2.2.95.so ntop 74263 ntop txt VREG 4,19 1603764 828071 /usr/local/lib/libntop-2.2.95.so ntop 74263 ntop txt VREG 4,19 112328 164935 /usr/lib/libc_r.so.5 ntop 74263 ntop txt VREG 4,19 886848 164931 /usr/lib/libc.so.5 ntop 74263 ntop txt VREG 4,19 32528 164876 /usr/lib/libcrypt.so.2 ntop 74263 ntop txt VREG 4,19 200928 168649 /usr/lib/libssl.so.3 ntop 74263 ntop txt VREG 4,19 1099348 168645 /usr/lib/libcrypto.so.3 ntop 74263 ntop txt VREG 4,19 113732 165000 /usr/lib/libpcap.so.2 ntop 74263 ntop txt VREG 4,19 24163 828020 /usr/local/lib/libgdbm.so.3 ntop 74263 ntop txt VREG 4,19 221784 827549 /usr/local/lib/libgd.so.4 ntop 74263 ntop txt VREG 4,19 149184 825895 /usr/local/lib/libpng.so.5 ntop 74263 ntop txt VREG 4,19 54712 165032 /usr/lib/libz.so.2 ntop 74263 ntop txt VREG 4,19 134321 827539 /usr/local/lib/libjpeg.so.9 ntop 74263 ntop txt VREG 4,19 402634 825879 /usr/local/lib/libfreetype.so.9 ntop 74263 ntop txt VREG 4,19 125840 164882 /usr/lib/libm.so.2 ntop 74263 ntop txt VREG 4,19 108796 828508 /usr/local/lib/ntop/plugins/icmpPlugin.so ntop 74263 ntop txt VREG 4,19 109963 828509 /usr/local/lib/ntop/plugins/lastSeenPlugin.so ntop 74263 ntop txt VREG 4,19 151853 828510 /usr/local/lib/ntop/plugins/netflowPlugin.so ntop 74263 ntop txt VREG 4,19 105699 828511 /usr/local/lib/ntop/plugins/nfsPlugin.so ntop 74263 ntop txt VREG 4,19 105407 828512 /usr/local/lib/ntop/plugins/pdaPlugin.so ntop 74263 ntop txt VREG 4,19 165000 828513 /usr/local/lib/ntop/plugins/sflowPlugin.so ntop 74263 ntop txt VREG 4,19 416906 828514 /usr/local/lib/ntop/plugins/rrdPlugin.so ntop 74263 ntop 0u VBAD (revoked) ntop 74263 ntop 1u VBAD (revoked) ntop 74263 ntop 2u VBAD (revoked) ntop 74263 ntop 3u PIPE 0xc25002c0 16384 ->0xc273b420 ntop 74263 ntop 4u PIPE 0xc273b420 16384 ->0xc25002c0 ntop 74263 ntop 5r VCHR 23,0 0x75a41b9 109 /dev/bpf0 ntop 74263 ntop 6uW VREG 4,21 49611 871425 /netflow/db/prefsCache.db ntop 74263 ntop 7uW VREG 4,21 49301 871426 /netflow/db/ntop_pw.db ntop 74263 ntop 8uW VREG 4,21 65531 871430 /netflow/db/addressQueue.db ntop 74263 ntop 9uW VREG 4,21 2212718 871428 /netflow/db/dnsCache.db ntop 74263 ntop 10uW VREG 4,21 1015808 871429 /netflow/db/macPrefix.db ntop 74263 ntop 11w VREG 4,21 66542 259073 /netflow/log/ntop.access.log ntop 74263 ntop 12uW VREG 4,21 49152 871431 /netflow/db/LsWatch.db ntop 74263 ntop 13u VREG 4,18 0 4 /tmp/ntop-000074263-0 ntop 74263 ntop 14u VREG 4,18 0 7 /tmp/ntop-000074263-1 ntop 74263 ntop 15u VREG 4,18 0 11 /tmp/ntop-000074263-2 ntop 74263 ntop 16u IPv4 0xc2804ca8 0t0 TCP *:3000 (LISTEN) ntop 74263 ntop 17u IPv4 0xc2804438 0t0 TCP *:3001 (LISTEN) ntop 74263 ntop 18u KQUEUE 0xc3882100 count=0, state=0 ntop 74263 ntop 19u KQUEUE 0xc3881600 count=0, state=0 ntop 74263 ntop 20u KQUEUE 0xc3878500 count=0, state=0 . . . ntop 74263 ntop 3564u KQUEUE 0xc3ae4500 count=0, state=0x1 ntop 74263 ntop 3565u KQUEUE 0xc3ae1400 count=0, state=0 lots of type KQUEUE open, but i admit i'm getting to the limit of my expertise here. $ lsof -p 74263 | grep KQUEUE |wc -l 3589 If i try to open the html gui, it causes the ntop process to chew up some cpu, but nothing happens, and the 501 error. USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND ntop 74263 97.6 6.7 26004 17144 p0- R 5:08PM 35:28.25 ntop -i fxp0 -w3000 -W3001 -a /netflow/log/ntop.access.log -u ntop Whats most interesting is that ntop appears to still be populating some of its databases, so its not completely gone zombie. -rw-r--r-- 1 ntop ntop 65531 Nov 12 09:46 addressQueue.db -rw-r--r-- 1 ntop ntop 2212718 Nov 12 09:47 dnsCache.db $ date Wed Nov 12 09:47:40 EST 2003 ntop-2.2.95 on FreeBSD 5.1-RELEASE-p10 #1: Wed Oct 15 10:46:17 EDT 2003 ./configure --with-tcp-wrappers Any details on resolving this would be appreciated. if you'd like any additional info don't hesitate to ask. -Tim P.S. i'm interested in following the most current source for ntop both for freebsd and openbsd, but i've had problems getting the source from CVS. i've used cvs before, but can't seem to find the proper info for the ntop cvs server. can someone forward that along for me? much abliged. --- Tim Brigley Senior System Administrator, Rogers Media 777 Jarvis Street, Toronto, ON, M4Y 3B7 mail: [EMAIL PROTECTED] or [EMAIL PROTECTED] phone: 416-935-8552 _______________________________________________ Ntop-dev mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop-dev _______________________________________________ Ntop-dev mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop-dev
