Hello everybody, I had my nfsen installation dumping files on a 246G partition and I had set the expire threshold for the live profile to 220G. I had 2 other profiles, each expiring at 1G. There was no time limit for expiration.
This morning I had the unpleasant surprise to see that the whole partition was 100% full, and nfsen was in agony... Here are some snipletts from the log file: ... Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Process_v9: output buffer size error. Abort v9 record processing Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Failed to write output buffer to disk: 'No space left on device' Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Process_v9: output buffer size error. Abort v9 record processing Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Failed to write output buffer to disk: 'No space left on device' Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Process_v9: output buffer size error. Abort v9 record processing Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Failed to write output buffer to disk: 'No space left on device' Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Process_v9: output buffer size error. Abort v9 record processing Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Failed to write output buffer to disk: 'No space left on device' Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Failed to write output buffer to disk: 'No space left on device' Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20199]: Can't rename dump file: No space left on device Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Ident: '7304bb2' Flows: 20164, Packets: 152128, Bytes: 70428325, Sequence Errors: 13468, Bad Packets: 0 Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20199]: Terminating nfcapd. Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20196]: Terminating nfcapd. Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20193]: Ident: '7304bcnt2' Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 26771, Bad Packets: 0 Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20193]: Terminating nfcapd. Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20205]: Ident: '7606_2_Lab' Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0 Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20208]: Ident: '7606_1_Lab' Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0 Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20205]: Terminating nfcapd. Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20208]: Terminating nfcapd. Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20202]: Can't rename dump file: No space left on device Jan 30 08:14:52 hail /usr/local/bin/nfcapd[20202]: Terminating nfcapd. This is not the issue - I understand that file expiration is not exact on the limit... So, I set out to free some space on the partition, by deleting old flow files. After I restarted nfsen, I got this problem: Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20241]: Standard setsockopt, SO_RCVBUF is 135168 Requested length is 200000 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20241]: System set setsockopt, SO_RCVBUF to 262142 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20243]: Startup. Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20243]: Process_v9: New exporter domain 0 Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20244]: Standard setsockopt, SO_RCVBUF is 135168 Requested length is 200000 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20244]: System set setsockopt, SO_RCVBUF to 262142 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20246]: Startup. Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20246]: Process_v9: New exporter domain 0 Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20247]: Standard setsockopt, SO_RCVBUF is 135168 Requested length is 200000 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20247]: System set setsockopt, SO_RCVBUF to 262142 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20249]: Startup. Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20250]: Standard setsockopt, SO_RCVBUF is 135168 Requested length is 200000 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20250]: System set setsockopt, SO_RCVBUF to 262142 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20252]: Startup. Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20253]: Standard setsockopt, SO_RCVBUF is 135168 Requested length is 200000 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20253]: System set setsockopt, SO_RCVBUF to 262142 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20255]: Startup. Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20256]: Standard setsockopt, SO_RCVBUF is 135168 Requested length is 200000 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20256]: System set setsockopt, SO_RCVBUF to 262142 bytes Jan 30 08:16:03 hail /usr/local/bin/nfcapd[20258]: Startup. Jan 30 08:16:03 hail nfsen[20259]: Startup. Version: snapshot-20070110 $Id: nfsend 60 2007-01-09 12:26:47Z peter $ Jan 30 08:16:03 hail nfsen[20259]: Verification sub hierarchy failed. Expected file '/data/nfsen/profiles/live/7304bcnt2//nfcapd.200701292045' does not exist! Jan 30 08:16:03 hail nfsen[20259]: This may indicate an inconsitency between configured sub hierarchy layout and real layout. Jan 30 08:16:03 hail nfsen[20259]: Rerun RebuildHierarchy.pl to fix. There were a lot of missing files, because they couldn't be created. Unfortunatelly, RebuildHierarchy couldn't recreate them :( (it had an undefined variable somewhere - I didn't write down the line number - sorry). I recreated some files by hand (with touch), and nfsen seemed to start allright - unfortunatelly, the web interface had 'Graph Errors'. Jan 30 08:23:17 hail nfsen[20437]: Update profile live in group . Jan 30 08:23:17 hail nfsen[20437]: Failed get stat info for requested time slot Jan 30 08:23:24 hail prefixStats: comm server started: 21578 Jan 30 08:23:24 hail prefixStats: Error generating details graph: Arg: 'live', '', 'TCP', 'flows', '', '-86400', '', '1170013200', '1170013200', '288', '100', '1', '0', '0' Jan 30 08:23:24 hail prefixStats: comm server started: 21581 Jan 30 08:23:24 hail prefixStats: Error generating details graph: Arg: 'live', '', 'UDP', 'flows', '', '-86400', '', '1170013200', '1170013200', '288', '100', '1', '0', '0' After a few more tries, I tried to delete the sources and add them again. I had problems again - lots of uninitialized variables and nothing done when trying to readd the sources. Unfortunatelly, I didn't save the error messages, but maybe you can get an idea from syslog (I was doing ./nfsen reconfig): Jan 30 08:31:58 hail nfsen[28716]: Startup. Version: snapshot-20070110 $Id: nfsend 60 2007-01-09 12:26:47Z peter $ Jan 30 08:31:58 hail nfsen[28718]: Comm server started: [28718] Jan 30 08:31:58 hail nfsen[28717]: nfsend: [28717] Jan 30 08:31:58 hail nfsen[28717]: Use of uninitialized value in join or string at /data/nfsen/libexec/NfProfile.pm line 772, <ProFILE> line 15. Jan 30 08:31:58 hail last message repeated 2 times Jan 30 08:31:58 hail nfsen[28717]: Update profile live in group . Jan 30 08:31:58 hail nfsen[28717]: Error GenGraph: Profile: live, traffic-day: parameter '7304bcnt21170138600' does not represent a number in line AREA:7304bc nt21170138600:7304bcnt2 Jan 30 08:31:58 hail last message repeated 3 times Jan 30 08:31:58 hail nfsen[28717]: Error GenGraph: Profile: live, flows-day: parameter '7304bcnt21170138600' does not represent a number in line AREA:7304bcnt 21170138600:7304bcnt2 Jan 30 08:31:58 hail last message repeated 3 times Jan 30 08:31:58 hail nfsen[28717]: Error GenGraph: Profile: live, packets-day: parameter '7304bcnt21170138600' does not represent a number in line AREA:7304bc nt21170138600:7304bcnt2 Jan 30 08:31:58 hail last message repeated 3 times Jan 30 08:31:58 hail nfsen[28717]: Error graph update: Error GenGraph: Profile: live, packets-day: parameter '7304bcnt21170138600' does not represent a number in line AREA:7304bcnt21170138600:7304bcnt2 Jan 30 08:31:58 hail nfsen[28717]: Use of uninitialized value in join or string at /data/nfsen/libexec/NfProfile.pm line 827. Jan 30 08:31:58 hail last message repeated 2 times The solution was to completely delete /data/nfsen and reinstall it from scratch. Now it works ok, but I don't know how I could have fixed it without reinstalling. Also, why didn't the expire settings work in the first place? How much free space do I have to leave on the partition to ensure that this doesn't happen again? Thank you for your time. -- Adrian Popa ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nfsen-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
