[Flow-tools] [patch] flow-cidr memory leak
The following patch fixes a memory leak in flow-cidr (flow-tools-0.68 / Inter.netPH-1.3). http://hhg.to/flow-cidr-memory-leak.diff Bug description: A memory leak occurs in the original version when filtering by src/dst ipblock (option -i or -I) because a new patricia tree is created in each iteration of the comparison loop. The patch moves the patricia tree creation code outside the loop. ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
[Flow-tools] flow-capture memory leak? (flow-tools 0.68)
Greetings, My flow-crunching server decided to yell at me this afternoon about excessive swapping - upon closer investigation it appears that flow-capture decided to chew up memory: 68335 root 1 760 2186M 602M select 0 20:50 0.00% flow-capture (pre-restart, about 2-3 weeks) Vs. 94506 root 1 760 9716K 5612K select 0 0:23 0.00% flow-capture ( post-restart, about 5 min.) I'm pretty sure that flow-capture shouldn't be chewing up 2GB of memory, so is anyone else seeing this? This is flow-tools 0.68 (built from FreeBSD Ports, host system is: FreeBSD aspen.invision.net 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Tue Jan 23 11:50:47 EST 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/aspen.invision.net amd64 ) I saw some stuff about a possible memory leak on this list from last January - was this confirmed or has nobody had a chance to debug this just yet? Thanks in advance, -MG ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
[Flow-tools] Memory leak ?
Hello list, I use flow-control from about 1 week. My OS is FreeBSD 6.0-RELEASE #0 for AMD64. All works fine but yesterday I found this in dmesg: Jan 4 21:30:58 katana kernel: swap_pager_getswapspace(16): failed Jan 4 21:30:58 katana kernel: swap_pager_getswapspace(12): failed Jan 4 21:30:58 katana kernel: swap_pager_getswapspace(6): failed I have 512MB RAM and 2GB swapspace On this PC I have squid-cache running, but when check which program use more memory I found this: last pid: 49917; load averages: 0.94, 0.96, 0.79 up 1+05:58:05 21:58:29 187 processes: 2 running, 184 sleeping, 1 lock CPU states: 5.5% user, 0.0% nice, 25.6% system, 21.3% interrupt, 47.6% idle Mem: 678M Active, 59M Inact, 166M Wired, 36M Cache, 111M Buf, 1636K Free Swap: 2048M Total, 2048M Used, K Free, 100% Inuse, 4K In, 32K Out PID USERNAME THR PRI NICE SIZERES STATETIME WCPU COMMAND 518 root1 960 2648M 104M select 0:33 0.00% flow-capture 635 www 1 40 18504K 8908K kqread 0:22 0.00% thttpd My starting line for flow-capture is: /usr/local/bin/flow-capture -p /var/run/flow-capture.pid -n 287 -N 0 -w /var/log/netflows/ -S 5 /127.0.0.1/8899 Is that huge memory usage is memory leak or I do something wrong ? Thanks. ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
Re: [Flow-tools] Memory leak ?
On Jan 04, Todor Dragnev wrote: Hello list, I use flow-control from about 1 week. My OS is FreeBSD 6.0-RELEASE #0 for AMD64. All works fine but yesterday I found this in dmesg: Jan 4 21:30:58 katana kernel: swap_pager_getswapspace(16): failed Jan 4 21:30:58 katana kernel: swap_pager_getswapspace(12): failed Jan 4 21:30:58 katana kernel: swap_pager_getswapspace(6): failed I have 512MB RAM and 2GB swapspace On this PC I have squid-cache running, but when check which program use more memory I found this: last pid: 49917; load averages: 0.94, 0.96, 0.79 up 1+05:58:05 21:58:29 187 processes: 2 running, 184 sleeping, 1 lock CPU states: 5.5% user, 0.0% nice, 25.6% system, 21.3% interrupt, 47.6% idle Mem: 678M Active, 59M Inact, 166M Wired, 36M Cache, 111M Buf, 1636K Free Swap: 2048M Total, 2048M Used, K Free, 100% Inuse, 4K In, 32K Out PID USERNAME THR PRI NICE SIZERES STATETIME WCPU COMMAND 518 root1 960 2648M 104M select 0:33 0.00% flow-capture 635 www 1 40 18504K 8908K kqread 0:22 0.00% thttpd My starting line for flow-capture is: /usr/local/bin/flow-capture -p /var/run/flow-capture.pid -n 287 -N 0 -w /var/log/netflows/ -S 5 /127.0.0.1/8899 Is that huge memory usage is memory leak or I do something wrong ? There could be a 64-bit problem. Our setup has some 32-bit collectors that feed (among other things) a 64-bit cruncher. Building out the 64-bit cruncher lead me to finding some 32-bit vs. 64-bit bugs: http://mailman.splintered.net/pipermail/flow-tools/2004-December/002501.html But I haven't had to deal with flow-capture on 64-bit Is anybody out there running flow-capture on amd64/x86_64 or another 64 bit platform? I think the easiest way to start looking at this would be to run flow-capture under a memory debugger of some sort, like efence (Electric Fence Malloc Debugger). Mike ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
[Flow-tools] flow-capture memory leak? (flow-tools 0.68 + FreeBSD amd64)
Hi! I have a server which flow-capture is set on. Through time he strongly occupies memory. Noticed it only on amd64 FreeBSD test.com 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Mon Feb 25 01:12:23 EET 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC amd64 /usr/local/bin/flow-capture -E60G -n287 -N0 -z1 -R/usr/local/bill.sh Result after 10 days: 26334 root 1 440 731040K 642840K select 0 0:26 0.00% flow-capture Read forums, write that on Linux of such problem it is not, possibly something with FREEBSD ? As possible to decide this problem? With best regard, Maxim. ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
Re: [Flow-tools] flow-capture memory leak? (flow-tools 0.68 + FreeBSD amd64)
Replying to Maxim Struchaev: Read forums, write that on Linux of such problem it is not, possibly something with FREEBSD ? As possible to decide this problem? As I said, I don't have a freebsd machine to run any tests right now. However if you want, you can try running flow-capture under valgrind (http://valgrind.org/docs/manual/quick-start.html#quick-start.intro) This _can_ detect some memory leaks. It would be helpful if you'll use flow-tools-0.68.4 http://flow-tools.googlecode.com/files/flow-tools-0.68.4.tar.bz2 and wll include lib/ftconfig.h file (which is generated by ./configure) in your reply. -- Paul P 'Stingray' Komkoff Jr // http://stingr.net/key - my pgp key This message represents the official view of the voices in my head ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
Re: [Flow-tools] flow-tools on FreeBSD/amd64
Paul, This is where I'm picking up the garbage data: ftnet.loc_addr.sin_addr.s_addr = ftnet.msgip.ip.s_addr The kernel is GENERIC plus IPFW. Nothing terribly oddball. I'm wondering if this is a variable type issue somewhere. Running a 32bit build on amd64 and the same box doesn't exhibit this issue. If you care to look into this, I can provide access to the box in question. If not, I'll might eventually figure it out. I'm not at the point of caring about a memory leak. When I get there, I can run it through valgrind. Thanks, Mark On Tue, 7 Oct 2008, Paul P Komkoff Jr wrote: Replying to Mark R.: Are there any known issues with flow-tools on 64-bit platforms? I'm trying to run 0.68 on FreeBSD 7.0/amd64 and running into some odd behavior with flow-capture and flow-fanout. There is at least one known memory leak that only happens on 64-bit FreeBSD, that I still haven't fixed (mainly because it requires to set up FreeBSD somewhere). This one that you telling here is new. And it's serious, I wonder why nobody else seeing it. Maybe your kernel is broken? Or maybe it's ipv4-over-ipv6 issue? I don't have access to freebsd system to verify how recvmsg works there, sorry. -- Paul P 'Stingray' Komkoff Jr // http://stingr.net/key - my pgp key This message represents the official view of the voices in my head ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
Re: [Flow-tools] flow-capture going crazy (memory leak or something local?)
[EMAIL PROTECTED] wrote on 02/28/2007 12:19:10 PM: I was troubleshooting an unrelated problem on our netflow cruncher this morning and discovered flow-capture chewing up RAM/Swap (to the point where it allocated all the swap on the box and was less-than-politely killed for doing so). This is the first time I've caught flow-capture being killed for using up too much RAM, but we've had to restart it on occasion for the same problem (flow-capture gets fat and makes our monitoring system cry about the flow-cruncher swapping too much). Two questions - 1) Is anyone else seeing this? I'm seeing something similar. I happened to be doing a 'top' from the command-line right after staring up several flow-captures. Was surprised to see them start to take up 30-40% and higher CPU. However, I noticed also that within 20 - 30 minutes each settled down to their typical about 1%. The problematic flow-captures generally were those with the least traffic. Joe___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
RE: [Flow-tools] flow-capture going crazy (memory leak or something local?)
[EMAIL PROTECTED] wrote on 02/28/2007 12:19:10 PM: I was troubleshooting an unrelated problem on our netflow cruncher this morning and discovered flow-capture chewing up RAM/Swap (to the point where it allocated all the swap on the box and was less-than-politely killed for doing so). This is the first time I've caught flow-capture being killed for using up too much RAM, but we've had to restart it on occasion for the same problem (flow-capture gets fat and makes our monitoring system cry about the flow-cruncher swapping too much). Two questions - 1) Is anyone else seeing this? I'm seeing something similar. I happened to be doing a 'top' from the command-line right after staring up several flow-captures. Was surprised to see them start to take up 30-40% and higher CPU. However, I noticed also that within 20 - 30 minutes each settled down to their typical about 1%. The problematic flow-captures generally were those with the least traffic. I've seen the CPU jumping on startup too - I've always attributed this to normal startup tasks, since the CPU utilization settles down after a few minutes but (at least in our case) the memory utilization keeps growing. Our setup is one flow-capture daemon which takes data from our edge routers (5 routers, for a total of about 4,000 flows during each 5-minute capture window), with each flow being post-processed when it gets rotated handed off to our billing system and flowscan to make pretty graphs - I'm guessing this is pretty standard, it was originally set up this way so no changes would be required if new edge routers sprang up (eg. At new sites). Are you running any on-rotation scripts as part of your setup? I'd be interested in seeing if the memory use climbing is related to that or if it's someting intrinsic to flow-capture. (In case anyone wants to see - the attached picture shows what happens to our graphs when flow-capture is doing its chew-up-swap-and-restart thing - it usually lives for 2-3 minutes, so the last half of the collection interval gets processed graphed, but we have a pile of tmp.xxx files for the first half of each period) funky.PNG Description: funky.PNG ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
Re: [Flow-tools] flow-tools on FreeBSD/amd64
Paul,
Upon further investigation, it appears as if there is an alignment issue in
'struct msgip' inside of 'struct ftnet'. I've looked at some other code, and
there appear to be a handful of useful CMSG_* macros to help deal with this.
RFC 2292 details them.
The attached patch works cleanly on my system and fixes the issue at hand.
Hopefully it doesn't break other platforms. It's there for anybody who is
interested.
Thanks,
Mark
On Tue, 7 Oct 2008, Mark R. wrote:
Paul,
This is where I'm picking up the garbage data:
ftnet.loc_addr.sin_addr.s_addr = ftnet.msgip.ip.s_addr
The kernel is GENERIC plus IPFW. Nothing terribly oddball. I'm wondering if
this is a variable type issue somewhere. Running a 32bit build on amd64 and
the same box doesn't exhibit this issue.
If you care to look into this, I can provide access to the box in question.
If not, I'll might eventually figure it out.
I'm not at the point of caring about a memory leak. When I get there, I can
run it through valgrind.
Thanks,
Mark
On Tue, 7 Oct 2008, Paul P Komkoff Jr wrote:
Replying to Mark R.:
Are there any known issues with flow-tools on 64-bit platforms? I'm
trying to run 0.68 on FreeBSD 7.0/amd64 and running into some odd
behavior with flow-capture and flow-fanout.
There is at least one known memory leak that only happens on 64-bit
FreeBSD, that I still haven't fixed (mainly because it requires to set up
FreeBSD somewhere).
This one that you telling here is new. And it's serious, I wonder why
nobody else seeing it. Maybe your kernel is broken?
Or maybe it's ipv4-over-ipv6 issue?
I don't have access to freebsd system to verify how recvmsg works
there, sorry.
--
Paul P 'Stingray' Komkoff Jr // http://stingr.net/key - my pgp key
This message represents the official view of the voices in my head
___
Flow-tools mailing list
[EMAIL PROTECTED]
http://mailman.splintered.net/mailman/listinfo/flow-tools--- lib/ftlib.h-orig2008-03-09 13:09:01.0 +
+++ lib/ftlib.h 2008-10-07 17:41:11.0 +
@@ -459,11 +459,16 @@
int fd; /* fd receiving flows on */
struct mymsghdr msg;/* recvmsg data */
struct {
-struct cmsghdr hdr;
#ifdef IP_RECVDSTADDR
+#ifdef CMSG_DATA
+char cbuf[CMSG_SPACE(sizeof(struct sockaddr_storage))];
+#else
+struct cmsghdr hdr;
struct in_addr ip;
+#endif /* CMSG_DATA */
#else
#ifdef IP_PKTINFO
+struct cmsghdr hdr;
struct in_pktinfo pktinfo;
#endif /* else */
#endif /* IP RECVDSTADDR */
--- src/flow-fanout.c-orig 2008-01-27 20:48:55.0 +
+++ src/flow-fanout.c 2008-10-07 17:42:33.0 +
@@ -124,6 +124,11 @@
int i, n, detach, one, ret, offset, hdr_len;
int npeers, tx_delay;
int stat_interval, stat_next, src_ip_spoof;
+#ifdef IP_RECVDSTADDR
+#ifdef CMSG_DATA
+ struct cmsghdr *cmsg;
+#endif
+#endif
time_startup = time((time_t)0L);
@@ -625,12 +630,24 @@
#ifdef IP_RECVDSTADDR
/* got destination IP back? */
+#ifdef CMSG_DATA
+ for (cmsg = CMSG_FIRSTHDR(ftnet.msg); cmsg != NULL;
+ cmsg = CMSG_NXTHDR(ftnet.msg, cmsg)) {
+ if (cmsg-cmsg_level == IPPROTO_IP
+ cmsg-cmsg_type == IP_RECVDSTADDR) {
+ memcpy(ftnet.loc_addr.sin_addr.s_addr,
+ CMSG_DATA(cmsg), sizeof(struct in_addr));
+ break;
+ }
+ }
+#else
if ((ftnet.msgip.hdr.cmsg_level == IPPROTO_IP)
(ftnet.msgip.hdr.cmsg_type == IP_RECVDSTADDR)) {
ftnet.loc_addr.sin_addr.s_addr = ftnet.msgip.ip.s_addr;
} else {
ftnet.loc_addr.sin_addr.s_addr = 0;
}
+#endif /* CMSG_DATA */
#else
#ifdef IP_PKTINFO
if ((ftnet.msgip.hdr.cmsg_level == IPPROTO_IP)
--- src/flow-receive.c-orig 2008-01-27 20:48:55.0 +
+++ src/flow-receive.c 2008-10-07 17:42:35.0 +
@@ -93,6 +93,11 @@
char fmt_src_ip[32], fmt_dst_ip[32], fmt_dst_port[32];
char xl_rec[FT_IO_MAXREC], *out_rec;
int stat_interval, stat_next;
+#ifdef IP_RECVDSTADDR
+#ifdef CMSG_DATA
+ struct cmsghdr *cmsg;
+#endif
+#endif
time_startup = time((time_t)0L);
@@ -488,12 +493,24 @@
#ifdef IP_RECVDSTADDR
/* got destination IP back? */
+#ifdef CMSG_DATA
+ for (cmsg = CMSG_FIRSTHDR(ftnet.msg); cmsg != NULL;
+ cmsg = CMSG_NXTHDR(ftnet.msg, cmsg)) {
+ if (cmsg-cmsg_level == IPPROTO_IP
+ cmsg-cmsg_type == IP_RECVDSTADDR) {
+ memcpy(ftnet.loc_addr.sin_addr.s_addr,
+ CMSG_DATA(cmsg), sizeof(struct in_addr));
+ break;
+ }
+ }
+#else
if ((ftnet.msgip.hdr.cmsg_level == IPPROTO_IP)
(ftnet.msgip.hdr.cmsg_type == IP_RECVDSTADDR)) {
ftnet.loc_addr.sin_addr.s_addr = ftnet.msgip.ip.s_addr;
} else
[Flow-tools] flow-tools 0.68.1 stable release
OK, after sitting for 3 weeks in -rc3, I'm releasing final flow-tools 0.68.1. The tarball is at http://flow-tools.googlecode.com/files/flow-tools-0.68.1.tar.bz2 Updated RPMs should hit fedora repositories shortly. Project page is at: http://flow-tools.googlecode.com/ Changes since 0.68: Alexey Bestchiokov (1): Fix flow-export problems described in Andreas Jochens (1): Fix an invalid lvalue in assignment compile error. Ben Feinstein (1): Add extra debugging to ftpdu_verify Denis Shaposhnikov (1): Fix crash on expiring old files. Ilya Anfimov (1): Fix ip-destination-address/source-tag Kurt Roeckx (1): Use time_t instead of u_int32 in ftfile_mkpath and ftfile_pathname Oleg Milaenko (1): Use temporary time_t type variables to hold time values before displaying. Otavio R. Piske (1): Fix memory leak in flow-export. Paul Hampson (1): Fix paths for jade rebuilding documentation. Paul P. Komkoff Jr (26): Remove autogenerated files from source tree Fix first line of included python scripts. Avoid external debug variable. Fix unitialized variable usage in flow-send Fix unitialized variable usage in flow-export.c On newer toolchain getopt won't work without including unistd.h Update build system to latest automake. Update version Fix typo in commit f673452e04ffca2a5592061374cda67b4492 Add gitignore patterns Update .gitignore We pass make distcheck now Remove some extra crap from configure.in Fix first line of included utility scripts Rework postgresql detection Add fork readme, script to make releases, update gitignore Update version Update distfiles list, remove pyflowtools tarball. Put bison source parser instead of C one. Remove unneeded yacc -d option Include unistd.h in lib/support.c Include unistd.h in ftsym.c Fix unitialized variables in lib/ftstat.c and lib/fttag.c Fix overlapping defines in src/flow-export.c Update version Release 0.68.1 Radu Spineanu (3): Write the pid file only after binding the socket Fix small error in flow-nfilter manpage. Compiles libft with -fPIC to work ok with libcflow-perl -- Paul P 'Stingray' Komkoff Jr // http://stingr.net/key - my pgp key This message represents the official view of the voices in my head ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
[Flow-tools] flow-capture going crazy (memory leak or something local?)
I was troubleshooting an unrelated problem on our netflow cruncher this morning and discovered flow-capture chewing up RAM/Swap (to the point where it allocated all the swap on the box and was less-than-politely killed for doing so). This is the first time I've caught flow-capture being killed for using up too much RAM, but we've had to restart it on occasion for the same problem (flow-capture gets fat and makes our monitoring system cry about the flow-cruncher swapping too much). Two questions - 1) Is anyone else seeing this? 2) Is there a way for me to recover the data in the tmp.x flow files? (we've got about 50 of them and I'd like to get that data into our system if possible) This is flow-tools 0.68 on FreeBSD 6.2-RELEASE (amd64); flow-tools runs as flow-capture -D -N 0 -n 288 -w /netflow/flows/raw -R /netflow/scripts/cook-flows.sh 0/0/2055 with the attached cook-flows.sh - Also (probably unrelated) - when flow-capture gets a SIGTERM, it does the right thing (rolls the flow file, runs the cook-flows script), but it doesn't let go of the netflow port until after cook-flows.sh exits. I tried a perl version of the above script that forks off and has the parent exit immediately while the child does the flow processing, but the same problem hangs around. Any ideas on that one? Any thoughts/ideas? -MG #!/bin/sh # # I take flow-tools files and convert them to cflowd format... # It's a drab, dull meaningless existence, but it makes # flowscan happy... # # # Give flow-tools a second to roll files # (Probably not strictly necessary) # sleep 1 # # Set up some variables we'll be using later # # Date/time/timezone for filename date=`date +%Y%m%d_%T%z` # # Cook flowfile (convert to cflowd format) ... # flow-export -f 0 /netflow/flows/raw/$1 /netflow/flows/raw/flows.$date 2/dev/null # ... and put it where flowscan expects it mv /netflow/flows/raw/flows.$date /netflow/flows/cooked/flows.$date # Move raw FT file to the saved directory mkdir -p /netflow/flows/saved/`date +%Y/%m/%d` mv /netflow/flows/raw/$1 /netflow/flows/saved/`date +%Y/%m/%d`/ # # Run aggregate-flows for bandwidth billing on # the cooked flow file too. # cd /netflow/bwbilling/bin ; ./aggregate_flows.pl -d -c aggflows.cfg /netflow/flows/cooked/flows.$date ___ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
