Caio (& everyone else) - My company currently uses a nifty combination of flow-tools (flow-report, flow-filter) and a modified FlowScan/JKFlow codebase for traffic monitoring. It's a pretty extensive system (it does per-client and per-purpose (CoLo, Leased Lines, Internal/Infrastructure) traffic monitoring, as well as implementing abnormal traffic detection (overall network & per-client)).
The FlowScan code is an ugly hack (cooking flows to cflow format before handing them off to be processed rather than hacking FlowScan & Friends), but it all comes together quite nicely. If there's any interest I'm sure I can convince the powers that be to let me package it for release :) Numbers-wise our system takes a lot of disk. For our network (medium-sized ISP) 5 days of stored flows (for reporting) is about 48GB, RRDs for all our graphs are about 1.5GB, and misc. bandwidth billing data (handled by the same system) is about 30GB. It's also a CPU-Intensive system (A shiny new Dell 1950 takes about 2.5 minutes to process a 5 minute window of data with FlowScan - Anyone got a multithreaded version of that kicking around? :) > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:flow-tools- > [EMAIL PROTECTED] On Behalf Of flow-tools- > [EMAIL PROTECTED] > Sent: Wednesday, September 19, 2007 12:07 PM > To: [email protected] > Subject: Flow-tools Digest, Vol 46, Issue 6 > > Send Flow-tools mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.splintered.net/mailman/listinfo/flow-tools > or, via email, send a message with subject or body 'help' to > [EMAIL PROTECTED] > > You can reach the person managing the list at > [EMAIL PROTECTED] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Flow-tools digest..." > > > Today's Topics: > > 1. report type in flow-report (Caio Brentano) > 2. Support Netflow v9 and IPFIX (Roque Gagliano) > 3. Store data (Caio Brentano) > 4. Re: Store data (Dave Plonka) > 5. RES: [Flow-tools] Store data (Caio Brentano) > 6. Re: RES: [Flow-tools] Store data (Dave Plonka) > 7. Re: RES: [Flow-tools] Store data (Dave Plonka) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 18 Sep 2007 15:43:12 -0300 > From: "Caio Brentano" <[EMAIL PROTECTED]> > Subject: [Flow-tools] report type in flow-report > To: <[email protected]> > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset="us-ascii" > > Hi all > > > > > > I'm trying to create some reports with flow-report. Can I create my own > "Report Type" for flow-report? > > For example: I need a report about "ip-source-port" + "ip-protocol". > Can I > create my own "Report Type" for it? > > > > I know that there is a report type with these information, but it has > some > informations that don't care for me, such as "ip-tos". > > > > Regards. > > > > -- > > Caio Brentano dos Passos > > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.splintered.net/pipermail/flow- > tools/attachments/20070918/d72435e7/attachment-0001.htm > > ------------------------------ > > Message: 2 > Date: Wed, 19 Sep 2007 08:48:34 -0300 > From: Roque Gagliano <[EMAIL PROTECTED]> > Subject: [Flow-tools] Support Netflow v9 and IPFIX > To: [email protected] > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset="us-ascii" > > Skipped content of type multipart/alternative-------------- next part - > ------------- > A non-text attachment was scrubbed... > Name: not available > Type: application/pgp-signature > Size: 189 bytes > Desc: This is a digitally signed message part > Url : http://mailman.splintered.net/pipermail/flow- > tools/attachments/20070919/c9f2aa33/attachment-0001.bin > > ------------------------------ > > Message: 3 > Date: Wed, 19 Sep 2007 11:52:23 -0300 > From: "Caio Brentano" <[EMAIL PROTECTED]> > Subject: [Flow-tools] Store data > To: <[email protected]> > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset="us-ascii" > > Hi all > > > > I'm developing a web-based system to show reports and graphs of data > collected from flows. > > > > What do you suggest me to store this data? I developed a netowork > monitoring > system based on SNMP that data is stored in RRD. > > Is it ok for flow? Is there a better way? > > > > Regards > > -- > > Caio Brentano > > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.splintered.net/pipermail/flow- > tools/attachments/20070919/dc7962ad/attachment-0001.htm > > ------------------------------ > > Message: 4 > Date: Wed, 19 Sep 2007 10:02:17 -0500 > From: Dave Plonka <[EMAIL PROTECTED]> > Subject: Re: [Flow-tools] Store data > To: [email protected] > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset=us-ascii > > > Hi Caio, > > On Wed, Sep 19, 2007 at 11:52:23AM -0300, Caio Brentano wrote: > <snip> > > What do you suggest me to store this data? I developed a netowork > monitoring > > system based on SNMP that data is stored in RRD. > > > > Is it ok for flow? Is there a better way? > > There are a number of FlowScan reports that digest raw flow data, in > flow-tools format or others, and populate RRD files. These include > the reports supplied with FlowScan and others such as CUFlow. > > There are mailing lists and online docs for both. > Here's one place to start: http://net.doit.wisc.edu/~plonka/FlowScan/ > > Dave > > P.S. Most of my reports write to RRD files for time-series graphing, > but some flow data, such as top talkers works better of course as > tabular data. Thus some reports produce HTML tables. > > -- > [EMAIL PROTECTED] http://net.doit.wisc.edu/~plonka/ Madison, WI > > > ------------------------------ > > Message: 5 > Date: Wed, 19 Sep 2007 12:15:10 -0300 > From: "Caio Brentano" <[EMAIL PROTECTED]> > Subject: RES: [Flow-tools] Store data > To: <[EMAIL PROTECTED]>, <[email protected]> > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset="us-ascii" > > These are RRD graphs http://wwwstats.net.wisc.edu/ ? > > -- > Caio Brentano > > > > ------------------------------ > > Message: 6 > Date: Wed, 19 Sep 2007 10:23:43 -0500 > From: Dave Plonka <[EMAIL PROTECTED]> > Subject: Re: RES: [Flow-tools] Store data > To: [email protected] > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset=us-ascii > > On Wed, Sep 19, 2007 at 12:15:10PM -0300, Caio Brentano wrote: > > These are RRD graphs http://wwwstats.net.wisc.edu/ ? > > Yes, of course. > > If this is new to you, perhaps you'd like to read the original paper: > > http://www.usenix.org/events/lisa2000/plonka.html > > Some of the most popular 3rd party documentation I've seen for using > FlowScan is from these onlamp articles. E.g.: > > http://www.onlamp.com/pub/a/bsd/2005/10/27/Big_Scary_Daemons.html > > A number of people use FlowScan, but use the CUFlow or other reports > instead of the original ones I wrote (CampusIO SubNetIO)... > > Since it has been a long time since a FlowScan release, you need to > patch it up by hand a bit to get it all working. This is documented > in the link "Tips on configuring FlowScan with flow-tools." at > http://www.splintered.net/sw/flow-tools/ : > > http://net.doit.wisc.edu/~plonka/list/flowscan/archive/1117.html > > Dave > > P.S. beyond that the mailing list archives have a lot of FAQs covered. > http://lists.wiscnet.net/mailman/listinfo/flowscan/ > > -- > [EMAIL PROTECTED] http://net.doit.wisc.edu/~plonka/ Madison, WI > > > ------------------------------ > > Message: 7 > Date: Wed, 19 Sep 2007 10:49:39 -0500 > From: Dave Plonka <[EMAIL PROTECTED]> > Subject: Re: RES: [Flow-tools] Store data > To: [email protected] > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset=us-ascii > > On Wed, Sep 19, 2007 at 10:23:43AM -0500, Dave Plonka wrote: > <snip> > > Some of the most popular 3rd party documentation I've seen for using > > FlowScan is from these onlamp articles. E.g.: > > > > http://www.onlamp.com/pub/a/bsd/2005/10/27/Big_Scary_Daemons.html > > Actually this is the link I meant: > > "Visualizing Network Traffic with Netflow and FlowScan" > http://www.onlamp.com/pub/a/bsd/2005/09/15/Big_Scary_Daemons.html > > -- > [EMAIL PROTECTED] http://net.doit.wisc.edu/~plonka/ Madison, WI > > > ------------------------------ > > _______________________________________________ > Flow-tools mailing list > [email protected] > http://mailman.splintered.net/mailman/listinfo/flow-tools > > > End of Flow-tools Digest, Vol 46, Issue 6 > ***************************************** _______________________________________________ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
