flow-rpt2rrd
Description: Binary data
flow-log2rrd
Description: Binary data
Description: Binary data
On Jul 23, 2004, at 12:06 PM, John Kristoff wrote:
Until direct html output is supported with flow-report I imagine a number of people have written their own parsers to convert the output to HTML. I have hacked up one that creates familiar top 10 tables in HTML format that appears to work for me. Others off-list indicated that it might be useful for them to use the methods I did so I'll post the details here. It seems there could be more examples of how others are using flow-tools so hopefully this will help someone.
It's not very generic and may even have a couple of minor problems. A pointer to something better or enhancements are of course welcome.
I hereby place any code below into the public domain.
Using the -R option to flow-capture, I run a script that, among other things, creates a top TCP/UDP port usage report that is run like this (where $1 is the recently rotated flow file passed to the script):
flow-cat $1 | flow-report -s report.conf -S top-ports
To create top 10 reports (change the records option to whatever is appropriate for you) a tcp-egress.report looks like this:
include-filter filter.conf stat-report tcp-dstport-egress-octets type ip-destination-port filter tcp-egress output format ascii records 10 # scale xxx if you're getting sampled flows path tcp-dstport-egress-octets.txt fields -duration sort +octets options +totals # stat-report tcp-srcport-egress-flows type ip-source-port filter tcp-egress output format ascii records 10 # scale xxx if you're getting sampled flows path tcp-srcport-egress-octets.txt fields -duration sort +flows options +totals # # ...add other tcp/udp definitions as necessary # stat-definition top-ports report tcp-dstport-egress-octets report tcp-srcport-egress-flows # # ...
Create the appropriate filter.conf file that sets up the needed primitives and definitions for the filters in report.conf. For example:
# ... # filter-primitive my-netblocks type ip-address-prefix permit 192.0.2.0/24 default deny # filter-primitive tcp type ip-protocol permit tcp default deny # filter-definition tcp-egress match ip-source-address my-netblocks match ip-protocol tcp # # ...
For TCP/UDP reports you can create a bunch of reports by mixing the
combination of protocol (e.g. TCP or UDP), direction (e.g. ingress
or egress), port (e.g. source or destination) and rank type (e.g. flows,
octets, packets), which should keep network analysts busy and asking
for more.
Now that those definitions are out of the way, it is time for the text to HTML conversion. I do this with perl and with the code below I make some assumptions about what the output looks like. The code may need to be tweaked if your reports are going to output something that the script below doesn't expect (e.g. bps or other columns).
I ended up making a command line options necessary:
[ --type stat-report-type ] [ --rows max-rows ] [ --key sortfield ] [ --title report-title ] [ --headfoot ]
--type type from stat-report-type (e.g. ip-source-port or just srcPort
for short)
--rows the number of rows to create in the HTML table
--key the sort field (e.g. flows, octets or packets)
--title <caption> text for the table
and --headfoot is a boolean option if present will add the HTML header
and footer tags. You might not enable this if you are taking the output
from this script and appending it to aggregate top ports file with other
tables. I'll show an example of this shortly.
In the meantime, here is the script:
#!/usr/bin/perl -wT use strict; $|=1;
### init variables to defaults # my $rows = 10; # maximum number of rows my $title = "Flow Report"; # report title my $key = "octets"; # sort on flows/octets/packets my $type = "n/a"; # report type (e.g. srcPort, dstPort) my $headfoot = 0; # do HTML header/footer tags?
### retrieve options
#
use Getopt::Long;
my $result = 0;
$result = GetOptions( "title=s" => \$title, # report title
"rows=i" => \$rows, # max no. of rows
"key=s" => \$key, # key (flows, packets, octets)
"type=s" => \$type, # type (e.g. srcPort, dstPort)
"headfoot" => \$headfoot # do HTML header/footer tags?
);
my $rank = 1; # current row/ranking in report
# doHeader
# HTML header
# if HTML header not disabled, so scripts can incorpriate into another page
if( $headfoot) {
print "<html>\n<body bgcolor=\"#ffffff\">\n<center>\n";
}
# table header print "<table border=\"1\" cellspacing=\"0\" cellpadding=\"3\">\n"; print "<caption align=\"top\">Top $rows <b>$title</b></caption>\n";
### use variable to determine where to put highlighted column my %highlight = ( "flows" => "", "octets" => "", "packets" => "" );
### total flow, octet and packet counts my %totals = ();
# highlight key table cell in green $highlight{$key} = ' bgcolor="#90ee90"';
# printer column heading
print "<tr bgcolor=\"#ffffcc\" align=\"center\"><td><b>rank</b></td><td><b>$type</ b><td$highlight{'flows'}><b>flows<b></ td><td$highlight{'octets'}><b>octets</b></ td><td$highlight{'packets'}><b>packets</b></td></tr>\n";
# highlight key column in light blue $highlight{$key} = ' bgcolor="#add8e6"';
while( defined(my $input = <>) ) { chomp $input;
# found a totals line, use for % output if( $input =~ /^# rec1: ignores,flows,octets,packets/ ) { $input = <>; # get totals ( $totals{'flows'}, $totals{'octets'}, $totals{'packets'} ) = $input =~ /^\d+,(\d+),(\d+),(\d+)(?:,\w+)*$/; next; }
if( $input =~ /^\s*(#.*)?$/) { next; } # skip comments or blank lines
my( $port, $flows, $octets, $packets) = ( "", "", "", "" ); ( $port, $flows, $octets, $packets ) = split ( /\,/, $input, 4 );
# print a row: rank, port, flows (%), octets (%), packets (%)
printf( "<tr align=\"right\"><td>%d</td><td>%s</td>", $rank++, $port );
printf( "<td$highlight{'flows'}>%s \(%3.2f%%\)</td>", commify($flows), ($flows/$totals{'flows'})*100 );
printf( "<td$highlight{'octets'}>%s \(%3.2f%%\)</td>", commify($octets), ($octets/$totals{'octets'})*100 );
printf( "<td$highlight{'packets'}>%s \(%3.2f%%\)</td></tr>\n", commify($packets), ($packets/$totals{'packets'})*100 );
}
# table footer print "</table>\n";
# html footer if ($headfoot) { print "</center>\n</body>\n</html>\n"; }
exit(0); ### EXIT SUCCESS
### SUB ROUTINES # # from Perl Cookbook sub commify { my $text = reverse $_[0]; $text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g; return scalar reverse $text; }
### EOF
What I do is run this script against multiple port report text files without the --headfoot option and concatenate each output together into one top ports report page. Doing something like this:
cat > ${HTMLREPORT} << EOF
<html>
<body>
`cat ${HTMLHEADER}`
<center>
<table border="0" cellspacing="0" cellpadding="1">
<caption align="top"For previous 5 minutes starting at <b>${FLOWDATE}.${FLOWTIME}</b></caption>
<tr>
<td>
`${TXT2HTML} --title="TCP source ports egress by octets" --type=srcPort --rows=10 --key=octets tcp-srcport-egress-octets.txt`
</td>
<td>
<! ...>
<! also by flows and packets>
</td>
</tr>
<! ...>
<! other reports and HTML tags here as appropriate>
</table>
</center>
`cat ${HTMLFOOTER}`
EOF
${HTMLREPORT} is the resulting output HTML file containing all the tables concatenated together. ${HTMLHEADER} points to a file that includes any introdoctury text/html that I want to appear at the top of the page. ${HTMLFOOTER} is for anything I want to appear at the end of the page. ${TXT2HTML} should point to the Perl script I included above.
One last tip. You may have noticed the ${FLOWDATE}.${FLOWTIME} variables that get put into the outer page table. I just pull out that info from the rotated flow file name. This will vary depending on how flow-capture is storing your files (-N nesting level parameter), but if we assume the default, the parameter passed to the script will look something take this format if you are using NetFlow version 5 format:
ft-v05.YYYY-MM-DD.HHMMSS-ZONE
where
YYYY is the year MM is the month DD is the day HH is the hour in 24-hour format MM is the minutes past the hour SS is the seconds past the minute ZONE is the timezone offset to UTC in 24-hour format
Grabbing the flow date and time the would be accomplished like this, where $1 is the flow filename passed to the script:
FLOWTIMESTAMP=`expr substr $1 8 17`
Using all those tips you can store the top ports HTML reports off in their own directories nested by date and hour if necessary. The end of the script can update a link to the most recent. for example:
ln -sf ${HTMLREP} /path/to/web/directory/top-ports.html
When all is said and done, if I typed in all of the above correctly and
your setup is similar to mine, you might end up with something that looks
a little like this:
<http://aharp.ittns.northwestern.edu/tmp/top-ports-sample.html>
John _______________________________________________ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
_______________________________________________ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
