If you are already using nfdump in combination with NfSen, you could easily
write a small NfSen Perl plugin for that,
which processes each timeslot. Find a sample plugin for this appended.
Take a look at sub run {}. If you post process the output, you may want to use
-o csv and -q.
Put the plugin into your plugin directory and try to run it:
for example:
/data/nfsen/bin/testPlugin -p top10 -P live -t 201210312355
Add your code for the database to sub run {} and finally add the plugin in
nfsen.conf
You may run any test cycles with testPlugin to check if it works.
Hope, this helps as a short primer
- Peter
On 10/31/12 21:55, Alan Whinery wrote:
> Yes, probably will work.
>
> Since I have not focused on the dynamics of when NFSen is writing a given
> file, I can't say, if the file name is the
> start, you may want to delay to 1,6,11,21,26,etc and do "6 minutes ago".
>
> Good luck!
>
> On 10/31/2012 9:46 AM, Tristan Rhodes wrote:
>> Alan - Actually, I am much more familiar with using the "date" command so I
>> am going to use your method. I also
>> learned that the file name shows 5 minutes before the time the file is
>> created, so I am using the built in -d 5 "mins
>> ago". (Or should I use "10 mins ago" to make sure I am not stepping on
>> nfcapd toes while it is writing the file? If
>> I use 10 mins, my real-time dashboard would have slightly older data...)
>>
>> Mark - Thank you very much for your solution!
>>
>> It looks like I can get the results I want by creating a cronjob to run
>> every 5 minutes (*/5 * * * *) and execute this
>> command:
>>
>> /usr/bin/nfdump -M
>> /usr/local/nfsen/profiles-data/live/Core1:Core2:Core3:Core4 -T -r `date
>> +%C%y/%m/%d/nfcapd.%C%y%m%d%H%M -d "5 mins ago"` -n 10 -s ip/flows
>>
>> Now to parse this output, insert it into a new database, and imagine some
>> useful things to do with the data!
>>
>> Thanks,
>>
>> Tristan
>>
>>
>> --
>> Tristan Rhodes
>> Network Engineer
>> Weber State University
>> (801) 626-8549
>>
>>
>> >>> On 10/30/2012 at 5:30 PM, in message <509062f9.9080...@hawaii.edu>, Alan
>> >>> Whinery <whin...@hawaii.edu> wrote:
>> Oh, well shoot, I went looking for the message with the question and then I
>> answered somebody's better answer.
>>
>> On 10/30/2012 10:53 AM, Mark D. Nagel wrote:
>>> On 10/30/2012 1:25 PM, Tristan Rhodes wrote:
>>>> The first step is to decide how to best export this data from our netflow
>>>> data. My initial idea was to run a nfdump
>>>> command with a cronjob every 5 minutes. The problem is, how do I
>>>> dynamically generate the proper date-based filename?
>>>>
>>>> "nfdump -M
>>>> /usr/local/nfsen/profiles-data/live/Core1:Core2:Core3:Core4:Core5 -T -r
>>>> 2012/10/11/nfcapd.201210111105 -n
>>>> 10 -s ip/flows"
>>>>
>>>> I also tried using the "nfcapd.current" file for this purpose, but I get
>>>> this error:
>>>>
>>>> "nfdump -M /usr/local/nfsen/profiles-data/live/Core1:Core2 -T -r
>>>> nfcapd.current -n 10 -s ip/flows"
>>>> Open file /usr/local/nfsen/profiles-data/live/Core1/nfcapd.current: bad
>>>> version: 0
>>>> Open file /usr/local/nfsen/profiles-data/live/Core2/nfcapd.current: bad
>>>> version: 0
>>>>
>>>>
>>>>
>>>
>>> Pretty sure you can't use nfcapd.current for analysis as it is where new
>>> flows are collected. You would want to back
>>> that down to the previous 5 minute interval. From a cron job, you could do
>>> something like this:
>>>
>>> */5 * * * * nfdump -M
>>> /usr/local/nfsen/profiles-data/live/Core1:Core2:Core3:Core4:Core5 -T -r
>>> $(perl -MPOSIX -e
>>> 'print strftime("%Y/%m/%d/nfcapd.%Y%m%d%M%S", localtime(time - 300))') -n
>>> 10 -s ip/flows
>>>
>>> You may be able to make that shorter, or wrap it in a script, but it should
>>> do the trick.
>>>
>>> Regards,
>>> Mark
>>> --
>>> Mark D. Nagel, CCIE #3177 <mna...@willingminds.com>
>>> Principal Consultant, Willing Minds LLC (http://www.willingminds.com)
>>> cell: 949-279-5817, desk: 714-495-4001, fax: 714-646-8277
>>>
>>> ** For faster support response time, please
>>> ** email supp...@willingminds.com or call 714-495-4000
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Everyone hates slow websites. So do we.
>>> Make your web apps faster with AppDynamics
>>> Download AppDynamics Lite for free today:
>>> http://p.sf.net/sfu/appdyn_sfd2d_oct
>>>
>>>
>>> _______________________________________________
>>> Nfsen-discuss mailing list
>>> Nfsen-discuss@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>
>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
>
>
>
> _______________________________________________
> Nfsen-discuss mailing list
> Nfsen-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>
--
--
Be nice to your netflow data
#!/usr/bin/perl
#
# Copyright (c) 2012, Peter Haag
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# * Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
# * Neither the name of the author nor the names of its contributors may be
# used to endorse or promote products derived from this software without
# specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
# Name of the plugin
package top10;
use strict;
use NfProfile;
use NfConf;
#
# The plugin may send any messages to syslog
# Do not initialize syslog, as this is done by
# the main process nfsen-run
use Sys::Syslog;
# This string identifies the plugin as a version 1.3.0 plugin.
our $VERSION = 130;
my ( $nfdump, $PROFILEDIR );
#
# Periodic data processing function
# input: hash reference including the items:
# 'profile' profile name
# 'profilegroup' profile group
# 'timeslot' time of slot to process: Format yyyymmddHHMM e.g. 200503031200
sub run {
my $argref = shift;
my $profile = $$argref{'profile'};
my $profilegroup = $$argref{'profilegroup'};
my $timeslot = $$argref{'timeslot'};
syslog('debug', "Top10 run: Profilegroup: $profilegroup, Profile: $profile, Time: $timeslot");
my %profileinfo = NfProfile::ReadProfile($profile, $profilegroup);
my $profilepath = NfProfile::ProfilePath($profile, $profilegroup);
my $all_sources = join ':', keys %{$profileinfo{'channel'}};
my $netflow_sources = "$PROFILEDIR/$profilepath/$all_sources";
syslog('debug', "top10 args: $profilepath $all_sources $netflow_sources");
print "top10 args: $profilepath $all_sources $netflow_sources\n";
my $cmd = "$nfdump -M $netflow_sources -r nfcapd.$timeslot -n 10 -s ip/flows -o csv -q";
syslog('debug', "Top10 cmd: $cmd");
my @output = `$cmd`;
print @output;
} # End of run
#
# The Init function is called when the plugin is loaded. It's purpose is to give the plugin
# the possibility to initialize itself. The plugin should return 1 for success or 0 for
# failure. If the plugin fails to initialize, it's disabled and not used. Therefore, if
# you want to temporarily disable your plugin return 0 when Init is called.
#
sub Init {
syslog("info", "top10: Init");
# Init some vars
$nfdump = "$NfConf::PREFIX/nfdump";
$PROFILEDIR = "$NfConf::PROFILEDATADIR";
return 1;
}
#
# The Cleanup function is called, when nfsend terminates. It's purpose is to give the
# plugin the possibility to cleanup itself. It's return value is discard.
sub Cleanup {
syslog("info", "top10 Cleanup");
# not used here
}
1;
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
