Depending on what you want to use the netflow data for, sampling could be
low or high. Low sampling gives you more accurate data for a specific flow,
high sampling can give you some average data for the whole box. The more
details you want, the lower the sampling has to be.
Keep in mind one more fact - newer versions of nfsen do "sampling
correction" - meaning, it can detect the sampling rate (it's normally
exported by the router), and adjusts the flow values according to this
sample rate.
In your case, I would say that nfsen received a flow record with only 2
packets, with duration 16ms and based on sampling 1:1000 it "adjusted" it
to 2000 packets. I'm pretty sure the traffic volume is adjusted as well.
This can be misleading for small traffic values, but you can generally
exclude these by filtering for flows with duration > 1000 (at least 1
second).
To disable this sampling correction, you would need to start your collector
with -s -1 parameter (set sampling to 1), but your graphs would probably be
1000 times smaller in values. :)
@Peter: I know sampling is tricky, but I've noticed an option in a
competing netflow product - Arbor Peakflow - that can get better results.
They also read the sampling values exported by the router, but make
periodic SNMP queries to read the traffic values on exporting interfaces.
They then try to see if the netflow traffic seen on router X, interface Y
matches the SNMP traffic for the same router and interface. If the ratio is
close to 1:1, then their sampling correction is ok. If it's offset, either
they are not getting the whole netflow traffic for that interface, or they
are not correcting it correctly. I think that in this case, they
dynamically change the sampling rate in their corrections in order to make
the two readings match...
This would involve quite a few changes in nfsen, and would probably annoy
router administrators (nobody wants yet another management app to read SNMP
values from an overloaded router), but might be something worth considering
in the future if this gets out of hand.
Regards,
Adrian
On Thu, Jan 10, 2013 at 12:12 AM, Jason Lixfeld <
jason-nfsen-disc...@lixfeld.ca> wrote:
>
> On 2013-01-07, at 2:03 AM, Adrian Popa <adrian.popa...@gmail.com> wrote:
>
> > If you are worried instead about the low volume of traffic seen from
> this AS, keep in mind the following:
> > 1. You are probably using sampling on your router. NFSEN accounts for
> sampling and tries to guesstimate some of the values.
>
> I am sampling. 1:1000.
>
> Maybe I don't quite understand sampling. Sampling doesn't quelch the
> number of flow records exported to the collector, it quelches the number of
> packets that are processed by the device in order to create the flow
> record. Is that accurate?
>
> So I just re-ran the math from the output below. Let's take this one for
> argument's sake:
>
> 2013-01-03 10:10:43.424 0.016 any 30513 2( 0.0)
> 2000( 0.0) 3.0 M( 0.0) 125000 1.5 G 1500
>
> So what that is saying is that the statistic entry for AS30513 was first
> seen on 2013-01-03 10:10:43.424, consists of 16ms worth of data where 2
> flows totalling 3MB of data volume spread across 2000 packets was collected
> within those 16ms. The flow records have no knowledge of pps, bps or bpp,
> so nfdump calculates those values based on the data that it knows about;
> time (16ms), volume (3MB) and total number of packets based on the exported
> flow records received by nfcapd.
>
> So if this is true, then trying to use bps as a statistic orderby will
> never provide you with decent results because those values are calculated
> based on data that might have been quelched based on the way the sampling
> works.
>
> If this is correct, it seems to me like sampling is bad (but I can't
> actually not sample or else my routers drop netflow packets; they can only
> handle 100k across the entire box), but I understand why it exists. So if
> sampling is the root cause of all these "bad" calculations, it would stand
> to reason that one should set the sampling rate as close to 1:1 as possible?
>
> > 2. You may have some spoofed traffic in your network that sends few
> packets (hence the very short duration), but because of sampling, you get a
> high count of packets (and usually this is a "round" number).
> >
> > On Sat, Jan 5, 2013 at 9:44 AM, Peter Haag <ph...@users.sourceforge.net>
> wrote:
> > Hi Jason,
> > Looking at your output, I can not find something weird. Please keep in
> mind:
> > Each flow has two ASes, so and so see on how many flows these ASes
> appear.
> > Your second example makes it clear: You filter for 'as 30513' which
> results
> > in two flows - AS 30513 <-> AS 0. AS 0 means the exporting router has no
> AS
> > info. These resulting two flows are now ordered by AS and by bps as
> requested.
> > Each AS appears in each flow -> in 100% of all flows.
> >
> > The same math is now applied for your first run. But you only have the
> flows
> > of the first top 10 ASes by bps. In % the digits are way below what can
> be
> > displayed. You may also use -N to prevent scaling (K, M, G, T) in order
> to
> > see the actual number. To sum up, you would need to output of all seen
> ASes
> > -n 0 .
> >
> > Hope, this helps, otherwise let me know, if I can help
> >
> > - Peter
> >
> > On 4/1/13 5:20 PM, Jason Lixfeld wrote:
> > > Hi there,
> > >
> > > So I'm just playing around with my first 36 hours worth of data and
> I'm seeing some stuff that looks sort of off:
> > >
> > > ** nfdump -M
> /opt/nfsen/profiles-data/live/bfr01-hudson:bfr01-mowat:bfr01-front -T -R
> 2013/01/02/nfcapd.201301022305:2013/01/04/nfcapd.201301041055 -n 10 -s
> as/bps
> > > nfdump filter:
> > > any
> > > Top 10 AS ordered by bps:
> > > Date first seen Duration Proto AS Flows(%)
> Packets(%) Bytes(%) pps bps bpp
> > > 2013-01-02 22:39:46.290 130797.681 any 0 21.1
> M(85.9) 42.2 G(87.5) 30.0 T(88.5) 322585 1.8 G 710
> > > 2013-01-03 10:10:43.424 0.016 any 30513 2(
> 0.0) 2000( 0.0) 3.0 M( 0.0) 125000 1.5 G 1500
> > > 2013-01-03 08:53:20.734 0.015 any 37957 2(
> 0.0) 2000( 0.0) 1.5 M( 0.0) 133333 810.7 M 760
> > > 2013-01-04 10:23:02.606 0.017 any 35414 2(
> 0.0) 2000( 0.0) 1.5 M( 0.0) 117647 727.5 M 773
> > > 2013-01-03 14:25:51.067 0.017 any 33428 2(
> 0.0) 2000( 0.0) 1.5 M( 0.0) 117647 692.7 M 736
> > > 2013-01-03 13:37:35.176 0.039 any 46676 1(
> 0.0) 2000( 0.0) 2.8 M( 0.0) 51282 582.6 M 1420
> > > 2013-01-04 00:43:04.529 0.048 any 15347 1(
> 0.0) 2000( 0.0) 2.8 M( 0.0) 41666 473.3 M 1420
> > > 2013-01-03 15:58:33.535 0.077 any 47045 1(
> 0.0) 3000( 0.0) 4.3 M( 0.0) 38961 442.6 M 1420
> > > 2013-01-02 23:02:16.952 129445.016 any 22822 4.0
> M(16.2) 8.9 G(18.5) 6.4 T(19.0) 68835 398.2 M 723
> > > 2013-01-03 14:52:54.865 0.031 any 19354 2(
> 0.0) 2000( 0.0) 1.5 M( 0.0) 64516 379.9 M 736
> > >
> > > Summary: total flows: 24583165, total bytes: 33.9 T, total packets:
> 48.2 G, avg bps: 2.1 G, avg pps: 368688, avg bpp: 702
> > > Time window: 2013-01-02 22:39:34 - 2013-01-04 10:59:43
> > > Total flows processed: 24583165, Blocks skipped: 0, Bytes read:
> 2261849088
> > > Sys: 8.970s flows/second: 2740403.8 Wall: 10.563s flows/second:
> 2327242.5
> > >
> > > Lines 1 and 9 seem OK, but lines 2-8,10 look really weird; the math
> just doesn't add up.
> > >
> > > If I filter specifically on AS 30513:
> > >
> > > ** nfdump -M
> /opt/nfsen/profiles-data/live/bfr01-hudson:bfr01-mowat:bfr01-front -T -R
> 2013/01/02/nfcapd.201301022305:2013/01/04/nfcapd.201301041055 -n 10 -s
> as/bps
> > > nfdump filter:
> > > AS 30513
> > > Top 10 AS ordered by bps:
> > > Date first seen Duration Proto AS Flows(%)
> Packets(%) Bytes(%) pps bps bpp
> > > 2013-01-03 10:10:43.424 0.016 any 0
> 2(100.0) 2000(100.0) 3.0 M(100.0) 125000 1.5 G 1500
> > > 2013-01-03 10:10:43.424 0.016 any 30513
> 2(100.0) 2000(100.0) 3.0 M(100.0) 125000 1.5 G 1500
> > >
> > > Summary: total flows: 2, total bytes: 3.0 M, total packets: 2000, avg
> bps: 1.5 G, avg pps: 125000, avg bpp: 1500
> > > Time window: 2013-01-02 22:39:34 - 2013-01-04 10:59:43
> > > Total flows processed: 24583165, Blocks skipped: 0, Bytes read:
> 2261849088
> > > Sys: 7.574s flows/second: 3245367.9 Wall: 8.594s flows/second:
> 2860278.3
> > >
> > > I have no idea how to even begin going about troubleshooting this, so
> any thoughts are welcomed.
> > >
> > > Thanks again in advance.
> > >
> ------------------------------------------------------------------------------
> > > Master HTML5, CSS3, ASP.NET, MVC, AJAX, Knockout.js, Web API and
> > > much more. Get web development skills now with LearnDevNow -
> > > 350+ hours of step-by-step video tutorials by Microsoft MVPs and
> experts.
> > > SALE $99.99 this month only -- learn more at:
> > > http://p.sf.net/sfu/learnmore_122812
> > > _______________________________________________
> > > Nfsen-discuss mailing list
> > > Nfsen-discuss@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
> > >
> >
> > --
> > Be nice to your netflow data. Use NfSen and nfdump :)
> >
> >
> ------------------------------------------------------------------------------
> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> > MVPs and experts. SALE $99.99 this month only -- learn more at:
> > http://p.sf.net/sfu/learnmore_122912
> > _______________________________________________
> > Nfsen-discuss mailing list
> > Nfsen-discuss@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
> >
>
>
------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
