It looks like I have been bitten by the same behavior as described in
http://sourceforge.net/p/nfdump/bugs/25/ and
https://sourceforge.net/p/nfdump/bugs/22/. So I guess Peter understands the
problem but has decided not to do anything.
Ok.
In the mean time, the patch in
https://sourceforge.net/p/nfdump/bugs/22/can be used to achieve what
I'm trying to achieve. (It fails because there
are whitespaces differences since bugs/22/ until 1.6.11, but the logic is
still the same.)
Peter, if someone were to introduce two new options, e.g. -ts and -te that
behave like -t but operate only on start and end, would you be open to
accepting such a patch?
Sincerely,
Another :-) Peter
On Fri, Nov 29, 2013 at 12:12 AM, Peter Valdemar Mørch <pe...@morch.com>wrote:
> Hi,
>
> I want to create statistics for every 5-minute interval based on a filter
> for use with something similar to RRDTool and I'd like to use nfdump to get
> it.
>
> The simple solution is to just use the appropriately named 5-minute file
> nfcapd.201311281820, and run a filter on that. But I'm curious as to
> whether I'm missing a way to do this with the -t option or some other
> option.
>
> Naively, I tried -t 2013/11/28.18:10-2013/11/28.18:15 to get statistics
> about all the data in the 18:10 - 18:15 interval. But that returned zero
> flows. It looks like nfdump only includes flows where the entire period is
> within the to-from times in -t. See "details" below. (Would be nice if man
> nfdump went into a little more detail about -t)
>
> But, if some flows are short lived (e.g. webserver) and some are
> long-lived (e.g. ssh-connection) I don't see how I can use the -t option to
> get an idea of how much traffic occurred between 18:10 and 18:15.
>
> I guess I was hoping for a -t option that looked exclusively at e.g. flow
> end, or some way to use flow-end in a filter. Then I could get an idea
> about traffic in a 5minute period. Where long-running flows would be
> calculated as-if they occurred entirely at the flow-end time, but this is
> exactly what you get by looking at one nfcapd.* file at a time, isn't it?
>
> The Rolls Royce would be that if a flow ran from 18:09-18:13, 75% of the
> traffic from that flow would be added to the 18:10-18:15, because 75% of
> the time is within that period, but hey, I can see that's a little wild.
>
> I also tried to experiment with +/-10, but could not get this to work at
> all.
>
> What we *can* do is use only a single time for -t as in "-t
> 2013/11/28.18:05" and then look at every flow and discard any that end
> outside the 18:10-18:15 interval. But that is very time consuming
> especially because one needs to comparisons on time strings such as e.g.
> "2013-11-28 18:27:46.362". (It would be nice if there was something similar
> to -N that printed times as unixtime, so the heavy conversion (also in
> nfdump?) isn't necessary.)
>
> Can -t be used to get data from 18:10-18:15? Am I missing something (else)
> obvious? ( Appart from "just use nfsen" - I have other reasons not to, and
> I'm trying to understand -t in nfdump )
>
> Sincerely,
>
> Peter
>
> ==========
> Details
> ==========
>
> I have a test nfcapd file[1], with NetFlow records from nothing but a
> single long-running ssh connection.
>
> Looking at all the data in the file, I get this:
>
> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp'
> Duration Date flow start Date flow end Src IP
> Addr Src Pt Dst IP Addr Dst Pt
> <snip>
> 300.716 2013-11-28 17:56:31.752 2013-11-28 18:01:32.468 1.2.3.4:
> 2222 172.22.216.119: 43654
> 300.710 2013-11-28 18:01:42.485 2013-11-28 18:06:43.195 1.2.3.4:
> 2222 172.22.216.119: 43654
> 300.796 2013-11-28 18:06:53.207 2013-11-28 18:11:54.003 1.2.3.4:
> 2222 172.22.216.119: 43654
> 310.813 2013-11-28 18:12:04.019 2013-11-28 18:17:14.832 1.2.3.4:
> 2222 172.22.216.119: 43654
> <snip>
> Summary: total flows: 21, total bytes: 12.1 M, total packets: 33103, avg
> bps: 14730, avg pps: 5, avg bpp: 364
> Time window: 2013-11-28 17:20:09 - 2013-11-28 19:09:22
> Total flows processed: 21, Blocks skipped: 0, Bytes read: 1168
> Sys: 0.008s flows/second: 2625.0 Wall: 0.000s flows/second: 30882.4
>
> And experimenting with the -t option, I see that
> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t
> 2013/11/28.18:12:04-2013/11/28.18:17:14
> is the tightest I can go with -t and still get any data around that time
> period.
>
> Assuming I know that the NetFlow collector transmits every 5 minutes, I
> guess I could do -t 18:10-18:20 and then know that I'll likely only get one
> flow record from the long-running ssh connection, and then in the next
> period use 18:15-18:25. That would work most of the time (tm) for the ssh
> connection. But any short lived flows e.g. entirely inside 18:18 will be
> counted in both intervals. :-(
>
> About +/-10: How am I supposed to use this?
>
> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t +10
> Time Window error: No time slot information available
> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t -10
> Time Window error: No time slot information available
> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t
> 2013/11/28.18:12:04+10
> Time format error at '04+10': unexpected character: '+'.
> > nfdump -r singleSSH.nfcapd -o 'fmt: %td %ts %te %sa:%sp %da:%dp' -t
> 2013/11/28.18:12:04-10
> Time format error: '10' unexpected.
>
> 1: attached and at http://ge.tt/5Rthyl41/v/0?c
>
> --
> Peter Valdemar Mørch
> http://www.morch.com
>
--
Peter Valdemar Mørch
http://www.morch.com
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
_______________________________________________
Nfdump-discuss mailing list
Nfdump-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
