Hi Stathis, Sending packets continuously would not be scalable: hence the idea behind the NetFlow protocol is to report flows, have an expiration mechanism, and batching in a single exported datagram as may flows as they fit. But, granted a small delay between end/timeout of a flow and export of the same is acceptable to your project, nfacctd will place the flow in the right time-bin (read: output file, SQL table), ie. defining "sql_history 1m" or "sql_history: 5m", which, in combination with the aggregation method we described in previous emails, would let you achieve precisely what you say.
So main question here is how much you need this to be real-time. Granted aggregation, historical accouting and time-binning are features offered by pmacct. Cheers, Paolo On Thu, Feb 06, 2014 at 03:18:26PM +0200, Stathis Gkotsis wrote: > Hi Paolo, > Thanks again. > I experimented with this setup running one instance of Netflow probe and one > instance of Netflow collector. What is not so clear to me is when the netflow > probe transmits information to the collector. I see that the netflow probe > sends flow data to the collector when the TCP session is finished (after > certain configurable timeouts). Is it possible for the probe to transmit this > data in a continuous way? > In the end, what I want to achieve is have 5min. (or 1min.) files with > aggregated data (packets, bytes) for each TCP quintuple (TCP session). On top > of that, I would like to be able to identify the true start and end timestamp > for each session. What could help in this direction is that, the first time a > record appears in the files the start timestamp is the start time of the > corresponding TCP session, while the last time that a record appears in the > file the end_timestamp is the end time of the session. Is it possible to do > this? > BR,Stathis > > Date: Tue, 4 Feb 2014 17:01:50 +0000 > > From: [email protected] > > To: [email protected] > > Subject: Re: [pmacct-discussion] TCP connection start timestamp and duration > > > > Hi Stathis, > > > > Inline: > > > > On Tue, Feb 04, 2014 at 02:09:05PM +0200, Stathis Gkotsis wrote: > > > > > I am mainly interested in TCP. I would like the final export to contain > > > one line per src_host,dst_host,src_port,dst_port,proto combination, along > > > with the start timestamp of the corresponding TCP connection (e.g. > > > timestamp of the SYN packet) and the end timestamp of the session if it > > > has ended (e.g. timestamp of the FIN or RST packet). Is this possible? > > > > As i was saying, yes. > > > > > Which timestamps are the timestamp_start and timestamp_end in the case > > > of connectionless transport protocols, e.g. UDP? > > > > timestamp_start intuitively represents the first packet captured > > of a connectionless transport protocol flow. timestamp_end is set > > basing on timeouts. You can check and/or re-set timeout values by > > looking at the nfprobe_timeouts config directive description in > > the CONFIG-KEYS doc. > > > > Cheers, > > Paolo > > > > > > > > Date: Mon, 3 Feb 2014 23:09:20 +0000 > > > > From: [email protected] > > > > To: [email protected] > > > > Subject: Re: [pmacct-discussion] TCP connection start timestamp and > > > > duration > > > > > > > > Hi Stathis, > > > > > > > > You do not outline what is the capturing method you intend to > > > > use, ie. libpcap, NetFlow/IPFIX, sFlow, etc. If using NetFlow/ > > > > IPFIX you are sorted already, as you just add timestamp_start > > > > and timestamp_end to your aggregation method to the quintuple. > > > > > > > > If using libpcap, well, a NetFlow probe helps precisely with > > > > creating flows out of sniffed packets. pmacct has a 'nfprobe' > > > > plugin for this. So the solution would be: pmacctd sniffs on > > > > an interface and is configured with a 'nfprobe' plugin that > > > > exports flows to a nfacctd daemon (co-located on the same box > > > > or on a different box) which, in turn, is configured to save > > > > data to the preferred backend and aggregate on the quintuple > > > > plus timestamp_start, timestamp_end. > > > > > > > > If using sFlow you might have an issue capturing the flags, > > > > depending on how heavily you sample. Let me know if you are > > > > in this case. > > > > > > > > Cheers, > > > > Paolo > > > > > > > > On Mon, Feb 03, 2014 at 11:28:55PM +0200, Stathis Gkotsis wrote: > > > > > Hi, > > > > > Let's say we configure pmacct to aggregate on: src ip, src port, dst > > > > > ip, dst port, proto. That means that it will produce flow records > > > > > aggregating on the TCP quintuple. > > > > > Would it be possible to get the start timestamp (time of TCP SYN) of > > > > > a TCP connection? Similarly, would it be possible to get the duration > > > > > of a connection (possibly the timestamp of FIN)? Is any of these > > > > > things possible through pmacct? > > > > > Thank you. > > > > > > > > > > > > > > _______________________________________________ > > > > > pmacct-discussion mailing list > > > > > http://www.pmacct.net/#mailinglists > > > > > > > > > > > > _______________________________________________ > > > > pmacct-discussion mailing list > > > > http://www.pmacct.net/#mailinglists > > > > > > > > > > > _______________________________________________ > > > pmacct-discussion mailing list > > > http://www.pmacct.net/#mailinglists > > > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
