Re: [pmacct-discussion] sampling

[Paolo Lucente]

Hi Steve,

As Tim was mentioning you can use 'sampling_rate' with a fixed sample
rate (not 'sample_rate'); this will make pmacct sample for you. In case
you are sampling outside pmacct, you can supply this info via, say,
pmacctd_ext_sampling_rate so that nfprobe will include such value in
the NetFlow/IPFIX stream (so that you can then renormalize with the
nfacctd_renormalize config directive at the receiving end).

Cheers,
Paolo

On Wed, Aug 24, 2016 at 11:06:27AM -0400, Stephen Clark wrote:
> Hi Tim,
> 
> My question is can nfprobe from pmacct package do the sampling at
> the origination of the flow.
> 
> Thanks,
> Steve
> 
> On 08/24/2016 10:55 AM, Tim Jackson wrote:
> >If the probe is doing sampling, you can have pmacct re-normalize
> >sampling via either a static sampling value, or via the flow/ipfix
> >information that tells it how it was sampled..
> >
> >sample_rate: 
> >sample_map: 
> >(s|n|p|u)facctd_renormalize:true
> >
> >If you're using a tap that samples traffic, you can tell the
> >daemon that upstream traffic is sampled by X:
> >
> >pmacctd_ext_sampling_rate | uacctd_ext_sampling_rate
> >
> >http://wiki.pmacct.net/OfficialConfigKeys
> >
> >--
> >Tim
> >
> >
> >On Wed, Aug 24, 2016 at 9:37 AM, Stephen Clark
> >> wrote:
> >
> >Hi Paolo,
> >
> >I looked thru the CONFIG_KEYS and didn't find the ability to do sampling
> >except
> >in the SQL_preprocess keys. Is it possible to do the sampling at the 
> > point
> >the neflow records are first created - in other words by nfprobe?
> >
> >Thanks,
> >Steve
> >
> >--
> >
> >"They that give up essential liberty to obtain temporary safety,
> >deserve neither liberty nor safety."  (Ben Franklin)
> >
> >"The course of history shows that as a government grows, liberty
> >decreases."  (Thomas Jefferson)
> >
> >
> >
> >___
> >pmacct-discussion mailing list
> >http://www.pmacct.net/#mailinglists 
> >
> >
> 
> 
> -- 
> 
> "They that give up essential liberty to obtain temporary safety,
> deserve neither liberty nor safety."  (Ben Franklin)
> 
> "The course of history shows that as a government grows, liberty
> decreases."  (Thomas Jefferson)
> 
> 

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] sampling

[Tim Jackson]

If the probe is doing sampling, you can have pmacct re-normalize sampling
via either a static sampling value, or via the flow/ipfix information that
tells it how it was sampled..

sample_rate: 
sample_map: 
(s|n|p|u)facctd_renormalize:true

If you're using a tap that samples traffic, you can tell the daemon that
upstream traffic is sampled by X:

pmacctd_ext_sampling_rate | uacctd_ext_sampling_rate

http://wiki.pmacct.net/OfficialConfigKeys

--
Tim


On Wed, Aug 24, 2016 at 9:37 AM, Stephen Clark 
wrote:

> Hi Paolo,
>
> I looked thru the CONFIG_KEYS and didn't find the ability to do sampling
> except
> in the SQL_preprocess keys. Is it possible to do the sampling at the point
> the neflow records are first created - in other words by nfprobe?
>
> Thanks,
> Steve
>
> --
>
> "They that give up essential liberty to obtain temporary safety,
> deserve neither liberty nor safety."  (Ben Franklin)
>
> "The course of history shows that as a government grows, liberty
> decreases."  (Thomas Jefferson)
>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] sampling incoming data

[Paul Lockaby]

Thanks for the response and especially the link. If I'm reading that thread 
correctly, it seems that the sampling_rate option in nfacct will only work if I 
use a non-template based flow export like Netflow v5.

I'll double check with the router folks on Monday but my understanding so far 
is that they can only export IPFIX to one destination with the version of Junos 
we've got running.

Thanks again,
-Paul

> On Feb 21, 2016, at 4:03 PM, Paolo Lucente  wrote:
> 
> 
> Hi Paul,
> 
> Your understanding is correct; please note that by doing so you
> will achieve flow sampling at the collector (as opposed to packet
> sampling at the router). Although slightly different, please read
> a few thoughts on flow sampling in this recent thread:
> 
> https://www.mail-archive.com/pmacct-discussion@pmacct.net/msg02974.html
> 
> If possible, i would recommend to export twice from the routers,
> one export with a 1:1 sampler for your colleagues, the other with
> the sampler matching your needs for you.
> 
> Cheers,
> Paolo
> 
> On Sat, Feb 20, 2016 at 03:35:43AM +, Paul Lockaby wrote:
>> Hello everyone,
>> 
>> The folks that maintain the routers want to start exporting IPFIX at a 1:1 
>> ratio so they can feed it into their network monitoring system. I, on the 
>> other hand, can't process 1:1 quickly enough to aggregate billing 
>> information in a timely manner. I *think* the "sampling_rate" option will 
>> allow nfacctd to sample the incoming data and write CSV files that are 
>> sampling of what is coming in, if I'm reading the docs correctly. But I want 
>> to make sure that it does by asking around to see. Am I reading the docs 
>> correctly?
>> 
>> Thank you,
>> -Paul
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] sampling incoming data

[itriA30110]

1 on 1 ratio is a crisis to database in busy data center. You'll need to define 
a level of granularity . And yes sampling_rate is for that purpose.

Sent from my ASUS

 原始郵件 
寄件者：Paul Lockaby
傳送日期：Sat, 20 Feb 2016 11:46:02 +0800
收件者：pmacct-discussion@pmacct.net
主旨:[pmacct-discussion] sampling incoming data

Hello everyone,

The folks that maintain the routers want to start exporting IPFIX at a 1:1 
ratio so they can feed it into their network monitoring system. I, on the other 
hand, can't process 1:1 quickly enough to aggregate billing information in a 
timely manner. I *think* the "sampling_rate" option will allow nfacctd to 
sample the incoming data and write CSV files that are sampling of what is 
coming in, if I'm reading the docs correctly. But I want to make sure that it 
does by asking around to see. Am I reading the docs correctly?

Thank you,
-Paul
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists



本信件可能包含工研院機密資訊，非指定之收件者，請勿使用或揭露本信件內容，並請銷毀此信件。 
This email may contain confidential information. Please do not use or disclose 
it in any way and delete it if you are not the intended recipient.___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Sampling Rate for NetFlow v9 Records

[Paolo Lucente]

Hi Steve,

Your config looks good. I suggest to check the NetFlow v9 export at
your router. One popular issue is that the options packet to specify
the sampling rate are not exported. Besides, especially on slightly
older software releases (i here refer particularly to IOS and JunOS,
feel to exclude IOS-XR, don't know others), this was an area fertile 
of bugs.

I will be happy to assist, in case you want an additional pair of eyes.
We can switch to 1:1 emails and you can send me a brief capture of your
NetFlow traffic.

Cheers,
Paolo


On Wed, Feb 04, 2015 at 05:26:09PM +, Steve Gerteisen wrote:
 I've got two devices exporting NetFlow v9 records to my nfacctd collector.  
 For some reason the sampling rate (1:1000) isn't getting expanded, resulting 
 in low pps/bps calcs.  To compensate in the meantime, I've hardcoded my 
 sample rate into my pps/bps calculations in the Perl script scraping the data 
 from the IMT.  Would be curious for some feedback from the list on why this 
 might be happening.  Version info, config file and sample data below.  Any 
 assistance would be greatly appreciated, thanks!
 
 #!--- version info ---!#
 
 [user@host ~]# nfacctd -V
 NetFlow Accounting Daemon, nfacctd 1.5.1-cvs (20141119-00)
  --enable-jansson
 
 For suggestions, critics, bugs, contact me: Paolo Lucente pa...@pmacct.net.
 [user@host ~]#
 
 #!--- nfacctd config file ---!#
 
 daemonize: true
 nfacctd_port: 5678
 plugins: memory[full]
 aggregate[full]: tag, tag2, in_iface, out_iface, peer_src_as, peer_dst_as, 
 src_host, dst_host, proto, src_port, dst_port, tcpflags, ext_comm, 
 src_ext_comm, tos, timestamp_start, timestamp_end
 refresh_maps: true
 nfacctd_renormalize: true
 nfacctd_time_new: true
 
 imt_path[full]: /opt/flows_to_es/cache/full.pipe
 
 pre_tag_map: /opt/pmacct/etc/pretag.map
 
 plugin_pipe_size: 131072000
 plugin_buffer_size: 6400
 imt_buckets: 65537
 imt_mem_pools_size: 1024000
 imt_mem_pools_number: 160
 
 #!--- sample data ---!#
 Duration: $VAR1 = 81;
 Bytes: $VAR1 = 105;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 1500;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 1500;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 52;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 1500;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 96;
 Packets: $VAR1 = 2;
 Bytes: $VAR1 = 1400;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 55;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 329;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 440;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 52;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 52;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 1500;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 1500;
 Packets: $VAR1 = 1;
 Bytes: $VAR1 = 152;
 Packets: $VAR1 = 1;
 
 -- 
 Steve M. Gerteisen
 steve.gertei...@metronetinc.com
 
 
 ___
 pmacct-discussion mailing list
 http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists