Re: [pmacct-discussion] Question about teeing and sampling

[KA PDE]

Hi,

Thanks for clarifying. I guess I was oversimplifying due to lack of
knowledge on the matter.

I will remember this just in case. As we will be sending to a team used to
process sFlow from Juniper MXs and we export both from a mixture of Cisco
devices with different Netflow versions.

Best regards and thanks for your attention.

Regards,

Pau

2016-02-10 18:29 GMT+01:00 Paolo Lucente :

> Hi Pau,
>
> On the sampling part: this is not supported but for a good reason, i
> would say. Sampling is, yes, about sending less data over but also
> about being able to renormalize data using some math; sampling packets
> passing via an interface makes sense; dropping some well-formed NetFlow
> packets (each of which contains several flows, each potentially passing
> via different interfaces, etc.) has less sense.
>
> What you really want is a thing that, supposing for simplicity you are
> sampling 1:1 at your routers, reads NetFlow packets, builds state on a
> per router or more fine-grained basis, samples flows algorithmically
> and constructs brand new NetFlow packets. Opposed to packet sampling
> (where you only need a sampling rate as a multiplication factor), flow
> sampling may require knowledge of how the algorithm does operate, in
> order to (more accurately) renormalize. For example: C7600 (at least
> until RSP720) was doing flow sampling and it took them three pages of
> high level algorithm description (*) to give you a chance to be somehow
> accurate renormalizing sampled data.
>
> Cheers,
> Paolo
>
> (*)
> http://www.cisco.com/c/en/us/td/docs/routers/7600/ios/12-2SR/configuration/guide/swcg/nde.pdf
>
> On Mon, Feb 08, 2016 at 03:27:34PM +0100, KA PDE wrote:
> > Hi all,
> >
> > I've recently discovered pmacct and I'm evaluating it to forward netflow
> > data for security purposes to a set of collectors, some of them requiring
> > less amount of data sent.
> >
> > I have a simple configuration using the tee plugin. I've managed to send
> > flow information to NFsen but I'm unable to find a way of sampling to the
> > other destination.Is this achievable with pmacct?
> >
> > ! nfacctd configuration
> > !
> > !
> > !
> > daemonize: true
> > pidfile: /var/run/nfacctd.pid
> > syslog: daemon
> >
> > nfacctd_port: 9996
> > nfacctd_ip: 88.22.33.99
> > plugin_pipe_size: 1024
> > plugin_buffer_size: 10240
> >
> > plugins: tee[nfsen], tee[pmacct]
> > tee_receiver[nfsen]: 127.0.0.1:9995
> > tee_receiver[pmacct]: 127.0.0.1:
> > ! sampling_rate[pmacct]: 4096
> > tee_transparent: true
> >
> > Thanks in advance and best regards,
> >
> > Pau
>
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[KA PDE]

Nice point here, Mario. Thanks for your insight.

Best regards,

Pau

2016-02-10 12:28 GMT+01:00 Jentsch, Mario <mjent...@cogentco.com>:

> Please note that for template based Netflow versions the destinations need
> them to process data flowsets. Using the samplicator not all destinations
> get all sent template flowsets – it will take some time and re-sending of
> these template flowsets till all destinations received them and understand
> the data flowsets.
>
>
>
> Regards,
>
> Mario
>
>
>
> *From:* pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] *On
> Behalf Of *Jordan Grigorov (Neterra NMT)
> *Sent:* Wednesday, February 10, 2016 10:35 AM
> *To:* pmacct-discussion@pmacct.net
> *Subject:* Re: [pmacct-discussion] Question about teeing and sampling
>
>
>
> Hello Pau,
>
> You can try *samplicate* tool (https://github.com/sleinen/samplicator) to
> forward netflow data to multiple IPs/ports.
>
> Just install it and issue:
>
> *samplicate -s 88.22.33.99 -p 9996 127.0.0.1/9995 <http://127.0.0.1/9995>
> 127.0.0.1/ <http://127.0.0.1/> -f*
>
> Best Regards,
>
>
> ---
> Jordan
>
>
>
> On 8.02.2016 16:27, KA PDE wrote:
>
> Hi all,
>
>
>
> I've recently discovered pmacct and I'm evaluating it to forward netflow
> data for security purposes to a set of collectors, some of them requiring
> less amount of data sent.
>
>
>
> I have a simple configuration using the tee plugin. I've managed to send
> flow information to NFsen but I'm unable to find a way of sampling to the
> other destination.Is this achievable with pmacct?
>
>
>
> ! nfacctd configuration
>
> !
>
> !
>
> !
>
> daemonize: true
>
> pidfile: /var/run/nfacctd.pid
>
> syslog: daemon
>
>
>
> nfacctd_port: 9996
>
> nfacctd_ip: 88.22.33.99
>
> plugin_pipe_size: 1024
>
> plugin_buffer_size: 10240
>
>
>
> plugins: tee[nfsen], tee[pmacct]
>
> tee_receiver[nfsen]: 127.0.0.1:9995
>
> tee_receiver[pmacct]: 127.0.0.1:
>
> ! sampling_rate[pmacct]: 4096
>
> tee_transparent: true
>
>
>
> Thanks in advance and best regards,
>
>
>
> Pau
>
>
>
>
> ___
>
> pmacct-discussion mailing list
>
> http://www.pmacct.net/#mailinglists
>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[Paolo Lucente]

Hi Pau,

On the sampling part: this is not supported but for a good reason, i
would say. Sampling is, yes, about sending less data over but also
about being able to renormalize data using some math; sampling packets
passing via an interface makes sense; dropping some well-formed NetFlow
packets (each of which contains several flows, each potentially passing
via different interfaces, etc.) has less sense. 

What you really want is a thing that, supposing for simplicity you are
sampling 1:1 at your routers, reads NetFlow packets, builds state on a
per router or more fine-grained basis, samples flows algorithmically
and constructs brand new NetFlow packets. Opposed to packet sampling
(where you only need a sampling rate as a multiplication factor), flow
sampling may require knowledge of how the algorithm does operate, in
order to (more accurately) renormalize. For example: C7600 (at least
until RSP720) was doing flow sampling and it took them three pages of
high level algorithm description (*) to give you a chance to be somehow
accurate renormalizing sampled data. 

Cheers,
Paolo 

(*) 
http://www.cisco.com/c/en/us/td/docs/routers/7600/ios/12-2SR/configuration/guide/swcg/nde.pdf

On Mon, Feb 08, 2016 at 03:27:34PM +0100, KA PDE wrote:
> Hi all,
> 
> I've recently discovered pmacct and I'm evaluating it to forward netflow
> data for security purposes to a set of collectors, some of them requiring
> less amount of data sent.
> 
> I have a simple configuration using the tee plugin. I've managed to send
> flow information to NFsen but I'm unable to find a way of sampling to the
> other destination.Is this achievable with pmacct?
> 
> ! nfacctd configuration
> !
> !
> !
> daemonize: true
> pidfile: /var/run/nfacctd.pid
> syslog: daemon
> 
> nfacctd_port: 9996
> nfacctd_ip: 88.22.33.99
> plugin_pipe_size: 1024
> plugin_buffer_size: 10240
> 
> plugins: tee[nfsen], tee[pmacct]
> tee_receiver[nfsen]: 127.0.0.1:9995
> tee_receiver[pmacct]: 127.0.0.1:
> ! sampling_rate[pmacct]: 4096
> tee_transparent: true
> 
> Thanks in advance and best regards,
> 
> Pau

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[Paolo Lucente]

Hi Mario,

Wrt the balancing algorithm & templates. Definitely the round-robin
balancing algorithm is suitable only for - pass me the term - non-
contextual protocols/protocol versions (ie. sFlow and NetFlow v5);
NetFlow v9/IPFIX, which are template-based, require the 'hash-agent'
one where the IP address of the NetFlow v9/IPFIX sender is hashed
over the pool of destinations so to ensure template/data records are
always making together to the same destination (precisely for the
issue you describe). Alternative to balancing is selective teeing,
ie. select which source to replicate to which destination (using
a tag mechanism) - which makes things more controllable especially
in rapidly expanding scenarios.

Cheers,
Paolo

On Wed, Feb 10, 2016 at 11:19:42AM +, Jentsch, Mario wrote:
> Hi Pau,
> 
> it depends on the Netflow version. With versions that use templates it may be 
> the easiest way to ignore the data at the end points that is ???too much???. 
> The problem is that without the templates the receiver can???t process the 
> data.
> 
> For versions without templates have a look at the ???balance-alg??? option in 
> the tee_receivers.lst example. You may send some data into ???blackhole 
> destinations??? to get rid of it.
> 
> I don???t know how Paolo handles balancing for packets that contain Template 
> FlowSet(s) ??? if they are forwarded to all pool destinations or not ??? 
> didn???t test this myself or checked the code trying to find out. In case the 
> templates are forwarded to all destinations in exception to the balancing 
> method, it looks like you can use it for Netflow v9 etc too.
> 
> Regards,
> Mario
> 
> From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On 
> Behalf Of KA PDE
> Sent: Monday, February 08, 2016 3:28 PM
> To: pmacct-discussion@pmacct.net
> Subject: [pmacct-discussion] Question about teeing and sampling
> 
> Hi all,
> 
> I've recently discovered pmacct and I'm evaluating it to forward netflow data 
> for security purposes to a set of collectors, some of them requiring less 
> amount of data sent.
> 
> I have a simple configuration using the tee plugin. I've managed to send flow 
> information to NFsen but I'm unable to find a way of sampling to the other 
> destination.Is this achievable with pmacct?
> 
> ! nfacctd configuration
> !
> !
> !
> daemonize: true
> pidfile: /var/run/nfacctd.pid
> syslog: daemon
> 
> nfacctd_port: 9996
> nfacctd_ip: 88.22.33.99
> plugin_pipe_size: 1024
> plugin_buffer_size: 10240
> 
> plugins: tee[nfsen], tee[pmacct]
> tee_receiver[nfsen]: 127.0.0.1:9995<http://127.0.0.1:9995>
> tee_receiver[pmacct]: 127.0.0.1:<http://127.0.0.1:>
> ! sampling_rate[pmacct]: 4096
> tee_transparent: true
> 
> Thanks in advance and best regards,
> 
> Pau

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[Jordan Grigorov (Neterra NMT)]

Hello Pau,

You can try /samplicate/ tool (https://github.com/sleinen/samplicator) 
to forward netflow data to multiple IPs/ports.


Just install it and issue:

/samplicate -s 88.22.33.99 -p 9996 127.0.0.1/9995 ///127.0.0.1// -f/

Best Regards,



---


   Jordan




On 8.02.2016 16:27, KA PDE wrote:

Hi all,

I've recently discovered pmacct and I'm evaluating it to forward 
netflow data for security purposes to a set of collectors, some of 
them requiring less amount of data sent.


I have a simple configuration using the tee plugin. I've managed to 
send flow information to NFsen but I'm unable to find a way of 
sampling to the other destination.Is this achievable with pmacct?


! nfacctd configuration
!
!
!
daemonize: true
pidfile: /var/run/nfacctd.pid
syslog: daemon

nfacctd_port: 9996
nfacctd_ip: 88.22.33.99
plugin_pipe_size: 1024
plugin_buffer_size: 10240

plugins: tee[nfsen], tee[pmacct]
tee_receiver[nfsen]: 127.0.0.1:9995 
tee_receiver[pmacct]: 127.0.0.1: 
! sampling_rate[pmacct]: 4096
tee_transparent: true

Thanks in advance and best regards,

Pau


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[Jentsch, Mario]

Please note that for template based Netflow versions the destinations need them 
to process data flowsets. Using the samplicator not all destinations get all 
sent template flowsets - it will take some time and re-sending of these 
template flowsets till all destinations received them and understand the data 
flowsets.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jordan Grigorov (Neterra NMT)
Sent: Wednesday, February 10, 2016 10:35 AM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Question about teeing and sampling

Hello Pau,

You can try samplicate tool (https://github.com/sleinen/samplicator) to forward 
netflow data to multiple IPs/ports.

Just install it and issue:

samplicate -s 88.22.33.99 -p 9996 127.0.0.1/9995 127.0.0.1/ -f

Best Regards,


---
Jordan

On 8.02.2016 16:27, KA PDE wrote:
Hi all,

I've recently discovered pmacct and I'm evaluating it to forward netflow data 
for security purposes to a set of collectors, some of them requiring less 
amount of data sent.

I have a simple configuration using the tee plugin. I've managed to send flow 
information to NFsen but I'm unable to find a way of sampling to the other 
destination.Is this achievable with pmacct?

! nfacctd configuration
!
!
!
daemonize: true
pidfile: /var/run/nfacctd.pid
syslog: daemon

nfacctd_port: 9996
nfacctd_ip: 88.22.33.99
plugin_pipe_size: 1024
plugin_buffer_size: 10240

plugins: tee[nfsen], tee[pmacct]
tee_receiver[nfsen]: 127.0.0.1:9995<http://127.0.0.1:9995>
tee_receiver[pmacct]: 127.0.0.1:<http://127.0.0.1:>
! sampling_rate[pmacct]: 4096
tee_transparent: true

Thanks in advance and best regards,

Pau




___

pmacct-discussion mailing list

http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[Markus Weber]

No problem with tee here - but wasn't Pau expecting tee to do further 
sampling (which it doesn't)?


==nfacctd_tee.conf:
pidfile: /pmacct/var/nfacctd_tee.pid
logfile: /pmacct/log/nfacctd_tee.log
daemonize: true
files_umask: 2
nfacctd_disable_checks: true
nfacctd_pipe_size: 8388608
plugin_buffer_size: 204800
plugin_pipe_size: 2048
nfacctd_ip: 
nfacctd_port: 

plugins: tee
tee_receivers: /pmacct/etc/tee_rec.lst
tee_transparent: true


==tee_rec.lst:
id=1ip=:,:

Run nfacctd with nfacctd_tee.conf to duplicate NF data received on 
: to : and :. Then run on 
: your NFsen and on : another nfacctd with 
another config to use pmacct's great features (what ever you want to 
aggregate on or do with the data).


Eventually you might need tee_transparent to be true (or -S with 
samplicator) to keep original source address.



Markus

On 10.02.2016 10:35, Jordan Grigorov (Neterra NMT) wrote:

Hello Pau,

You can try /samplicate/ tool (https://github.com/sleinen/samplicator) 
to forward netflow data to multiple IPs/ports.


Just install it and issue:

/samplicate -s 88.22.33.99 -p 9996 127.0.0.1/9995 ///127.0.0.1// -f/

Best Regards,



---


Jordan




On 8.02.2016 16:27, KA PDE wrote:

Hi all,

I've recently discovered pmacct and I'm evaluating it to forward 
netflow data for security purposes to a set of collectors, some of 
them requiring less amount of data sent.


I have a simple configuration using the tee plugin. I've managed to 
send flow information to NFsen but I'm unable to find a way of 
sampling to the other destination.Is this achievable with pmacct?


! nfacctd configuration
!
!
!
daemonize: true
pidfile: /var/run/nfacctd.pid
syslog: daemon

nfacctd_port: 9996
nfacctd_ip: 88.22.33.99
plugin_pipe_size: 1024
plugin_buffer_size: 10240

plugins: tee[nfsen], tee[pmacct]
tee_receiver[nfsen]: 127.0.0.1:9995 
tee_receiver[pmacct]: 127.0.0.1: 
! sampling_rate[pmacct]: 4096
tee_transparent: true

Thanks in advance and best regards,

Pau


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists




___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[Jentsch, Mario]

Hi Pau,

it depends on the Netflow version. With versions that use templates it may be 
the easiest way to ignore the data at the end points that is “too much”. The 
problem is that without the templates the receiver can’t process the data.

For versions without templates have a look at the ‘balance-alg’ option in the 
tee_receivers.lst example. You may send some data into “blackhole destinations” 
to get rid of it.

I don’t know how Paolo handles balancing for packets that contain Template 
FlowSet(s) – if they are forwarded to all pool destinations or not – didn’t 
test this myself or checked the code trying to find out. In case the templates 
are forwarded to all destinations in exception to the balancing method, it 
looks like you can use it for Netflow v9 etc too.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of KA PDE
Sent: Monday, February 08, 2016 3:28 PM
To: pmacct-discussion@pmacct.net
Subject: [pmacct-discussion] Question about teeing and sampling

Hi all,

I've recently discovered pmacct and I'm evaluating it to forward netflow data 
for security purposes to a set of collectors, some of them requiring less 
amount of data sent.

I have a simple configuration using the tee plugin. I've managed to send flow 
information to NFsen but I'm unable to find a way of sampling to the other 
destination.Is this achievable with pmacct?

! nfacctd configuration
!
!
!
daemonize: true
pidfile: /var/run/nfacctd.pid
syslog: daemon

nfacctd_port: 9996
nfacctd_ip: 88.22.33.99
plugin_pipe_size: 1024
plugin_buffer_size: 10240

plugins: tee[nfsen], tee[pmacct]
tee_receiver[nfsen]: 127.0.0.1:9995<http://127.0.0.1:9995>
tee_receiver[pmacct]: 127.0.0.1:<http://127.0.0.1:>
! sampling_rate[pmacct]: 4096
tee_transparent: true

Thanks in advance and best regards,

Pau
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Question about teeing and sampling

[KA PDE]

Hi all,

I've recently discovered pmacct and I'm evaluating it to forward netflow
data for security purposes to a set of collectors, some of them requiring
less amount of data sent.

I have a simple configuration using the tee plugin. I've managed to send
flow information to NFsen but I'm unable to find a way of sampling to the
other destination.Is this achievable with pmacct?

! nfacctd configuration
!
!
!
daemonize: true
pidfile: /var/run/nfacctd.pid
syslog: daemon

nfacctd_port: 9996
nfacctd_ip: 88.22.33.99
plugin_pipe_size: 1024
plugin_buffer_size: 10240

plugins: tee[nfsen], tee[pmacct]
tee_receiver[nfsen]: 127.0.0.1:9995
tee_receiver[pmacct]: 127.0.0.1:
! sampling_rate[pmacct]: 4096
tee_transparent: true

Thanks in advance and best regards,

Pau
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists