Re: [pmacct-discussion] pmact to netflow collector

2018-12-18 Thread Edvinas K 
thanks,

I tried to change the pipe size. As i noticed my OS (centos) default and
max size are the same:

prod [root@netvpn001prpjay pmacct-1.7.2]# cat
/proc/sys/net/core/[rw]mem_default
212992
212992

prod [root@netvpn001prpjay pmacct-1.7.2]# cat /proc/sys/net/core/[rw]mem_max
212992
212992

I tried to set the pmacctd_pipe_size: to 20  and later to 212992.
Seems the drops is still occuring.
Tomorrow i will try to look at that PF_RING thing.

Thanks





On Tue, Dec 18, 2018 at 5:32 PM Paolo Lucente  wrote:

>
> Hi Edvinas,
>
> Easier thing first, i recommend to inject some test traffic and see that
> one how it looks like.
>
> The dropped packets highlight a buffering issue. You could take an
> intermediate step and see if enlarging buffers helps. Configure
> pmacctd_pipe_size to 20 and follow instructions here for the
> /proc files to touch:
>
> https://github.com/pmacct/pmacct/blob/1.7.2/CONFIG-KEYS#L203-#L216
>
> If it helps, good. If not: you should really look into one of the
> frameworks i was pointing you to in my previous email. PF_RING, for
> example, can do sampling and/or balancing. Sampling should not be done
> inside pmacct because the dropped packets are between the kernel and the
> application.
>
> Paolo
>
> On Mon, Dec 17, 2018 at 02:52:48PM +0200, Edvinas K wrote:
> > Seems there're lots of dropped packets:
> >
> > prod [root@netvpn001prpjay pmacct-1.7.2]# pmacctd -i ens1f0.432 -f
> > flowexport.cfg
> > WARN: [flowexport.cfg:2] Invalid value. Ignored.
> > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> > 1.7.2-git (20181018-00+c3)
> > INFO ( default/core ):  '--enable-l2' '--enable-ipv6' '--enable-64bit'
> > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> > '--enable-st-bins'
> > INFO ( default/core ): Reading configuration file
> > '/opt/pmacct-1.7.2/flowexport.cfg'.
> > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally
> based
> > on softflowd 0.9.7 software, Copyright 2002 Damien Miller <
> d...@mindrot.org>
> > All rights reserved.
> > INFO ( default_nfprobe/nfprobe ):   TCP timeout: 3600s
> > INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> > INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> > INFO ( default_nfprobe/nfprobe ):   UDP timeout: 300s
> > INFO ( default_nfprobe/nfprobe ):  ICMP timeout: 300s
> > INFO ( default_nfprobe/nfprobe ):   General timeout: 3600s
> > INFO ( default_nfprobe/nfprobe ):  Maximum lifetime: 604800s
> > INFO ( default_nfprobe/nfprobe ):   Expiry interval: 60s
> > INFO ( default_nfprobe/nfprobe ): Exporting flows to
> > [10.3.14.101]:rtcm-sc104
> > INFO ( default/core ): [ens1f0.432,0] link type is: 1
> > ^C^C^C^C^C^C^C^C
> >
> > after 1 minute:
> >
> > WARN ( default_nfprobe/nfprobe ): Shutting down on user request.
> > INFO ( default/core ): OK, Exiting ...
> > NOTICE ( default/core ): +++
> > NOTICE ( default/core ): [ens1f0.432,0] received_packets=3441854
> > *dropped_packets=2365166*
> >
> > About 1GB of traffic is passing through the router where i'm capturing
> the
> > packets. Isn't it too much traffic for nfrpobe to process ? CPUs seems
> not
> > in 100% usage. We're using  Intel Xeon E5-2620 0 @ 2.00GHz
> > <
> http://netmon.adform.com/device/device=531/tab=health/metric=processor/processor_id=1466/
> >
> > x
> > 24.
> >
> > prod [root@netvpn001prpjay ~]# ps -aux | grep pmacct
> > root 41840 30.9  0.0  18964  7760 ?Rs   Dec14 1309:50
> pmacctd:
> > Core Process [default]
> > root 41841 *68.4%*  0.0  22932  9756 ?RDec14 2898:29
> > pmacctd: Netflow Probe Plugin [default_nfprobe]
> > root 41869 32.5  0.0  19360  8128 ?Ss   Dec14 1378:29
> pmacctd:
> > Core Process [default]
> > root 41870 *67.6%* 0.0  22928  9760 ?RDec14 2865:35
> > pmacctd: Netflow Probe Plugin [default_nfprobe]
> >
> > Before starting with your mentioned 'steroid' things, i would like to
> ask,
> > is't really worth to go to that kernel "things", or start with techniques
> > for example like sampling, or like Nikola recommended try to fidle with
> > nfprobe_engine settings ?
> >
> > Thanks
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Sun, Dec 16, 2018 at 6:25 PM Paolo Lucente  wrote:
> >
> > >
> > > Hi Edvinas,
> > >
> > > You may want to check whether libpcap is dropping packets on input to
> > > pmacctd. You can achieve that sending a SIGUSR1 and checking the output
> > > in the logfile/syslog/console. You will get something a-la:
> > >
> > > https://github.com/pmacct/pmacct/blob/master/docs/SIGNALS#L16-#L34
> > >
> > > Should amount of dropped packets be non-zero and visibly increasing
> then
> > > you may want to put your libpcap on steroids:
> > >
> > > https://github.com/pmacct/pmacct/blob/master/FAQS#L71-#L101
> > >
> > > Should, instead, that not be the case, i am unsure and would need
> > > further investigation. You could try to produce a controlled st

Re: [pmacct-discussion] pmact to netflow collector

2018-12-18 Thread Paolo Lucente 


Hi Edvinas,

Easier thing first, i recommend to inject some test traffic and see that
one how it looks like.

The dropped packets highlight a buffering issue. You could take an
intermediate step and see if enlarging buffers helps. Configure
pmacctd_pipe_size to 20 and follow instructions here for the
/proc files to touch:

https://github.com/pmacct/pmacct/blob/1.7.2/CONFIG-KEYS#L203-#L216

If it helps, good. If not: you should really look into one of the
frameworks i was pointing you to in my previous email. PF_RING, for
example, can do sampling and/or balancing. Sampling should not be done
inside pmacct because the dropped packets are between the kernel and the
application.

Paolo 

On Mon, Dec 17, 2018 at 02:52:48PM +0200, Edvinas K wrote:
> Seems there're lots of dropped packets:
> 
> prod [root@netvpn001prpjay pmacct-1.7.2]# pmacctd -i ens1f0.432 -f
> flowexport.cfg
> WARN: [flowexport.cfg:2] Invalid value. Ignored.
> INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> 1.7.2-git (20181018-00+c3)
> INFO ( default/core ):  '--enable-l2' '--enable-ipv6' '--enable-64bit'
> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> '--enable-st-bins'
> INFO ( default/core ): Reading configuration file
> '/opt/pmacct-1.7.2/flowexport.cfg'.
> INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally based
> on softflowd 0.9.7 software, Copyright 2002 Damien Miller 
> All rights reserved.
> INFO ( default_nfprobe/nfprobe ):   TCP timeout: 3600s
> INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> INFO ( default_nfprobe/nfprobe ):   UDP timeout: 300s
> INFO ( default_nfprobe/nfprobe ):  ICMP timeout: 300s
> INFO ( default_nfprobe/nfprobe ):   General timeout: 3600s
> INFO ( default_nfprobe/nfprobe ):  Maximum lifetime: 604800s
> INFO ( default_nfprobe/nfprobe ):   Expiry interval: 60s
> INFO ( default_nfprobe/nfprobe ): Exporting flows to
> [10.3.14.101]:rtcm-sc104
> INFO ( default/core ): [ens1f0.432,0] link type is: 1
> ^C^C^C^C^C^C^C^C
> 
> after 1 minute:
> 
> WARN ( default_nfprobe/nfprobe ): Shutting down on user request.
> INFO ( default/core ): OK, Exiting ...
> NOTICE ( default/core ): +++
> NOTICE ( default/core ): [ens1f0.432,0] received_packets=3441854
> *dropped_packets=2365166*
> 
> About 1GB of traffic is passing through the router where i'm capturing the
> packets. Isn't it too much traffic for nfrpobe to process ? CPUs seems not
> in 100% usage. We're using  Intel Xeon E5-2620 0 @ 2.00GHz
> 
> x
> 24.
> 
> prod [root@netvpn001prpjay ~]# ps -aux | grep pmacct
> root 41840 30.9  0.0  18964  7760 ?Rs   Dec14 1309:50 pmacctd:
> Core Process [default]
> root 41841 *68.4%*  0.0  22932  9756 ?RDec14 2898:29
> pmacctd: Netflow Probe Plugin [default_nfprobe]
> root 41869 32.5  0.0  19360  8128 ?Ss   Dec14 1378:29 pmacctd:
> Core Process [default]
> root 41870 *67.6%* 0.0  22928  9760 ?RDec14 2865:35
> pmacctd: Netflow Probe Plugin [default_nfprobe]
> 
> Before starting with your mentioned 'steroid' things, i would like to ask,
> is't really worth to go to that kernel "things", or start with techniques
> for example like sampling, or like Nikola recommended try to fidle with
> nfprobe_engine settings ?
> 
> Thanks
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Sun, Dec 16, 2018 at 6:25 PM Paolo Lucente  wrote:
> 
> >
> > Hi Edvinas,
> >
> > You may want to check whether libpcap is dropping packets on input to
> > pmacctd. You can achieve that sending a SIGUSR1 and checking the output
> > in the logfile/syslog/console. You will get something a-la:
> >
> > https://github.com/pmacct/pmacct/blob/master/docs/SIGNALS#L16-#L34
> >
> > Should amount of dropped packets be non-zero and visibly increasing then
> > you may want to put your libpcap on steroids:
> >
> > https://github.com/pmacct/pmacct/blob/master/FAQS#L71-#L101
> >
> > Should, instead, that not be the case, i am unsure and would need
> > further investigation. You could try to produce a controlled stream of
> > data and sniff nfprobe output. Or collect with a different software for
> > a quick counter-test (nfacctd itself or another of your choice).
> >
> > Paolo
> >
> > On Fri, Dec 14, 2018 at 03:02:35PM +0200, Edvinas K wrote:
> > > Thanks, i really appreciate your help.
> > >
> > > Everything seems working OK, on NFSEN (NFDUMP) graphs of flows statistics
> > > looks good, but the traffic rate Mb/s (45 Mb/s) is somehow 10x lower than
> > > really is. Maybe some tips to troubleshoot that ?
> > >
> > > [image: image.png]
> > >
> > > Is there any hidden things to check about ?
> > >
> > > My config:
> > >
> > > 1050  pmacctd -i ens1f0.432 -f flowexport.cfg
> > > 1051  pmacctd -i ens1f1.433 -f flowexport.cfg
> > >
> > > cat flowexport.cfg
> > >