Hi Edvinas,

I wanted to confirm that when you changed pmacctd_pipe_size to 2GB you
ALSO changed /proc/sys/net/core/[rw]mem_max to 2GB and ALSO restarted
pmacctd after having done so.

Wrt PF_RING: i can't voice since i don't use it myself. While i never
heard any horror story with it (thumbs up!), i think doing a proof of
concept first is always a good idea; this is also to answer your second
question: it will improve things for sure but how much you have to test.

Another thing you may do is also to increase buffering internal to
pmacct (it may help reduce CPU cycles by the core process and hence help
it process more data), i see that in your config you have NO buffering
enabled. For a quick test you could set:

plugin_pipe_size: 1000000
plugin_buffer_size: 10000

And depending if you see any benefits/improvement and if you have memory
you could ramp these values up. Or alternatively you could introduce
ZeroMQ. Again, this is internal queueuing (whereas in my previous email
i was tackling the queueing between kernel and pmacct):

https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L234-#L292

Paolo 

On Wed, Dec 19, 2018 at 06:40:14PM +0200, Edvinas K wrote:
> Hello,
> 
> How would you recommend to test PF_RING:
> 
> Some questions:
> 
> Is't safe to install it on production server ?
> Is't possible to hope, that this PF_RING will solve all the discards ?
> 
> Thanks
> 
> On Tue, Dec 18, 2018 at 5:59 PM Edvinas K <edvinas.em...@gmail.com> wrote:
> 
> > thanks,
> >
> > I tried to change the pipe size. As i noticed my OS (centos) default and
> > max size are the same:
> >
> > prod [root@netvpn001prpjay pmacct-1.7.2]# cat
> > /proc/sys/net/core/[rw]mem_default
> > 212992
> > 212992
> >
> > prod [root@netvpn001prpjay pmacct-1.7.2]# cat
> > /proc/sys/net/core/[rw]mem_max
> > 212992
> > 212992
> >
> > I tried to set the pmacctd_pipe_size: to 2000000000  and later to 212992.
> > Seems the drops is still occuring.
> > Tomorrow i will try to look at that PF_RING thing.
> >
> > Thanks
> >
> >
> >
> >
> >
> > On Tue, Dec 18, 2018 at 5:32 PM Paolo Lucente <pa...@pmacct.net> wrote:
> >
> >>
> >> Hi Edvinas,
> >>
> >> Easier thing first, i recommend to inject some test traffic and see that
> >> one how it looks like.
> >>
> >> The dropped packets highlight a buffering issue. You could take an
> >> intermediate step and see if enlarging buffers helps. Configure
> >> pmacctd_pipe_size to 2000000000 and follow instructions here for the
> >> /proc files to touch:
> >>
> >> https://github.com/pmacct/pmacct/blob/1.7.2/CONFIG-KEYS#L203-#L216
> >>
> >> If it helps, good. If not: you should really look into one of the
> >> frameworks i was pointing you to in my previous email. PF_RING, for
> >> example, can do sampling and/or balancing. Sampling should not be done
> >> inside pmacct because the dropped packets are between the kernel and the
> >> application.
> >>
> >> Paolo
> >>
> >> On Mon, Dec 17, 2018 at 02:52:48PM +0200, Edvinas K wrote:
> >> > Seems there're lots of dropped packets:
> >> >
> >> > prod [root@netvpn001prpjay pmacct-1.7.2]# pmacctd -i ens1f0.432 -f
> >> > flowexport.cfg
> >> > WARN: [flowexport.cfg:2] Invalid value. Ignored.
> >> > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> >> > 1.7.2-git (20181018-00+c3)
> >> > INFO ( default/core ):  '--enable-l2' '--enable-ipv6' '--enable-64bit'
> >> > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> >> > '--enable-st-bins'
> >> > INFO ( default/core ): Reading configuration file
> >> > '/opt/pmacct-1.7.2/flowexport.cfg'.
> >> > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally
> >> based
> >> > on softflowd 0.9.7 software, Copyright 2002 Damien Miller <
> >> d...@mindrot.org>
> >> > All rights reserved.
> >> > INFO ( default_nfprobe/nfprobe ):           TCP timeout: 3600s
> >> > INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> >> > INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> >> > INFO ( default_nfprobe/nfprobe ):           UDP timeout: 300s
> >> > INFO ( default_nfprobe/nfprobe ):          ICMP timeout: 300s
> >> > INFO ( default_nfprobe/nfprobe ):       General timeout: 3600s
> >> > INFO ( default_nfprobe/nfprobe ):      Maximum lifetime: 604800s
> >> > INFO ( default_nfprobe/nfprobe ):       Expiry interval: 60s
> >> > INFO ( default_nfprobe/nfprobe ): Exporting flows to
> >> > [10.3.14.101]:rtcm-sc104
> >> > INFO ( default/core ): [ens1f0.432,0] link type is: 1
> >> > ^C^C^C^C^C^C^C^C
> >> >
> >> > after 1 minute:
> >> >
> >> > WARN ( default_nfprobe/nfprobe ): Shutting down on user request.
> >> > INFO ( default/core ): OK, Exiting ...
> >> > NOTICE ( default/core ): +++
> >> > NOTICE ( default/core ): [ens1f0.432,0] received_packets=3441854
> >> > *dropped_packets=2365166*
> >> >
> >> > About 1GB of traffic is passing through the router where i'm capturing
> >> the
> >> > packets. Isn't it too much traffic for nfrpobe to process ? CPUs seems
> >> not
> >> > in 100% usage. We're using  Intel Xeon E5-2620 0 @ 2.00GHz
> >> > <
> >> http://netmon.adform.com/device/device=531/tab=health/metric=processor/processor_id=1466/
> >> >
> >> > x
> >> > 24.
> >> >
> >> > prod [root@netvpn001prpjay ~]# ps -aux | grep pmacct
> >> > root     41840 30.9  0.0  18964  7760 ?        Rs   Dec14 1309:50
> >> pmacctd:
> >> > Core Process [default]
> >> > root     41841 *68.4%*  0.0  22932  9756 ?        R    Dec14 2898:29
> >> > pmacctd: Netflow Probe Plugin [default_nfprobe]
> >> > root     41869 32.5  0.0  19360  8128 ?        Ss   Dec14 1378:29
> >> pmacctd:
> >> > Core Process [default]
> >> > root     41870 *67.6%* 0.0  22928  9760 ?        R    Dec14 2865:35
> >> > pmacctd: Netflow Probe Plugin [default_nfprobe]
> >> >
> >> > Before starting with your mentioned 'steroid' things, i would like to
> >> ask,
> >> > is't really worth to go to that kernel "things", or start with
> >> techniques
> >> > for example like sampling, or like Nikola recommended try to fidle with
> >> > nfprobe_engine settings ?
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Sun, Dec 16, 2018 at 6:25 PM Paolo Lucente <pa...@pmacct.net> wrote:
> >> >
> >> > >
> >> > > Hi Edvinas,
> >> > >
> >> > > You may want to check whether libpcap is dropping packets on input to
> >> > > pmacctd. You can achieve that sending a SIGUSR1 and checking the
> >> output
> >> > > in the logfile/syslog/console. You will get something a-la:
> >> > >
> >> > > https://github.com/pmacct/pmacct/blob/master/docs/SIGNALS#L16-#L34
> >> > >
> >> > > Should amount of dropped packets be non-zero and visibly increasing
> >> then
> >> > > you may want to put your libpcap on steroids:
> >> > >
> >> > > https://github.com/pmacct/pmacct/blob/master/FAQS#L71-#L101
> >> > >
> >> > > Should, instead, that not be the case, i am unsure and would need
> >> > > further investigation. You could try to produce a controlled stream of
> >> > > data and sniff nfprobe output. Or collect with a different software
> >> for
> >> > > a quick counter-test (nfacctd itself or another of your choice).
> >> > >
> >> > > Paolo
> >> > >
> >> > > On Fri, Dec 14, 2018 at 03:02:35PM +0200, Edvinas K wrote:
> >> > > > Thanks, i really appreciate your help.
> >> > > >
> >> > > > Everything seems working OK, on NFSEN (NFDUMP) graphs of flows
> >> statistics
> >> > > > looks good, but the traffic rate Mb/s (45 Mb/s) is somehow 10x
> >> lower than
> >> > > > really is. Maybe some tips to troubleshoot that ?
> >> > > >
> >> > > > [image: image.png]
> >> > > >
> >> > > > Is there any hidden things to check about ?
> >> > > >
> >> > > > My config:
> >> > > >
> >> > > > 1050  pmacctd -i ens1f0.432 -f flowexport.cfg
> >> > > > 1051  pmacctd -i ens1f1.433 -f flowexport.cfg
> >> > > >
> >> > > > cat flowexport.cfg
> >> > > >    !
> >> > > >    daemonize: true
> >> > > >    aggregate: src_host, dst_host, src_port, dst_port, proto, tos
> >> > > >    plugins: nfprobe
> >> > > >    nfprobe_receiver: 10.3.14.101:2101
> >> > > >    nfprobe_version: 9
> >> > > >    ! nfprobe_engine: 1:1
> >> > > >    ! nfprobe_timeouts: tcp=120:maxlife=3600
> >> > > >    !
> >> > > >    ! networks_file: /path/to/networks.lst
> >> > > >
> >> > > > On Thu, Dec 13, 2018 at 4:32 AM Paolo Lucente <pa...@pmacct.net>
> >> wrote:
> >> > > >
> >> > > > >
> >> > > > > Hi Nikola,
> >> > > > >
> >> > > > > I see, makes sense. Thanks very much for clarifying.
> >> > > > >
> >> > > > > Paolo
> >> > > > >
> >> > > > > On Wed, Dec 12, 2018 at 06:20:58PM -0800, Nikola Kolev wrote:
> >> > > > > > Hi Paollo,
> >> > > > > >
> >> > > > > > Sorry for being cryptic - what I meant was that I wasn't able to
> >> > > > > > launch pmacctd/uacctd in a way that it deals with dynamic
> >> interfaces
> >> > > as
> >> > > > > > ppp. Basically I failed to find any reference in the docs on
> >> how to
> >> > > make
> >> > > > > > it run in such a way, that it collects info from ppp* (a-la the
> >> ppp+
> >> > > > > > syntax of iptables), without launching a separate pmacctd
> >> instance
> >> > > for
> >> > > > > > each interface, hence the complicated setup with
> >> > > > > > iptables-nflog-uacctd-nfdump.
> >> > > > > >
> >> > > > > > On Thu, 13 Dec 2018 01:35:00 +0000
> >> > > > > > Paolo Lucente <pa...@pmacct.net> wrote:
> >> > > > > >
> >> > > > > > >
> >> > > > > > > Hi Nikola,
> >> > > > > > >
> >> > > > > > > Can you please elaborate a bit more? The cryptic part for me
> >> is "as
> >> > > > > > > nfacctd is not supporting wildcard addresses to be bound to".
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Paolo
> >> > > > > > >
> >> > > > > > > On Wed, Dec 12, 2018 at 04:50:33PM -0800, Nikola Kolev wrote:
> >> > > > > > > > Hey,
> >> > > > > > > >
> >> > > > > > > > If I may add to that:
> >> > > > > > > >
> >> > > > > > > > I'm doing something similar, but in a slightly different
> >> manner:
> >> > > > > > > >
> >> > > > > > > > as nfacctd is not supporting wildcard addresses to be bound
> >> to,
> >> > > I'm
> >> > > > > > > > using iptables' rules to export via nflog to uacctd, which
> >> then
> >> > > can
> >> > > > > > > > send to nfdump. Just food for thought...
> >> > > > > > > >
> >> > > > > > > > On 2018-12-12 14:58, Paolo Lucente wrote:
> >> > > > > > > > >Hi Edvinas,
> >> > > > > > > > >
> >> > > > > > > > >You are looking for the nfprobe plugin. You can follow the
> >> > > relevant
> >> > > > > > > > >section in the QUICKSTART to get going:
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > https://github.com/pmacct/pmacct/blob/1.7.2/QUICKSTART#L1167-#L1302
> >> > > > > > > > >
> >> > > > > > > > >Paolo
> >> > > > > > > > >
> >> > > > > > > > >On Wed, Dec 12, 2018 at 03:12:39PM +0200, Edvinas K wrote:
> >> > > > > > > > >>Hello,
> >> > > > > > > > >>
> >> > > > > > > > >>I managed to run basic pmacct to capture linux router
> >> (FRR)
> >> > > flows
> >> > > > > > > > >>from libcap:
> >> > > > > > > > >>"pmacctd -P print -O formatted -r 10 -i bond0.2170 -c
> >> > > > > > > > >>src_host,dst_host,src_port,dst_port,proto"
> >> > > > > > > > >>
> >> > > > > > > > >>now I need to push all the flows as a netflow format to
> >> the
> >> > > > > > > > >>netflow collector (nfdump). Could you give me some advice
> >> how
> >> > > to
> >> > > > > > > > >>configure that ?
> >> > > > > > > > >>Thank you
> >> > > > > > > > >
> >> > > > > > > > >>_______________________________________________
> >> > > > > > > > >>pmacct-discussion mailing list
> >> > > > > > > > >>http://www.pmacct.net/#mailinglists
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >_______________________________________________
> >> > > > > > > > >pmacct-discussion mailing list
> >> > > > > > > > >http://www.pmacct.net/#mailinglists
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Nikola
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Nikola
> >> > > > >
> >> > > > > _______________________________________________
> >> > > > > pmacct-discussion mailing list
> >> > > > > http://www.pmacct.net/#mailinglists
> >> > > > >
> >> > >
> >> > >
> >> > >
> >>
> >

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to