Re: [pmacct-discussion] performance issue

2016-11-21 Thread Paolo Lucente
Hi Steve, fsrc seems to need a review. I advice you to move away from it for now. Can you move to any of the simpler thresholds part of the same framework, like minimum packets (minp) or bytes (minb) per dump? Cheers, Paolo On Mon, Nov 14, 2016 at 06:44:16AM -0500, Stephen Clark wrote: > Hi

Re: [pmacct-discussion] performance issue

2016-11-14 Thread Stephen Clark
Hi Paolo, This is our config. daemonize: true debug: false pidfile: /var/run/nfacctd.pid syslog: daemon pre_tag_map: /etc/pmacct/my.pretag.map !nfacctd_disable_checks: false nfacctd_disable_checks: true nfacctd_time_new: false aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos

Re: [pmacct-discussion] performance issue

2016-11-12 Thread Paolo Lucente
Hi Steve, Canyou please post your integral config to try to reproduce the issue? It smells like something is wrong (bug). Cheers, Paolo On Wed, Nov 09, 2016 at 10:38:02AM -0500, Stephen Clark wrote: > Hi Paolo, > > it seems that using the sql_preprocess: fsrc=20 > causes the problem when

Re: [pmacct-discussion] performance issue

2016-11-09 Thread Stephen Clark
Oops - we just hit 10 writers. On 11/09/2016 08:54 AM, Stephen Clark wrote: Hi Paolo, Thanks for the response. Do you see anything in our confguration that I could adjust to mitigate the situation. We never reach 10 sql writers. Would increasing the any of these help? sql_refresh_time: 60

Re: [pmacct-discussion] performance issue

2016-11-09 Thread Stephen Clark
Hi Paolo, Thanks for the response. Do you see anything in our confguration that I could adjust to mitigate the situation. We never reach 10 sql writers. Would increasing the any of these help? sql_refresh_time: 60 sql_optimize_clauses: true !sql_history: 5m sql_history: 1m

Re: [pmacct-discussion] performance issue

2016-11-09 Thread Paolo Lucente
Hi Steve, You are experiencing a few connected problems, i guess. The root issue should be that the PostgreSQL database is not coping with the insert or update rate and/or with the size of the dataset. The list of plugins you see there are, in fact, all DB writers. They are queued up, waiting

[pmacct-discussion] performance issue

2016-11-08 Thread Stephen Clark
Hi, I am having a problem with nfacctd getting way behind with ver 1.5.3 Everything is ok until I add a server that is sending a lot of netflows then things start bogging down. I see the nfacctd plugins using 100% cpu using top. Then I start getting seg faults: Nov 8 15:28:01 netflow2 kernel: