Hi Steve, Canyou please post your integral config to try to reproduce the issue? It smells like something is wrong (bug).
Cheers, Paolo On Wed, Nov 09, 2016 at 10:38:02AM -0500, Stephen Clark wrote: > Hi Paolo, > > it seems that using the sql_preprocess: fsrc=200000 > causes the problem when start getting more than 200000 netflows per > minute. > > If we comment this line out we don't see any issue, minimal cpu usage only one > sql writer process. > > We are trying to limit the size of our database, that is why we used it. > > Any ideas why using this causes a problem? > > Thanks, > Steve > > On 11/09/2016 09:47 AM, Stephen Clark wrote: > >Oops - we just hit 10 writers. > > > >On 11/09/2016 08:54 AM, Stephen Clark wrote: > >>Hi Paolo, > >> > >>Thanks for the response. Do you see anything in our confguration > >>that I could adjust to mitigate the situation. > >> > >>We never reach 10 sql writers. > >> > >>Would increasing the any of these help? > >> sql_refresh_time: 60 > >>sql_optimize_clauses: true > >>!sql_history: 5m > >>sql_history: 1m > >>sql_history_roundoff: m > >> > >>Thanks, > >>Steve > >> > >>On 11/09/2016 07:58 AM, Paolo Lucente wrote: > >>>Hi Steve, > >>> > >>>You are experiencing a few connected problems, i guess. The root issue > >>>should be that the PostgreSQL database is not coping with the insert or > >>>update rate and/or with the size of the dataset. > >>> > >>>The list of plugins you see there are, in fact, all DB writers. They are > >>>queued up, waiting for the table they want to write to will unlock. What > >>>typically happens, wrt the seg fault, is that each of these processes > >>>does take some memory; you stack many of them, memory goes away; you try > >>>to stack more, they go seg fault due to lack of system resources. You > >>>can prevent that lowering the amoung of writers allowed to stack up, by > >>>default 10, via sql_max_writers. > >>> > >>>Finally, it is normal that the active DB writer can take 100% CPU in order > >>>to dump all its data to the backend. On shortage of system resources the > >>>situation can get unstabe and you may see multiple processes in such > >>>state as they start competing with kernel, ie. swap, etc. > >>> > >>>Cheers, > >>>Paolo > >>> > >>>On Tue, Nov 08, 2016 at 03:34:57PM -0500, Stephen Clark wrote: > >>>>Hi, > >>>> > >>>>I am having a problem with nfacctd getting way behind with ver 1.5.3 > >>>>Everything is ok until I add a server that is sending a lot of netflows > >>>>then things start bogging down. I see the nfacctd plugins > >>>>using 100% cpu using top. > >>>> > >>>>Then I start getting seg faults: > >>>>Nov 8 15:28:01 netflow2 kernel: nfacctd[14296]: segfault at 0 ip > >>>>000000000046d588 sp 00007fffe95e72c0 error 4 in > >>>>nfacctd[400000+14f000] > >>>> > >>>>11389 ? R 8:15 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>11993 ? R 7:11 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>12229 ? R 6:14 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>12372 ? R 5:15 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>12435 ? R 4:14 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>12678 ? R 3:17 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>13187 ? R 2:17 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>13499 ? R 1:13 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>13711 ? R 0:24 nfacctd: pgsql Plugin -- DB Writer [default] > >>>> > >>>> > >>>>daemonize: true > >>>>debug: false > >>>>pidfile: /var/run/nfacctd.pid > >>>>syslog: daemon > >>>> > >>>>pre_tag_map: /etc/pmacct/my.pretag.map > >>>> > >>>>nfacctd_disable_checks: true > >>>> > >>>>nfacctd_time_new: false > >>>> > >>>>aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos > >>>> > >>>> > >>>>!plugin_pipe_size: 8192000 > >>>>!plugin_buffer_size: 8192 > >>>>plugin_pipe_size: 4096000 > >>>>plugin_buffer_size: 4096 > >>>> > >>>>plugins: pgsql > >>>> > >>>>sql_table: netflow > >>>>sql_data: typed > >>>> > >>>>sql_dont_try_update: true > >>>>sql_use_copy: true > >>>>sql_db: pmacct > >>>>sql_host: 127.0.0.1 > >>>>sql_passwd: ****** > >>>>sql_user: pmacct > >>>> > >>>>sql_refresh_time: 60 > >>>>sql_optimize_clauses: true > >>>> > >>>>sql_history: 1m > >>>>sql_history_roundoff: m > >>>>sql_recovery_logfile: /var/lib/pmacct/recovery_log > >>>> > >>>>sql_preprocess: fsrc=200000 > >>>> > >>>>sql_locking_style: row > >>>>sql_cache_entries: 299999 > >>>> > >>>>imt_buckets: 65537 > >>>>imt_mem_pools_size: 1024000 > >>>> > >>>>nfacctd_port: 2055 > >>>> > >>>> > >>>>Suggestions? > >>>> > >>>>Thanks, > >>>>Steve > >>>> > >>>>_______________________________________________ > >>>>pmacct-discussion mailing list > >>>>http://www.pmacct.net/#mailinglists > >> > >> > > > > > > > -- > > "They that give up essential liberty to obtain temporary safety, > deserve neither liberty nor safety." (Ben Franklin) > > "The course of history shows that as a government grows, liberty > decreases." (Thomas Jefferson) > > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
