Hi Steve, fsrc seems to need a review. I advice you to move away from it for now. Can you move to any of the simpler thresholds part of the same framework, like minimum packets (minp) or bytes (minb) per dump?
Cheers, Paolo On Mon, Nov 14, 2016 at 06:44:16AM -0500, Stephen Clark wrote: > Hi Paolo, > > This is our config. > daemonize: true > debug: false > pidfile: /var/run/nfacctd.pid > syslog: daemon > pre_tag_map: /etc/pmacct/my.pretag.map > !nfacctd_disable_checks: false > nfacctd_disable_checks: true > > nfacctd_time_new: false > > aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos > > > !plugin_pipe_size: 8192000 > !plugin_buffer_size: 8192 > plugin_pipe_size: 4096000 > plugin_buffer_size: 4096 > > plugins: pgsql > > !sql_table: acct_uni_custom > sql_table: netflow > sql_data: typed > > !sql_multi_values: 512000 > sql_dont_try_update: true > sql_use_copy: true > sql_db: pmacct > sql_host: 127.0.0.1 > sql_passwd: arealsmartpwd > sql_user: pmacct > !sql_refresh_time: 300 > sql_refresh_time: 60 > sql_optimize_clauses: true > !sql_history: 5m > sql_history: 1m > sql_history_roundoff: m > sql_preprocess: fsrc=20000 > sql_locking_style: row > sql_cache_entries: 800011 > > imt_buckets: 65537 > imt_mem_pools_size: 1024000 > > nfacctd_port: 2055 > > Thanks for your support. > > Steve > > On 11/12/2016 09:31 AM, Paolo Lucente wrote: > >Hi Steve, > > > >Canyou please post your integral config to try to reproduce the issue? > >It smells like something is wrong (bug). > > > >Cheers, > >Paolo > > > >On Wed, Nov 09, 2016 at 10:38:02AM -0500, Stephen Clark wrote: > >>Hi Paolo, > >> > >>it seems that using the sql_preprocess: fsrc=200000 > >>causes the problem when start getting more than 200000 netflows per > >>minute. > >> > >>If we comment this line out we don't see any issue, minimal cpu usage only > >>one > >>sql writer process. > >> > >>We are trying to limit the size of our database, that is why we used it. > >> > >>Any ideas why using this causes a problem? > >> > >>Thanks, > >>Steve > >> > >>On 11/09/2016 09:47 AM, Stephen Clark wrote: > >>>Oops - we just hit 10 writers. > >>> > >>>On 11/09/2016 08:54 AM, Stephen Clark wrote: > >>>>Hi Paolo, > >>>> > >>>>Thanks for the response. Do you see anything in our confguration > >>>>that I could adjust to mitigate the situation. > >>>> > >>>>We never reach 10 sql writers. > >>>> > >>>>Would increasing the any of these help? > >>>>sql_refresh_time: 60 > >>>>sql_optimize_clauses: true > >>>>!sql_history: 5m > >>>>sql_history: 1m > >>>>sql_history_roundoff: m > >>>> > >>>>Thanks, > >>>>Steve > >>>> > >>>>On 11/09/2016 07:58 AM, Paolo Lucente wrote: > >>>>>Hi Steve, > >>>>> > >>>>>You are experiencing a few connected problems, i guess. The root issue > >>>>>should be that the PostgreSQL database is not coping with the insert or > >>>>>update rate and/or with the size of the dataset. > >>>>> > >>>>>The list of plugins you see there are, in fact, all DB writers. They are > >>>>>queued up, waiting for the table they want to write to will unlock. What > >>>>>typically happens, wrt the seg fault, is that each of these processes > >>>>>does take some memory; you stack many of them, memory goes away; you try > >>>>>to stack more, they go seg fault due to lack of system resources. You > >>>>>can prevent that lowering the amoung of writers allowed to stack up, by > >>>>>default 10, via sql_max_writers. > >>>>> > >>>>>Finally, it is normal that the active DB writer can take 100% CPU in > >>>>>order > >>>>>to dump all its data to the backend. On shortage of system resources the > >>>>>situation can get unstabe and you may see multiple processes in such > >>>>>state as they start competing with kernel, ie. swap, etc. > >>>>> > >>>>>Cheers, > >>>>>Paolo > >>>>> > >>>>>On Tue, Nov 08, 2016 at 03:34:57PM -0500, Stephen Clark wrote: > >>>>>>Hi, > >>>>>> > >>>>>>I am having a problem with nfacctd getting way behind with ver 1.5.3 > >>>>>>Everything is ok until I add a server that is sending a lot of netflows > >>>>>>then things start bogging down. I see the nfacctd plugins > >>>>>>using 100% cpu using top. > >>>>>> > >>>>>>Then I start getting seg faults: > >>>>>>Nov 8 15:28:01 netflow2 kernel: nfacctd[14296]: segfault at 0 ip > >>>>>>000000000046d588 sp 00007fffe95e72c0 error 4 in > >>>>>>nfacctd[400000+14f000] > >>>>>> > >>>>>>11389 ? R 8:15 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>>11993 ? R 7:11 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>>12229 ? R 6:14 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>>12372 ? R 5:15 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>>12435 ? R 4:14 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>>12678 ? R 3:17 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>>13187 ? R 2:17 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>>13499 ? R 1:13 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>>13711 ? R 0:24 nfacctd: pgsql Plugin -- DB Writer [default] > >>>>>> > >>>>>> > >>>>>>daemonize: true > >>>>>>debug: false > >>>>>>pidfile: /var/run/nfacctd.pid > >>>>>>syslog: daemon > >>>>>> > >>>>>>pre_tag_map: /etc/pmacct/my.pretag.map > >>>>>> > >>>>>>nfacctd_disable_checks: true > >>>>>> > >>>>>>nfacctd_time_new: false > >>>>>> > >>>>>>aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos > >>>>>> > >>>>>> > >>>>>>!plugin_pipe_size: 8192000 > >>>>>>!plugin_buffer_size: 8192 > >>>>>>plugin_pipe_size: 4096000 > >>>>>>plugin_buffer_size: 4096 > >>>>>> > >>>>>>plugins: pgsql > >>>>>> > >>>>>>sql_table: netflow > >>>>>>sql_data: typed > >>>>>> > >>>>>>sql_dont_try_update: true > >>>>>>sql_use_copy: true > >>>>>>sql_db: pmacct > >>>>>>sql_host: 127.0.0.1 > >>>>>>sql_passwd: ****** > >>>>>>sql_user: pmacct > >>>>>> > >>>>>>sql_refresh_time: 60 > >>>>>>sql_optimize_clauses: true > >>>>>> > >>>>>>sql_history: 1m > >>>>>>sql_history_roundoff: m > >>>>>>sql_recovery_logfile: /var/lib/pmacct/recovery_log > >>>>>> > >>>>>>sql_preprocess: fsrc=200000 > >>>>>> > >>>>>>sql_locking_style: row > >>>>>>sql_cache_entries: 299999 > >>>>>> > >>>>>>imt_buckets: 65537 > >>>>>>imt_mem_pools_size: 1024000 > >>>>>> > >>>>>>nfacctd_port: 2055 > >>>>>> > >>>>>> > >>>>>>Suggestions? > >>>>>> > >>>>>>Thanks, > >>>>>>Steve > >>>>>> > >>>>>>_______________________________________________ > >>>>>>pmacct-discussion mailing list > >>>>>>http://www.pmacct.net/#mailinglists > >>>> > >>> > >> > >>-- > >> > >>"They that give up essential liberty to obtain temporary safety, > >>deserve neither liberty nor safety." (Ben Franklin) > >> > >>"The course of history shows that as a government grows, liberty > >>decreases." (Thomas Jefferson) > >> > >> > > > -- > > "They that give up essential liberty to obtain temporary safety, > deserve neither liberty nor safety." (Ben Franklin) > > "The course of history shows that as a government grows, liberty > decreases." (Thomas Jefferson) > > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
