Re: [pmacct-discussion] performance issue

[Stephen Clark]

Hi Paolo,

This is our config.
daemonize: true
debug: false
pidfile: /var/run/nfacctd.pid
syslog: daemon
pre_tag_map: /etc/pmacct/my.pretag.map
!nfacctd_disable_checks: false
nfacctd_disable_checks: true

nfacctd_time_new: false

aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos


!plugin_pipe_size: 8192000
!plugin_buffer_size: 8192
plugin_pipe_size: 4096000
plugin_buffer_size: 4096

plugins: pgsql

!sql_table: acct_uni_custom
sql_table: netflow
sql_data: typed

!sql_multi_values: 512000
sql_dont_try_update: true
sql_use_copy: true
sql_db: pmacct
sql_host: 127.0.0.1
sql_passwd: arealsmartpwd
sql_user: pmacct
!sql_refresh_time: 300
sql_refresh_time: 60
sql_optimize_clauses: true
!sql_history: 5m
sql_history: 1m
sql_history_roundoff: m
sql_preprocess:  fsrc=2
sql_locking_style: row
sql_cache_entries: 800011

imt_buckets: 65537
imt_mem_pools_size: 1024000

nfacctd_port: 2055

Thanks for your support.

Steve

On 11/12/2016 09:31 AM, Paolo Lucente wrote:

Hi Steve,

Canyou please post your integral config to try to reproduce the issue?
It smells like something is wrong (bug).

Cheers,
Paolo

On Wed, Nov 09, 2016 at 10:38:02AM -0500, Stephen Clark wrote:

Hi Paolo,

it seems that using the sql_preprocess:  fsrc=20
causes the problem when start getting more than 20 netflows per
minute.

If we comment this line out we don't see any issue, minimal cpu usage only one
sql writer process.

We are trying to limit the size of our database, that is why we used it.

Any ideas why using this causes a problem?

Thanks,
Steve

On 11/09/2016 09:47 AM, Stephen Clark wrote:

Oops - we just hit 10 writers.

On 11/09/2016 08:54 AM, Stephen Clark wrote:

Hi Paolo,

Thanks for the response. Do you see anything in our confguration
that I could adjust to mitigate the situation.

We never reach 10 sql writers.

Would increasing the any of these help?
sql_refresh_time: 60
sql_optimize_clauses: true
!sql_history: 5m
sql_history: 1m
sql_history_roundoff: m

Thanks,
Steve

On 11/09/2016 07:58 AM, Paolo Lucente wrote:

Hi Steve,

You are experiencing a few connected problems, i guess. The root issue
should be that the PostgreSQL database is not coping with the insert or
update rate and/or with the size of the dataset.

The list of plugins you see there are, in fact, all DB writers. They are
queued up, waiting for the table they want to write to will unlock. What
typically happens, wrt the seg fault, is that each of these processes
does take some memory; you stack many of them, memory goes away; you try
to stack more, they go seg fault due to lack of system resources. You
can prevent that lowering the amoung of writers allowed to stack up, by
default 10, via sql_max_writers.

Finally, it is normal that the active DB writer can take 100% CPU in order
to dump all its data to the backend. On shortage of system resources the
situation can get unstabe and you may see multiple processes in such
state as they start competing with kernel, ie. swap, etc.

Cheers,
Paolo

On Tue, Nov 08, 2016 at 03:34:57PM -0500, Stephen Clark wrote:

Hi,

I am having a problem with nfacctd getting way behind with ver 1.5.3
Everything is ok until I add a server that is sending a lot of netflows
then things start bogging down. I see the nfacctd plugins
using 100% cpu using top.

Then I start getting seg faults:
Nov  8 15:28:01 netflow2 kernel: nfacctd[14296]: segfault at 0 ip
0046d588 sp 7fffe95e72c0 error 4 in
nfacctd[40+14f000]

11389 ?R  8:15 nfacctd: pgsql Plugin -- DB Writer [default]
11993 ?R  7:11 nfacctd: pgsql Plugin -- DB Writer [default]
12229 ?R  6:14 nfacctd: pgsql Plugin -- DB Writer [default]
12372 ?R  5:15 nfacctd: pgsql Plugin -- DB Writer [default]
12435 ?R  4:14 nfacctd: pgsql Plugin -- DB Writer [default]
12678 ?R  3:17 nfacctd: pgsql Plugin -- DB Writer [default]
13187 ?R  2:17 nfacctd: pgsql Plugin -- DB Writer [default]
13499 ?R  1:13 nfacctd: pgsql Plugin -- DB Writer [default]
13711 ?R  0:24 nfacctd: pgsql Plugin -- DB Writer [default]


daemonize: true
debug: false
pidfile: /var/run/nfacctd.pid
syslog: daemon

pre_tag_map: /etc/pmacct/my.pretag.map

nfacctd_disable_checks: true

nfacctd_time_new: false

aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos


!plugin_pipe_size: 8192000
!plugin_buffer_size: 8192
plugin_pipe_size: 4096000
plugin_buffer_size: 4096

plugins: pgsql

sql_table: netflow
sql_data: typed

sql_dont_try_update: true
sql_use_copy: true
sql_db: pmacct
sql_host: 127.0.0.1
sql_passwd: **
sql_user: pmacct

sql_refresh_time: 60
sql_optimize_clauses: true

sql_history: 1m
sql_history_roundoff: m
sql_recovery_logfile: /var/lib/pmacct/recovery_log

sql_preprocess:  fsrc=20

sql_locking_style: row
sql_cache_entries: 29

imt_buckets: 65537
imt_mem_pools_size: 1024000

nfacctd_port: 2055


Suggestions?

Thanks,
Steve

[pmacct-discussion] Redis support

[Lorenzo Mainardi]

Hi Paolo,
I'm writing to asking if you are thinking about supporting Redis as plugin

The advantage to use Redis would be that both Logstash both Telegraf support it 
in a native way, then it would be easy to store pmacct data in native way in 
Elasticsearch or InfluxDB.
Regards


digitel

Via della Fortezza 6 - 50129 Firenze
www.digitelitalia.com - 800 901 669

Ing. Lorenzo Mainardi

Tel +39 055 4624933
Fax +39 055 4624 947
l...@digitelitalia.com


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Redis support

[Raphael Mazelier]

On 14/11/2016 19:42, Rasto Rickardt wrote:

Hello Paolo,

+1 for me, main reasons are the usual Redis ones:



Pmacct have already a lot of backend plugins, but if another is needed I 
vote for redis. Redis is blazing fast and can act both as a message 
queue and a an ephemeral storage. I think I can use it for live 
data/trends, and passing the history to influx ?


With sql/amqp/kafka + redis pmacct will cover all the current 
technologies. (perhaps the mongodb plugins was not necessary ;)


--
Raphael Mazelier

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Redis support

[Rasto Rickardt]

Hello Paolo,

+1 for me, main reasons are the usual Redis ones:

Speed - 100k GET by_id per second
Data persistence across restarts
Multi node deployment is not a MUST like other NoSQL document stores
Lightweight

My use-case is storing time-based events from flows in InfluxDB (bps,
pps, flows per host/subnet) and details (tcp flags, packet sizes,
as_path etc.) in redis. Storing timestamp, tag and value is where Influx
is good, storing documentation is where redis shines.

Storing directly to redis can be usefull, because most of data amount
per flow goes there for me.

The same will apply for InfluxDB but i am interfacing it over http, so
there is bunch of overhead already, i do not think there will be major
performance improvement writing directly, and tag:value pairs are really
lightweight.

r.


On 11/14/2016 06:02 PM, Paolo Lucente wrote:
> 
> Hi Lorenzo,
> 
> You are the second person asking for this - i'd be curious to see on
> the list if there would be anybody else interested in this.
> 
> I have a question, maybe basic: would not Kafka or RabbitMQ be
> integrated the same way with Logstash and/or Telegraf?
> 
> Cheers, Paolo
> 
> On Mon, Nov 14, 2016 at 09:38:57AM +, Lorenzo Mainardi wrote:
>> Hi Paolo, I'm writing to asking if you are thinking about
>> supporting Redis as plugin
>> 
>> The advantage to use Redis would be that both Logstash both
>> Telegraf support it in a native way, then it would be easy to store
>> pmacct data in native way in Elasticsearch or InfluxDB. Regards
>> 
>> 
>> digitel
>> 
>> Via della Fortezza 6 - 50129 Firenze 
>> www.digitelitalia.com - 800 901 669
>> 
>> Ing. Lorenzo Mainardi
>> 
>> Tel +39 055 4624933 Fax +39 055 4624 947 
>> l...@digitelitalia.com
>> 
>> 
> 
>> ___ pmacct-discussion
>> mailing list http://www.pmacct.net/#mailinglists
> 
> 
> ___ pmacct-discussion
> mailing list http://www.pmacct.net/#mailinglists
> 

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Redis support

[Lorenzo Mainardi]

Logstash supports both Kafka both Rabbitmq as input 
(https://www.elastic.co/guide/en/logstash/current/input-plugins.html)


Telegraf does not (https://docs.influxdata.com/telegraf/v1.1/inputs/)


Redis is fast (it's written in C), easy to setup and deploy and very easy to 
use as message queue.


Input plugins | Logstash Reference [5.0] | 
Elastic
www.elastic.co
An input plugin enables a specific source of events to be read by Logstash. The 
following input plugins are available:



Da: pmacct-discussion  per conto di Paolo 
Lucente 
Inviato: lunedì 14 novembre 2016 18.02.19
A: pmacct-discussion@pmacct.net
Oggetto: Re: [pmacct-discussion] Redis support


Hi Lorenzo,

You are the second person asking for this - i'd be curious to see on the
list if there would be anybody else interested in this.

I have a question, maybe basic: would not Kafka or RabbitMQ be integrated
the same way with Logstash and/or Telegraf?

Cheers,
Paolo

On Mon, Nov 14, 2016 at 09:38:57AM +, Lorenzo Mainardi wrote:
> Hi Paolo,
> I'm writing to asking if you are thinking about supporting Redis as plugin
>
> The advantage to use Redis would be that both Logstash both Telegraf support 
> it in a native way, then it would be easy to store pmacct data in native way 
> in Elasticsearch or InfluxDB.
> Regards
>
>
> digitel
>
> Via della Fortezza 6 - 50129 Firenze
> www.digitelitalia.com>
>  - 800 901 669
>
> Ing. Lorenzo Mainardi
>
> Tel +39 055 4624933
> Fax +39 055 4624 947
> l...@digitelitalia.com
>
>

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists