Re: [pmacct-discussion] pgsql insert only on version 1.5.3

[Stephen Clark]

Thanks Paolo,

I'll give that a try.

On 07/28/2016 02:39 PM, Paolo Lucente wrote:

Hi Steve,

Try setting 'nfacctd_time_new: true' which would take as reference
time of arrival of the flow to the collector; you should get your
desired behaviour. Another solution is to keep nfacctd_time_new to
false and decrease to the minimum the active timeout on your NetFlow
exporter (what is happening now is that some long-lived flows is
being trapped at the exporter long time before being exported to
the collector).

Cheers,
Paolo

On Wed, Jul 27, 2016 at 11:30:47AM -0400, Stephen Clark wrote:

Hi List,

Maybe someone can point out what I am doing wrong. I am trying to
get nfacctd to only do inserts and not do updates
but my data looks like it is still doing updates, see row from pgsql below:
tag | ip_src  | ip_dst  | port_src | port_dst |
ip_proto | tos | packets |   bytes   |   stamp_inserted|
stamp_updated|   id| agent_id
-+-+-+--+--+--+-+-+---+-+-+-+--
   0 | 172.24.110.112  | 19x.xx.xxx.xx   |60391 | 443 |6
|   0 |   8 |   328 | 2016-07-27 10:55:00 | 2016-07-27
11:10:01 | 1313720 |  246

Notice stamp_inserted and stamp_updated - I would expect them to be
the same if the pgsql plugin was only doing inserts.

Here is my config.

daemonize: true
debug: false
pidfile: /var/run/nfacctd.pid
syslog: daemon
!logfile: /home/arodriguez/pmacct/pmacct-1.5.3/logfile
pre_tag_map: ./my.pretag.map
nfacctd_disable_checks: false

nfacctd_time_new: false

aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos


plugin_pipe_size: 4096000
plugin_buffer_size: 4096

plugins: pgsql

sql_table: acct_uni_custom
sql_data: typed

!sql_multi_values: 512000
sql_dont_try_update: true
sql_use_copy: true
sql_db: pmacct
sql_host: 127.0.0.1
sql_passwd: arealsmartpwd
sql_user: pmacct
sql_refresh_time: 300
sql_optimize_clauses: true
sql_history: 5m
sql_history_roundoff: m
sql_recovery_logfile: /var/lib/pmacct/recovery_log
!sql_table_version: 9
sql_preprocess: qnum=1000, minp=5
sql_locking_style: row
sql_cache_entries: 19

imt_buckets: 65537
imt_mem_pools_size: 1024000

nfacctd_port: 2055
!nfacctd_ip: 127.0.0.1
!nfacctd_time_new: true
!nfacctd_allow_file: /etc/pmacct/allow

Any clarification would be appreciated.

Thanks,
Steve

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists



--

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Kafka Plugin

[Catalin Petrescu]

Hi Paolo,

was using 1.6.0 but using master , same issues.

for me the steps to reproduce are start nfacctd with the kafka plugin ,
don't create any topic in kafka . This triggers nfacctd to create "acct"
topic and the following error in log:

 Connection failed to Kafka: p_kafka_check_outq_len()
Jul 28 20:05:02 ERROR ( test/kafka ): Connection failed to Kafka:
p_kafka_close()

Firts interval there is no data in kafka , assume it's because of above.
Second interval (5m in this case) i have data.

Stop nfacctd , create pmacct1 topic using ( ./kafka-topics.sh --create
--topic pmacct1 --replication-factor 1 --partitions 1 --zookeeper localhost
). Start nfacctd , data is updated to acct rather than pmacct1.

hope this helps. let me know if you need more info.

Regards,
Catalin

On Thu, Jul 28, 2016 at 7:11 PM, Paolo Lucente  wrote:

>
> Hi Catalin,
>
> What version this is? I've tried to reproduce with code in master
> on GitHub and all appears to work fine and i see data pusehd into
> the expected topic 'pmacct1'. If you are not using code in master,
> can you please give it a try as well?
>
> Cheers,
> Paolo
>
> On Thu, Jul 28, 2016 at 09:11:05AM +0100, Catalin Petrescu wrote:
> > Not sure if it's a know limitation .
> >
> > looks like the pointer is acct rather than the one defined in the config.
> >
> > ./kafka-simple-consumer-shell.sh  --topic acct --broker-list
> localhost:9092
> > --partition 0
> >
> > regards.
> > Catalin
>
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Looking for a fresh pmacct UI

[raf]

Le 28/07/2016 à 08:50, Linas Lesauskas a écrit :

Hi,

just my 2 cents.

I found very useful InfluxDB (https://influxdata.com/) for time-series
data storage. It is extremely fast and lean on storage questions, uses
kind of 20 bytes per record.
Combining with Grafana (http://grafana.org/) you can build classy and
stylish user interface.

Currently we are playing with pmacct->bunch of scripts->influx on the
lab and waiting for a more free time to push this to production.

best regards,



I've used both approach :

pmacct > amqp > scripts > influxdb > grafana to display some nice graphs.

and custom home made gui (in php :/ ) to display specific informations 
(top 10 interface / asn / as path / talkers - in/out).


All this material is available on https://github.com/ut0mt8 .

I think there are trivial to adapt on your needs.

--
Raphael Mazelier


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] pgsql insert only on version 1.5.3

[Paolo Lucente]

Hi Steve,

Try setting 'nfacctd_time_new: true' which would take as reference
time of arrival of the flow to the collector; you should get your
desired behaviour. Another solution is to keep nfacctd_time_new to
false and decrease to the minimum the active timeout on your NetFlow
exporter (what is happening now is that some long-lived flows is
being trapped at the exporter long time before being exported to
the collector). 

Cheers,
Paolo

On Wed, Jul 27, 2016 at 11:30:47AM -0400, Stephen Clark wrote:
> Hi List,
> 
> Maybe someone can point out what I am doing wrong. I am trying to
> get nfacctd to only do inserts and not do updates
> but my data looks like it is still doing updates, see row from pgsql below:
> tag | ip_src  | ip_dst  | port_src | port_dst |
> ip_proto | tos | packets |   bytes   |   stamp_inserted|
> stamp_updated|   id| agent_id
> -+-+-+--+--+--+-+-+---+-+-+-+--
>   0 | 172.24.110.112  | 19x.xx.xxx.xx   |60391 | 443 |6
> |   0 |   8 |   328 | 2016-07-27 10:55:00 | 2016-07-27
> 11:10:01 | 1313720 |  246
> 
> Notice stamp_inserted and stamp_updated - I would expect them to be
> the same if the pgsql plugin was only doing inserts.
> 
> Here is my config.
> 
> daemonize: true
> debug: false
> pidfile: /var/run/nfacctd.pid
> syslog: daemon
> !logfile: /home/arodriguez/pmacct/pmacct-1.5.3/logfile
> pre_tag_map: ./my.pretag.map
> nfacctd_disable_checks: false
> 
> nfacctd_time_new: false
> 
> aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos
> 
> 
> plugin_pipe_size: 4096000
> plugin_buffer_size: 4096
> 
> plugins: pgsql
> 
> sql_table: acct_uni_custom
> sql_data: typed
> 
> !sql_multi_values: 512000
> sql_dont_try_update: true
> sql_use_copy: true
> sql_db: pmacct
> sql_host: 127.0.0.1
> sql_passwd: arealsmartpwd
> sql_user: pmacct
> sql_refresh_time: 300
> sql_optimize_clauses: true
> sql_history: 5m
> sql_history_roundoff: m
> sql_recovery_logfile: /var/lib/pmacct/recovery_log
> !sql_table_version: 9
> sql_preprocess: qnum=1000, minp=5
> sql_locking_style: row
> sql_cache_entries: 19
> 
> imt_buckets: 65537
> imt_mem_pools_size: 1024000
> 
> nfacctd_port: 2055
> !nfacctd_ip: 127.0.0.1
> !nfacctd_time_new: true
> !nfacctd_allow_file: /etc/pmacct/allow
> 
> Any clarification would be appreciated.
> 
> Thanks,
> Steve
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Aggregation suggestions

[Paolo Lucente]

Hi Andrey,

Unfortunately this is not possible. You have a networks_file_no_lpm
switch, which does not really apply to your case since you have all
three networks in a networks_file, but in the end you can account
traffic only to one net - then you would have to summarize yourself
as part of the post-processing.

What alternatively you may do is to tag or label flows so to add
qualities to it via a pre_tag_map. Imagine the /26 is the customer
itself (and that is what you account on), then /24 is the city and
/22 is the region; then you can have a label like: "FROM_REGION_X,
FROM_CITY_Y,TO_REGION_Z,TO_CITY_W". Don't know how feasible it is
and/or if it solves you anything since then you would have to
extract information anyway from the 'label' field.

Cheers,
Paolo

On Wed, Jul 27, 2016 at 06:57:51PM +0300, Андрей Евтеев wrote:
> Hello,
> 
> we use nfacctd to account our customers inbound/outbound traffic
> with dst_net,dst_mask/src_net,src_mask aggregation, but if the
> networks_file list contains networks like:
> 10.0.0.0/26
> 10.0.0.0/24
> 10.0.0.0/22
> 
> and traffic goes to some IP belonging to 10.0.0.0/26, it only
> accounted for 10.0.0.0/26, is it possible to account traffic for all
> networks?
> 
> 
> sorry for my bad english.
> 
> 
> Best regards,
> 
> --
> Andrey Evteev
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Kafka Plugin

[Paolo Lucente]

Hi Catalin,

What version this is? I've tried to reproduce with code in master
on GitHub and all appears to work fine and i see data pusehd into
the expected topic 'pmacct1'. If you are not using code in master,
can you please give it a try as well?

Cheers,
Paolo

On Thu, Jul 28, 2016 at 09:11:05AM +0100, Catalin Petrescu wrote:
> Not sure if it's a know limitation .
> 
> looks like the pointer is acct rather than the one defined in the config.
> 
> ./kafka-simple-consumer-shell.sh  --topic acct --broker-list localhost:9092
> --partition 0
> 
> regards.
> Catalin

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Kafka Plugin

[Catalin Petrescu]

Not sure if it's a know limitation .

looks like the pointer is acct rather than the one defined in the config.

./kafka-simple-consumer-shell.sh  --topic acct --broker-list localhost:9092
--partition 0

regards.
Catalin
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Dynamic filtering of packets

[Jentsch, Mario]

Hi Mehul

> Is there any configuration to start/stop accounting at run time ?

In Pmacct only by using maps + filtering and recreating + reloading the maps 
for changes at runtime. For all I know - there is no API provided by Pmacct for 
that and there is no way implemented to trigger actions based on decoded 
packets.

Regards,
Mario


From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Wednesday, July 27, 2016 8:30 AM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mario,

I want to make configuration in PMacct for my requirement.
Let me reframe this question.

-I get triggering message in PMacct (e.g. from TCP/UDP port).
-I decode the message.
-I get an IP address and database logging on/off information.

If I get logging ON for an IP address then I want to make its entry in MySQL 
and start accounting.
If I get logging OFF for an IP address then I want to stop accounting for that 
IP.

I will ignore accounting for all other packets for which logging ON information 
is not received.

Is there any configuration to start/stop accounting at run time ?


Mehul Prajapati
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists