[pmacct-discussion] Kafka Plugin

[Catalin Petrescu]

Hi all,

Wondering if anyone has a simillar problem with kafka plugin. nfacctd
dosn't send to kafka , using :

conf
---
kafka_topic: pmacct1
kafka_topic[test]: pmacct1
kafka_refresh_time[test]: 300
kafka_history[test]: 5m
kafka_history_roundoff[test]: m
---
logs:
Jul 27 23:55:01 DEBUG ( test/kafka ): {"peer_as_src": 0, "bytes": 2584,
"tag": 0, "tos": 192, "as_dst": 0, "comms": "", "net_dst": "0.0.0.0",
"as_src": 0, "stamp_updated": "2016-07-27 23:55:01", "net_src": "0.0.0.0",
"as_path": "", "mask_src": 0, "ip_proto": "ospf", "mask_dst": 0,
"local_pref": 0, "peer_ip_dst": "", "stamp_inserted": "2016-07-27
23:50:00", "med": 0, "peer_as_dst": 0, "peer_ip_src": "10.255.255.1",
"src_as_path":Jul 27 23:55:01 DEBUG ( ukb/kafka ): Kafka message delivery
successful (446 bytes): {"peer_as_src": 0, "bytes": 2584, "tag": 0, "tos":
192, "as_dst": 0, "comms": "", "net_dst": "0.0.0.0", "as_src": 0,
"stamp_updated": "2016-07-27 23:55:01", "net_src": "0.0.0.0", "as_path":
"", "mask_src": 0, "ip_proto": "ospf", "mask_dst": 0, "local_pref": 0,
"peer_ip_dst": "", "stamp_inserted": "2016-07-27 23:50:00", "med": 0,
"peer_as_dst": 0Jul 27 23:55:02 INFO ( test/kafka ): *** Purging cache -
END (PID: 7008, QN: 1/1, ET: 1) ***


--- consumer

./kafka-console-consumer.sh --topic pmacct1 --zookeeper localhost
--from-beginning
^CConsumed 0 messages

using librdkafka_0.8

Any pointers are much appreciated .

Regards,
Catalin
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] sfacct feature suggestion - traffic in/out direction

[Jordan]

Hello,

I mean that when you enable sflow on an interface you cannot configure 
ingress/egress option.

It captures both directions while we need only data for ingress traffic.

There are two major problems with your solution. I think /direction /is 
not a valid sfacct key and we already use pretagging(both tag,tag2) for 
other purposes.


Regards,


On 07/27/2016 06:27 PM, Jentsch, Mario wrote:


Hi Jordan,

not sure what you mean with “equipment that cannot separate 
inbound/outbound traffic” but as long as you have /direction/ in your 
flow data you can add a pre-tag map like


/!/

/! tag=1  - inbound IPv4 traffic/

/! tag=2  - outbound IPv4 traffic/

/! tag=3  - inbound IPv6 traffic/

/! tag=4  - outbound IPv6 traffic/

/!/

/set_tag=1 ip=0.0.0.0/0 direction=0 filter='ip'/

/set_tag=2 ip=0.0.0.0/0 direction=1 filter='ip'/

/set_tag=3 ip=0.0.0.0/0 direction=0 filter='ip6'/

/set_tag=4 ip=0.0.0.0/0 direction=1 filter='ip6'/

/set_tag=0 ip=0.0.0.0/0/

/!/

and filter e.g. the ingress flows with

/!/

/pre_tag_filter[ingress]: 1,3/

/aggregate[ingress]: …/

/!/

Regards,

Mario

*From:*pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] 
*On Behalf Of *Jordan

*Sent:* Wednesday, July 27, 2016 5:06 PM
*To:* pmacct-discussion@pmacct.net
*Subject:* [pmacct-discussion] sfacct feature suggestion - traffic 
in/out direction


Hello,

We're having issues with equipment that cannot separate 
inbound/outbound traffic using sflow V5.


Looking at the sflow V5 protocol it's having the following fields. 
Usually they match the snmp interface indexes.

/source_id/
/interface input/
/interface output/


What I suggest as a new feature are the following cases:

*Match_all_traffic*(by default) - matches all packets (as it currently 
works)
*Match_input_only *- (if /source_id==interface input /permit, else 
drop the rest of the samples)
*Match_output_only* - (if/source_id==interface output/permit, 
else drop the rest of the samples)



Please let me know if such feature would be possible?
If there is any other already implemented solution I would be glad to 
know.


Thank you in advance.

Best Regards,


--
---


Jordan



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Aggregation suggestions

[Андрей Евтеев]

Hello,

we use nfacctd to account our customers inbound/outbound traffic with 
dst_net,dst_mask/src_net,src_mask aggregation, but if the networks_file 
list contains networks like:

10.0.0.0/26
10.0.0.0/24
10.0.0.0/22

and traffic goes to some IP belonging to 10.0.0.0/26, it only accounted 
for 10.0.0.0/26, is it possible to account traffic for all networks?



sorry for my bad english.


Best regards,

--
Andrey Evteev

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] pgsql insert only on version 1.5.3

[Stephen Clark]

Hi List,

Maybe someone can point out what I am doing wrong. I am trying to get nfacctd to 
only do inserts and not do updates

but my data looks like it is still doing updates, see row from pgsql below:
tag | ip_src  | ip_dst  | port_src | port_dst | ip_proto | tos | 
packets |   bytes   |   stamp_inserted| stamp_updated|   id| agent_id

-+-+-+--+--+--+-+-+---+-+-+-+--
  0 | 172.24.110.112  | 19x.xx.xxx.xx   |60391 | 443 |6 |   0 
|   8 |   328 | 2016-07-27 10:55:00 | 2016-07-27 11:10:01 | 1313720 
|  246


Notice stamp_inserted and stamp_updated - I would expect them to be the same if 
the pgsql plugin was only doing inserts.


Here is my config.

daemonize: true
debug: false
pidfile: /var/run/nfacctd.pid
syslog: daemon
!logfile: /home/arodriguez/pmacct/pmacct-1.5.3/logfile
pre_tag_map: ./my.pretag.map
nfacctd_disable_checks: false

nfacctd_time_new: false

aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos


plugin_pipe_size: 4096000
plugin_buffer_size: 4096

plugins: pgsql

sql_table: acct_uni_custom
sql_data: typed

!sql_multi_values: 512000
sql_dont_try_update: true
sql_use_copy: true
sql_db: pmacct
sql_host: 127.0.0.1
sql_passwd: arealsmartpwd
sql_user: pmacct
sql_refresh_time: 300
sql_optimize_clauses: true
sql_history: 5m
sql_history_roundoff: m
sql_recovery_logfile: /var/lib/pmacct/recovery_log
!sql_table_version: 9
sql_preprocess: qnum=1000, minp=5
sql_locking_style: row
sql_cache_entries: 19

imt_buckets: 65537
imt_mem_pools_size: 1024000

nfacctd_port: 2055
!nfacctd_ip: 127.0.0.1
!nfacctd_time_new: true
!nfacctd_allow_file: /etc/pmacct/allow

Any clarification would be appreciated.

Thanks,
Steve

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] sfacct feature suggestion - traffic in/out direction

[Jentsch, Mario]

Hi Jordan,

not sure what you mean with “equipment that cannot separate inbound/outbound 
traffic” but as long as you have direction in your flow data you can add a 
pre-tag map like

!
! tag=1  - inbound IPv4 traffic
! tag=2  - outbound IPv4 traffic
! tag=3  - inbound IPv6 traffic
! tag=4  - outbound IPv6 traffic
!
set_tag=1 ip=0.0.0.0/0 direction=0 filter='ip'
set_tag=2 ip=0.0.0.0/0 direction=1 filter='ip'
set_tag=3 ip=0.0.0.0/0 direction=0 filter='ip6'
set_tag=4 ip=0.0.0.0/0 direction=1 filter='ip6'
set_tag=0 ip=0.0.0.0/0
!

and filter e.g. the ingress flows with

!
pre_tag_filter[ingress]: 1,3
aggregate[ingress]: …
!

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jordan
Sent: Wednesday, July 27, 2016 5:06 PM
To: pmacct-discussion@pmacct.net
Subject: [pmacct-discussion] sfacct feature suggestion - traffic in/out 
direction

Hello,

We're having issues with equipment that cannot separate inbound/outbound 
traffic using sflow V5.

Looking at the sflow V5 protocol it's having the following fields. Usually they 
match the snmp interface indexes.
source_id
interface input
interface output


What I suggest as a new feature are the following cases:

Match_all_traffic(by default) - matches all packets (as it currently works)
Match_input_only - (ifsource_id==interface inputpermit, else drop the 
rest of the samples)
Match_output_only - (ifsource_id==interface outputpermit, else drop the 
rest of the samples)


Please let me know if such feature would be possible?
If there is any other already implemented solution I would be glad to know.

Thank you in advance.

Best Regards,


--
---
Jordan

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] sfacct feature suggestion - traffic in/out direction

[Jordan]

Hello,

We're having issues with equipment that cannot separate inbound/outbound 
traffic using sflow V5.


Looking at the sflow V5 protocol it's having the following fields. 
Usually they match the snmp interface indexes.

/source_id/
/interface input/
/interface output/


What I suggest as a new feature are the following cases:

*Match_all_traffic*(by default) - matches all packets (as it currently 
works)
*Match_input_onl**y *- (if /source_id==//interface input /permit, else 
drop the rest of the samples)
*Match_output_only* - (if///source_id==//interface//output/permit, 
else drop the rest of the samples)



Please let me know if such feature would be possible?
If there is any other already implemented solution I would be glad to know.

Thank you in advance.

Best Regards,



--
---


   Jordan


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Dynamic filtering of packets

[Mehul Prajapati]

Hi Mario,

I want to make configuration in PMacct for my requirement.
Let me reframe this question.

-I get triggering message in PMacct (e.g. from TCP/UDP port).
-I decode the message.
-I get an IP address and database logging on/off information.

If I get logging ON for an IP address then I want to make its entry in MySQL 
and start accounting.
If I get logging OFF for an IP address then I want to stop accounting for that 
IP.

I will ignore accounting for all other packets for which logging ON information 
is not received.

Is there any configuration to start/stop accounting at run time ?


From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jentsch, Mario
Sent: Tuesday, July 26, 2016 8:46 PM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

The mentioned user groups need to be build when the user authenticates and get 
the IP address assigned. This is most probably done in your Radius 
configuration and not in pmacct.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Tuesday, July 26, 2016 3:21 PM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi,

RADIUS packets are received on port 1812 & 1813.
I am decoding RADIUS packet in core process and then I get Accounting ON/OFF 
information in payload of packet.

I like your suggestion of user groups.

How can I configure user groups in PMacct ?
Is there any provision such that this Accounting ON/OFF requests configuration 
can be changes at run time ?




From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jentsch, Mario
Sent: Tuesday, July 26, 2016 6:18 PM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mehul.

the way you detect "Accounting ON/OFF" in pmacct affects possible 
recommendations. Same applies to other preconditions/requirements you may have.

If we don't need to care about them I would think you can filter out all IPs 
that you don't want accounting enabled for - this assumes there is no dynamic 
assignment from users to IP addresses.

With such a dynamic assignment a solution could be: assign the user groups 
("accounting on" vs "accounting off") different IP blocks and have pmacct 
collect the data only for the IP block for users with "accounting on".

In case your situation is more complicated and you can't take one of these ways 
it is helpful to know how you "receive some event i.e. Accounting ON in PMacct" 
and process it. Depending on if you're allowed to keep data about users with 
"accounting off" you also may solve it on MySQL side (presuming this is your 
data store). Receiving an "accounting on" you put the user/IP address for it in 
table1. Accounting data for this IP address is only stored in table2 if an 
appropriate entry in table1 is there. Receiving an "accounting off" you remove 
user/IP address for it in table1 and if required all appropriate entries from 
table2. This behavior should be possible with DB triggers / helper table.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Tuesday, July 26, 2016 11:48 AM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mario,

Remote Authentication Dial-In User Service (RADIUS) is a networking 
protocol that provides 
centralized Authentication, Authorization, and Accounting 
(AAA) management for users who 
connect and use a network service.

You can ignore the RADIUS decoding part.

Requirement:
I want to do accounting for only selective users/IP addresses.

Let say, I receive some event i.e. Accounting ON in PMacct. Now, processing 
this event, I want to start accounting for only this IP address/user.
After some time, I receive Accounting OFF event in PMacct. Now, processing this 
event, I want to stop accounting for only this IP address/user.

Is there any mechanism to achieve it ?



From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jentsch, Mario
Sent: Tuesday, July 26, 2016 2:55 PM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mehul,

> I have explored about "refresh_maps" config key.
> If I use it, then I need to make changes in map file at run time.

that is working fine in one of our setups. We detect changes in the network, 
re-create the map file and have the daemon reload it without restart.

> But, I want to make filtering such that changes reside in memory only.

That sounds like you need to patch pmacct what IMHO is the least best solution.

>