Re: [pmacct-discussion] nfacct accounts traffic twice

Fri, 08 Jul 2022 09:15:08 -0700 


Hi Michael,

Thanks for the feedback & great to see your setup is back on its feet.

Paolo


On 8/7/22 16:46, Muenz, Michael wrote:

Dear Paolo,

Forget everything I said, double fail!
I had wrong counts in Netflow cause OPNsense expects a LAN and a WAN interface to not double the values when natting. Then I set the correct values in OPNsense (LAN = WAN) cause it doesn't do NAT at all and added the IP below for testing. 
When adding this IP I broke the aggregate filter, stupid mistake!
I'll write a small article on medium.com about how to account traffic and share it here :) 

Thx,
Michael

Am 08.07.2022 um 16:32 schrieb Muenz, Michael:

Dear Paolo,

I'm very sorry but it seems my relayer was down for a couple of days.
I turned on debugging and restarted, seems aggregate_filter is not kicking in correct? 
(I removed all other networks from output)

root@flow01:/etc/pmacct# nfacctd -f /etc/pmacct/nfacctd.conf
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.6-git (RELEASE) INFO ( default/core ):  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-pgsql-includes=/usr/include/postgresql' '--enable-l2' '--enable-ipv6' '--enable-plabel' '--enable-mysql' '--enable-pgsql' '--enable-sqlite3' '--enable-rabbitmq' '--enable-zmq' '--enable-kafka' '--enable-geoipv2' '--enable-jansson' '--enable-64bit' '--enable-threads' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins' '--enable-nflog' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-fcommon' 'LDFLAGS=-Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/build/pmacct-UhNuxu/pmacct-1.7.6=. -fstack-protector-strong -Wformat -Werror=format-security' INFO ( default/core ): Reading configuration file '/etc/pmacct/nfacctd.conf'. INFO ( inbound/mysql ): plugin_pipe_size=4096000 bytes plugin_buffer_size=344 bytes INFO ( inbound/mysql ): ctrl channel: obtained=212992 bytes target=95248 bytes INFO ( outbound/mysql ): plugin_pipe_size=4096000 bytes plugin_buffer_size=344 bytes INFO ( outbound/mysql ): ctrl channel: obtained=212992 bytes target=95248 bytes DEBUG ( inbound/mysql ): [/etc/pmacct/networks.lst] v4 nh:  peer asn: 0 asn: 0 net: 46.16.78.247 mask: 32 DEBUG ( inbound/mysql ): [/etc/pmacct/networks.lst] IPv4 Networks Cache successfully created: 99991 entries. INFO ( inbound/mysql ): [/etc/pmacct/networks.lst] map successfully (re)loaded. DEBUG ( inbound/mysql ): [/etc/pmacct/networks.lst] v6 nh: peer_asn: 0 asn: 0 net: :: mask: 0 DEBUG ( inbound/mysql ): [/etc/pmacct/networks.lst] v6 contains a default route DEBUG ( inbound/mysql ): [/etc/pmacct/networks.lst] IPv6 Networks Cache successfully created: 32771 entries. INFO ( inbound/mysql ): cache entries=32771 base cache memory=13728736 bytes 
INFO ( default/core ): [/etc/pmacct/pre_tag.map] (re)loading map.
DEBUG ( outbound/mysql ): [/etc/pmacct/networks.lst] v4 nh:  peer asn: 0 asn: 0 net: 46.16.78.247 mask: 32 DEBUG ( outbound/mysql ): [/etc/pmacct/networks.lst] IPv4 Networks Cache successfully created: 99991 entries. INFO ( outbound/mysql ): [/etc/pmacct/networks.lst] map successfully (re)loaded. INFO ( default/core ): [/etc/pmacct/pre_tag.map] map successfully (re)loaded. 
INFO ( default/core ): [/etc/pmacct/pre_tag.map] (re)loading map.
DEBUG ( outbound/mysql ): [/etc/pmacct/networks.lst] v6 nh: peer_asn: 0 asn: 0 net: :: mask: 0 DEBUG ( outbound/mysql ): [/etc/pmacct/networks.lst] v6 contains a default route DEBUG ( outbound/mysql ): [/etc/pmacct/networks.lst] IPv6 Networks Cache successfully created: 32771 entries. INFO ( outbound/mysql ): cache entries=32771 base cache memory=13728736 bytes INFO ( default/core ): [/etc/pmacct/pre_tag.map] map successfully (re)loaded. 
INFO ( default/core ): [/etc/pmacct/pre_tag.map] (re)loading map.
INFO ( default/core ): [/etc/pmacct/pre_tag.map] map successfully (re)loaded. 
WARN: can't parse filter expression: syntax error
WARN ( inbound/mysql ): aggregation filter disabled.
WARN: can't parse filter expression: syntax error
WARN ( outbound/mysql ): aggregation filter disabled.
DEBUG ( default/core ): [/etc/pmacct/networks.lst] v4 nh:  peer asn: 0 asn: 0 net: 46.16.78.247 mask: 32 DEBUG ( default/core ): [/etc/pmacct/networks.lst] IPv4 Networks Cache successfully created: 99991 entries. INFO ( default/core ): [/etc/pmacct/networks.lst] map successfully (re)loaded. DEBUG ( default/core ): [/etc/pmacct/networks.lst] v6 nh: peer_asn: 0 asn: 0 net: :: mask: 0 DEBUG ( default/core ): [/etc/pmacct/networks.lst] v6 contains a default route DEBUG ( default/core ): [/etc/pmacct/networks.lst] IPv6 Networks Cache successfully created: 32771 entries. 
INFO ( default/core ): waiting for NetFlow/IPFIX data on :::5678

Best,
Michael

Am 04.07.2022 um 21:29 schrieb Paolo Lucente:


Hi Michael,
Welcome back! :-) What version of pmacct are you using? I see you daemonize but there is no logfile specified: did you check the log on startup to make sure that the filter in 'aggregate_filter' is being accepted and loaded?
Your understanding of how 'aggregate_filter' should work, ie. filter you out 1.2.3.4 if it's not specified among the networks listed in the filter, is right. 

Paolo


On 1/7/22 16:59, Muenz, Michael wrote:

Hi,
after over 15 years I'm back using pmacct for an open source accounting project. I'm using OPNsense to ingest Netflow v5 traffic into pmacct with MySQL backend. 
I'm intersted only in specific networks so I'm doing it like this:

daemonize: true
debug: false

nfacctd_port: 5678
nfacctd_time_new: true
plugins: mysql[inbound],mysql[outbound]

aggregate[inbound]: tag,dst_host
aggregate[outbound]: tag,src_host

aggregate_filter[inbound]: (dst net 46.16.78.247/32 ...)
aggregate_filter[outbound]: (src net 46.16.78.247/32 ...)

The different networks in in aggregate filter are differenct customers.
Now my idea was that I add a pretagging so when a packet comes with filter X it add tag Y: 

! 1101 = OPNREPO
id=1101 ip=81.33.44.75 filter='host 46.16.78.247'
Now every flow from 81.33.44.75 with traffic going from/to 46.16.78.247 gets tag 1101. 
After this I can select * from X where 1101 and sum up.
My problem is that aggregate_filter will also aggregate the source of the other side. Lets say I transfer a 1GB file from 1.2.3.4 to 46.16.78.247 I have 4 records: 

src 0.0.0.0, dst 46.16.78.247

src 0.0.0.0, dst 1.2.3.4

src 46.16.78.247, dst 0.0.0.0

src 1.2.3.4, dst 0.0.0.0
I thought that with aggregate_filter the lines with 1.2.3.4 wont get into the db but maybe I'm wrong? 

Any ideas?

Thanks!
Michael


_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists





_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to