Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-27 Thread Alex K
Thank you Paolo,

I see I can use aggregation filters also. So I guess will find a way to
implement what is needed without having a convoluted configuration file.

cheers,
Alex

On Thu, Feb 27, 2020 at 12:24 PM Paolo Lucente  wrote:

>
> Hi Alex,
>
> Ack. The other way you could "filter" out is with a networks_file: in
> there you specify the network(s) you are interested in following the
> example here:
>
> https://github.com/pmacct/pmacct/blob/master/examples/networks.lst.example
>
> In the simplest case, you just want to list networks of interest one per
> line. Then in the config you want to set 'networks_file_filter: true' as
> well. This is kind-of filtering: networks / IPs not of interest will be
> just zeroed out and rolled up as a 0.0.0.0 src_host / dst_host.
>

> Paolo
>
> On Wed, Feb 26, 2020 at 11:32:31AM +0200, Alex K wrote:
> > Hi Paolo,
> >
> > On Tue, Feb 25, 2020 at 6:41 PM Paolo Lucente  wrote:
> >
> > >
> > > Hi Alex,
> > >
> > > Thanks for your feedback. I see you did run "tcpdump -n -vv -i nflog:1"
> > > which is equivalent to run uacctd without any filters; as you may know,
> > > you can append a BPF-style filter to the tcpdump command-line,
> precisely
> > > as you express it in pre_tag_map. Can you give that a try and see if
> you
> > > get any luck?
> > >
> > Bad luck... I get:
> > tcpdump -nvv -i  nflog:1 src net 192.168.28.0/24
> > tcpdump: NFLOG link-layer type filtering not implemented
> > It seems that filtering at nflog interface is not supported.
> > Running tcpdump -nvv -i eth0 src net 192.168.28.0/24 does capture
> traffic
> > normally.
> > Is there any other way I could apply some filtering with uacctd? I need
> to
> > use uacctd since I get all the pre-nat, post-nat details of the flows, so
> > as to account traffic at the WAN interfaces with the real source details.
> >
> >
> > > My expextation is: if something does not work with pre_tag_map, it
> > > should also not work with tcpdump; if you work out a filter to work
> > > against tcpdump, that should work in pre_tag_map as well. Any
> disconnect
> > > among the two may bring the scent of a bug.
> > >
> > > Paolo
> > >
> > > On Tue, Feb 25, 2020 at 11:20:21AM +0200, Alex K wrote:
> > > > Here is the output when running in debug mode:
> > > >
> > > > INFO ( default/core ): Linux NetFilter NFLOG Accounting Daemon,
> uacctd
> > > > (20200222-01)
> > > > INFO ( default/core ):  '--prefix=/usr' '--enable-mysql'
> '--enable-nflog'
> > > > '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins'
> > > > '--enable-bmp-bins' '--enable-st-bins'
> > > > INFO ( default/core ): Reading configuration file
> > > > '/root/pmacct/uacctd2.conf'.
> > > > INFO ( print_wan0_in/print ): plugin_pipe_size=4096000 bytes
> > > > plugin_buffer_size=280 bytes
> > > > INFO ( print_wan0_in/print ): ctrl channel: obtained=212992 bytes
> > > > target=117024 bytes
> > > > INFO ( print_wan0_out/print ): plugin_pipe_size=4096000 bytes
> > > > plugin_buffer_size=280 bytes
> > > > INFO ( print_wan0_out/print ): ctrl channel: obtained=212992 bytes
> > > > target=117024 bytes
> > > > INFO ( print_wan0_in/print ): cache entries=16411 base cache
> > > > memory=54878384 bytes
> > > > INFO ( default/core ): [pretag2.map] (re)loading map.
> > > > INFO ( print_wan0_out/print ): cache entries=16411 base cache
> > > > memory=54878384 bytes
> > > > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > > > INFO ( default/core ): [pretag2.map] (re)loading map.
> > > > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > > > INFO ( default/core ): [pretag2.map] (re)loading map.
> > > > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > > > INFO ( default/core ): Successfully connected Netlink NFLOG socket
> > > >
> > > > It doesn't seem to have any issues loading the maps, though it is not
> > > > collecting anything. When capturing with tcpdump I see packets going
> > > > through:
> > > >
> > > > tcpdump -n -vv -i nflog:1
> > > > 09:16:05.831131 IP (tos 0x0, ttl 64, id 36511, offset 0, flags [DF],
> > > proto
> > > > ICMP (1), length 84)
> > > > 192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 1,
> length
> > > 64
> > > > 09:16:05.831362 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none],
> proto
> > > > ICMP (1), length 84)
> > > > 8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 1,
> length 64
> > > > 09:16:05.831392 IP (tos 0x0, ttl 64, id 36682, offset 0, flags [DF],
> > > proto
> > > > ICMP (1), length 84)
> > > > 192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 2,
> length
> > > 64
> > > > 09:16:06.855200 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none],
> proto
> > > > ICMP (1), length 84)
> > > > 8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 2,
> length 64
> > > >
> > > > The pmacct  version I am running is latest master.
> > > > Thank you for your assistance.
> > > >
> > > > Alex
> > > >
> > > >
> > > > On Mon, Feb 24, 2020 at 6:20 PM Alex K 
> wrote:
> > 

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-27 Thread Paolo Lucente


Hi Alex,

Ack. The other way you could "filter" out is with a networks_file: in
there you specify the network(s) you are interested in following the
example here:

https://github.com/pmacct/pmacct/blob/master/examples/networks.lst.example

In the simplest case, you just want to list networks of interest one per
line. Then in the config you want to set 'networks_file_filter: true' as
well. This is kind-of filtering: networks / IPs not of interest will be
just zeroed out and rolled up as a 0.0.0.0 src_host / dst_host.

Paolo

On Wed, Feb 26, 2020 at 11:32:31AM +0200, Alex K wrote:
> Hi Paolo,
> 
> On Tue, Feb 25, 2020 at 6:41 PM Paolo Lucente  wrote:
> 
> >
> > Hi Alex,
> >
> > Thanks for your feedback. I see you did run "tcpdump -n -vv -i nflog:1"
> > which is equivalent to run uacctd without any filters; as you may know,
> > you can append a BPF-style filter to the tcpdump command-line, precisely
> > as you express it in pre_tag_map. Can you give that a try and see if you
> > get any luck?
> >
> Bad luck... I get:
> tcpdump -nvv -i  nflog:1 src net 192.168.28.0/24
> tcpdump: NFLOG link-layer type filtering not implemented
> It seems that filtering at nflog interface is not supported.
> Running tcpdump -nvv -i eth0 src net 192.168.28.0/24 does capture traffic
> normally.
> Is there any other way I could apply some filtering with uacctd? I need to
> use uacctd since I get all the pre-nat, post-nat details of the flows, so
> as to account traffic at the WAN interfaces with the real source details.
> 
> 
> > My expextation is: if something does not work with pre_tag_map, it
> > should also not work with tcpdump; if you work out a filter to work
> > against tcpdump, that should work in pre_tag_map as well. Any disconnect
> > among the two may bring the scent of a bug.
> >
> > Paolo
> >
> > On Tue, Feb 25, 2020 at 11:20:21AM +0200, Alex K wrote:
> > > Here is the output when running in debug mode:
> > >
> > > INFO ( default/core ): Linux NetFilter NFLOG Accounting Daemon, uacctd
> > > (20200222-01)
> > > INFO ( default/core ):  '--prefix=/usr' '--enable-mysql' '--enable-nflog'
> > > '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins'
> > > '--enable-bmp-bins' '--enable-st-bins'
> > > INFO ( default/core ): Reading configuration file
> > > '/root/pmacct/uacctd2.conf'.
> > > INFO ( print_wan0_in/print ): plugin_pipe_size=4096000 bytes
> > > plugin_buffer_size=280 bytes
> > > INFO ( print_wan0_in/print ): ctrl channel: obtained=212992 bytes
> > > target=117024 bytes
> > > INFO ( print_wan0_out/print ): plugin_pipe_size=4096000 bytes
> > > plugin_buffer_size=280 bytes
> > > INFO ( print_wan0_out/print ): ctrl channel: obtained=212992 bytes
> > > target=117024 bytes
> > > INFO ( print_wan0_in/print ): cache entries=16411 base cache
> > > memory=54878384 bytes
> > > INFO ( default/core ): [pretag2.map] (re)loading map.
> > > INFO ( print_wan0_out/print ): cache entries=16411 base cache
> > > memory=54878384 bytes
> > > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > > INFO ( default/core ): [pretag2.map] (re)loading map.
> > > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > > INFO ( default/core ): [pretag2.map] (re)loading map.
> > > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > > INFO ( default/core ): Successfully connected Netlink NFLOG socket
> > >
> > > It doesn't seem to have any issues loading the maps, though it is not
> > > collecting anything. When capturing with tcpdump I see packets going
> > > through:
> > >
> > > tcpdump -n -vv -i nflog:1
> > > 09:16:05.831131 IP (tos 0x0, ttl 64, id 36511, offset 0, flags [DF],
> > proto
> > > ICMP (1), length 84)
> > > 192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 1, length
> > 64
> > > 09:16:05.831362 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none], proto
> > > ICMP (1), length 84)
> > > 8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 1, length 64
> > > 09:16:05.831392 IP (tos 0x0, ttl 64, id 36682, offset 0, flags [DF],
> > proto
> > > ICMP (1), length 84)
> > > 192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 2, length
> > 64
> > > 09:16:06.855200 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none], proto
> > > ICMP (1), length 84)
> > > 8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 2, length 64
> > >
> > > The pmacct  version I am running is latest master.
> > > Thank you for your assistance.
> > >
> > > Alex
> > >
> > >
> > > On Mon, Feb 24, 2020 at 6:20 PM Alex K  wrote:
> > >
> > > > Hi Paolo,
> > > >
> > > > On Sat, Feb 22, 2020 at 4:18 PM Paolo Lucente 
> > wrote:
> > > >
> > > >>
> > > >> Hi Alex,
> > > >>
> > > >> Is it possible with the new setup - the one where pre_tag_map does not
> > > >> match anything - the traffic is VLAN-tagged (or MPLS-labelled)? If so,
> > > >> you should adjust filters accordingly and add 'vlan and', ie. "vlan
> > and
> > > >> src net 192.168.28.0/24 or vlan and src net 192.168.100.0/24".
> > > >>
> 

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-26 Thread Alex K
Hi Paolo,

On Tue, Feb 25, 2020 at 6:41 PM Paolo Lucente  wrote:

>
> Hi Alex,
>
> Thanks for your feedback. I see you did run "tcpdump -n -vv -i nflog:1"
> which is equivalent to run uacctd without any filters; as you may know,
> you can append a BPF-style filter to the tcpdump command-line, precisely
> as you express it in pre_tag_map. Can you give that a try and see if you
> get any luck?
>
Bad luck... I get:
tcpdump -nvv -i  nflog:1 src net 192.168.28.0/24
tcpdump: NFLOG link-layer type filtering not implemented
It seems that filtering at nflog interface is not supported.
Running tcpdump -nvv -i eth0 src net 192.168.28.0/24 does capture traffic
normally.
Is there any other way I could apply some filtering with uacctd? I need to
use uacctd since I get all the pre-nat, post-nat details of the flows, so
as to account traffic at the WAN interfaces with the real source details.


> My expextation is: if something does not work with pre_tag_map, it
> should also not work with tcpdump; if you work out a filter to work
> against tcpdump, that should work in pre_tag_map as well. Any disconnect
> among the two may bring the scent of a bug.
>
> Paolo
>
> On Tue, Feb 25, 2020 at 11:20:21AM +0200, Alex K wrote:
> > Here is the output when running in debug mode:
> >
> > INFO ( default/core ): Linux NetFilter NFLOG Accounting Daemon, uacctd
> > (20200222-01)
> > INFO ( default/core ):  '--prefix=/usr' '--enable-mysql' '--enable-nflog'
> > '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins'
> > '--enable-bmp-bins' '--enable-st-bins'
> > INFO ( default/core ): Reading configuration file
> > '/root/pmacct/uacctd2.conf'.
> > INFO ( print_wan0_in/print ): plugin_pipe_size=4096000 bytes
> > plugin_buffer_size=280 bytes
> > INFO ( print_wan0_in/print ): ctrl channel: obtained=212992 bytes
> > target=117024 bytes
> > INFO ( print_wan0_out/print ): plugin_pipe_size=4096000 bytes
> > plugin_buffer_size=280 bytes
> > INFO ( print_wan0_out/print ): ctrl channel: obtained=212992 bytes
> > target=117024 bytes
> > INFO ( print_wan0_in/print ): cache entries=16411 base cache
> > memory=54878384 bytes
> > INFO ( default/core ): [pretag2.map] (re)loading map.
> > INFO ( print_wan0_out/print ): cache entries=16411 base cache
> > memory=54878384 bytes
> > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > INFO ( default/core ): [pretag2.map] (re)loading map.
> > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > INFO ( default/core ): [pretag2.map] (re)loading map.
> > INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> > INFO ( default/core ): Successfully connected Netlink NFLOG socket
> >
> > It doesn't seem to have any issues loading the maps, though it is not
> > collecting anything. When capturing with tcpdump I see packets going
> > through:
> >
> > tcpdump -n -vv -i nflog:1
> > 09:16:05.831131 IP (tos 0x0, ttl 64, id 36511, offset 0, flags [DF],
> proto
> > ICMP (1), length 84)
> > 192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 1, length
> 64
> > 09:16:05.831362 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none], proto
> > ICMP (1), length 84)
> > 8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 1, length 64
> > 09:16:05.831392 IP (tos 0x0, ttl 64, id 36682, offset 0, flags [DF],
> proto
> > ICMP (1), length 84)
> > 192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 2, length
> 64
> > 09:16:06.855200 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none], proto
> > ICMP (1), length 84)
> > 8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 2, length 64
> >
> > The pmacct  version I am running is latest master.
> > Thank you for your assistance.
> >
> > Alex
> >
> >
> > On Mon, Feb 24, 2020 at 6:20 PM Alex K  wrote:
> >
> > > Hi Paolo,
> > >
> > > On Sat, Feb 22, 2020 at 4:18 PM Paolo Lucente 
> wrote:
> > >
> > >>
> > >> Hi Alex,
> > >>
> > >> Is it possible with the new setup - the one where pre_tag_map does not
> > >> match anything - the traffic is VLAN-tagged (or MPLS-labelled)? If so,
> > >> you should adjust filters accordingly and add 'vlan and', ie. "vlan
> and
> > >> src net 192.168.28.0/24 or vlan and src net 192.168.100.0/24".
> > >>
> > > The traffic is not VLAN or MPLS. It is simple one. I confirm I can
> collect
> > > traffic when removing the pretag directives. Also when stopping
> uacctd, I
> > > can capture traffic at nflog:1 interface.
> > > I simplified the configuration as below:
> > >
> > > !
> > > daemonize: true
> > > promisc:   false
> > > uacctd_group: 1
> > > !
> > > pre_tag_map: pretag2.map
> > > pre_tag_filter[print_wan0_in]: 1
> > > pre_tag_filter[print_wan0_out]: 2
> > > !
> > > !-
> > > plugins: print[print_wan0_in], print[print_wan0_out]
> > > print_refresh_time: 10
> > > print_history: 15m
> > > print_output_file_append: true
> > > !
> > > print_output[print_wan0_in]: csv
> > > print_output[print_wan0_out]: csv
> > > print_output_file[print_wan0_in]: 

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-25 Thread Paolo Lucente


Hi Alex,

Thanks for your feedback. I see you did run "tcpdump -n -vv -i nflog:1"
which is equivalent to run uacctd without any filters; as you may know,
you can append a BPF-style filter to the tcpdump command-line, precisely
as you express it in pre_tag_map. Can you give that a try and see if you
get any luck?

My expextation is: if something does not work with pre_tag_map, it
should also not work with tcpdump; if you work out a filter to work
against tcpdump, that should work in pre_tag_map as well. Any disconnect
among the two may bring the scent of a bug.

Paolo
 
On Tue, Feb 25, 2020 at 11:20:21AM +0200, Alex K wrote:
> Here is the output when running in debug mode:
> 
> INFO ( default/core ): Linux NetFilter NFLOG Accounting Daemon, uacctd
> (20200222-01)
> INFO ( default/core ):  '--prefix=/usr' '--enable-mysql' '--enable-nflog'
> '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins'
> '--enable-bmp-bins' '--enable-st-bins'
> INFO ( default/core ): Reading configuration file
> '/root/pmacct/uacctd2.conf'.
> INFO ( print_wan0_in/print ): plugin_pipe_size=4096000 bytes
> plugin_buffer_size=280 bytes
> INFO ( print_wan0_in/print ): ctrl channel: obtained=212992 bytes
> target=117024 bytes
> INFO ( print_wan0_out/print ): plugin_pipe_size=4096000 bytes
> plugin_buffer_size=280 bytes
> INFO ( print_wan0_out/print ): ctrl channel: obtained=212992 bytes
> target=117024 bytes
> INFO ( print_wan0_in/print ): cache entries=16411 base cache
> memory=54878384 bytes
> INFO ( default/core ): [pretag2.map] (re)loading map.
> INFO ( print_wan0_out/print ): cache entries=16411 base cache
> memory=54878384 bytes
> INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> INFO ( default/core ): [pretag2.map] (re)loading map.
> INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> INFO ( default/core ): [pretag2.map] (re)loading map.
> INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
> INFO ( default/core ): Successfully connected Netlink NFLOG socket
> 
> It doesn't seem to have any issues loading the maps, though it is not
> collecting anything. When capturing with tcpdump I see packets going
> through:
> 
> tcpdump -n -vv -i nflog:1
> 09:16:05.831131 IP (tos 0x0, ttl 64, id 36511, offset 0, flags [DF], proto
> ICMP (1), length 84)
> 192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 1, length 64
> 09:16:05.831362 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none], proto
> ICMP (1), length 84)
> 8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 1, length 64
> 09:16:05.831392 IP (tos 0x0, ttl 64, id 36682, offset 0, flags [DF], proto
> ICMP (1), length 84)
> 192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 2, length 64
> 09:16:06.855200 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none], proto
> ICMP (1), length 84)
> 8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 2, length 64
> 
> The pmacct  version I am running is latest master.
> Thank you for your assistance.
> 
> Alex
> 
> 
> On Mon, Feb 24, 2020 at 6:20 PM Alex K  wrote:
> 
> > Hi Paolo,
> >
> > On Sat, Feb 22, 2020 at 4:18 PM Paolo Lucente  wrote:
> >
> >>
> >> Hi Alex,
> >>
> >> Is it possible with the new setup - the one where pre_tag_map does not
> >> match anything - the traffic is VLAN-tagged (or MPLS-labelled)? If so,
> >> you should adjust filters accordingly and add 'vlan and', ie. "vlan and
> >> src net 192.168.28.0/24 or vlan and src net 192.168.100.0/24".
> >>
> > The traffic is not VLAN or MPLS. It is simple one. I confirm I can collect
> > traffic when removing the pretag directives. Also when stopping uacctd, I
> > can capture traffic at nflog:1 interface.
> > I simplified the configuration as below:
> >
> > !
> > daemonize: true
> > promisc:   false
> > uacctd_group: 1
> > !
> > pre_tag_map: pretag2.map
> > pre_tag_filter[print_wan0_in]: 1
> > pre_tag_filter[print_wan0_out]: 2
> > !
> > !-
> > plugins: print[print_wan0_in], print[print_wan0_out]
> > print_refresh_time: 10
> > print_history: 15m
> > print_output_file_append: true
> > !
> > print_output[print_wan0_in]: csv
> > print_output[print_wan0_out]: csv
> > print_output_file[print_wan0_in]: traffic-wan0-in.csv
> > print_output_file[print_wan0_out]: traffic-wan0-out.csv
> > !
> > aggregate[print_wan0_in]: tag, src_host, dst_host, src_port, dst_port,
> > proto
> > aggregate[print_wan0_out]: tag, src_host, dst_host, src_port, dst_port,
> > proto
> > !
> >
> > with pretag2.map
> > set_tag=1 filter='src net 192.168.28.0/24'
> > set_tag=2 filter='dst net 192.168.28.0/24'
> >
> > As soon as I enable the pretag directives as below, I do not see any
> > traffic being collected from uacctd at NFLOG goup 1
> >
> > pre_tag_map: pretag2.map
> > pre_tag_filter[print_wan0_in]: 1
> > pre_tag_filter[print_wan0_out]: 2
> >
> > I am running pmacct 1.7.4.
> >
> >
> >> Paolo
> >>
> >> On Fri, Feb 21, 2020 at 01:04:25PM +0200, Alex K wrote:
> >> > Working further on this, 

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-25 Thread Alex K
Here is the output when running in debug mode:

INFO ( default/core ): Linux NetFilter NFLOG Accounting Daemon, uacctd
(20200222-01)
INFO ( default/core ):  '--prefix=/usr' '--enable-mysql' '--enable-nflog'
'--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins'
'--enable-bmp-bins' '--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/root/pmacct/uacctd2.conf'.
INFO ( print_wan0_in/print ): plugin_pipe_size=4096000 bytes
plugin_buffer_size=280 bytes
INFO ( print_wan0_in/print ): ctrl channel: obtained=212992 bytes
target=117024 bytes
INFO ( print_wan0_out/print ): plugin_pipe_size=4096000 bytes
plugin_buffer_size=280 bytes
INFO ( print_wan0_out/print ): ctrl channel: obtained=212992 bytes
target=117024 bytes
INFO ( print_wan0_in/print ): cache entries=16411 base cache
memory=54878384 bytes
INFO ( default/core ): [pretag2.map] (re)loading map.
INFO ( print_wan0_out/print ): cache entries=16411 base cache
memory=54878384 bytes
INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
INFO ( default/core ): [pretag2.map] (re)loading map.
INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
INFO ( default/core ): [pretag2.map] (re)loading map.
INFO ( default/core ): [pretag2.map] map successfully (re)loaded.
INFO ( default/core ): Successfully connected Netlink NFLOG socket

It doesn't seem to have any issues loading the maps, though it is not
collecting anything. When capturing with tcpdump I see packets going
through:

tcpdump -n -vv -i nflog:1
09:16:05.831131 IP (tos 0x0, ttl 64, id 36511, offset 0, flags [DF], proto
ICMP (1), length 84)
192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 1, length 64
09:16:05.831362 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none], proto
ICMP (1), length 84)
8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 1, length 64
09:16:05.831392 IP (tos 0x0, ttl 64, id 36682, offset 0, flags [DF], proto
ICMP (1), length 84)
192.168.28.11 > 8.8.8.8: ICMP echo request, id 17353, seq 2, length 64
09:16:06.855200 IP (tos 0x0, ttl 49, id 0, offset 0, flags [none], proto
ICMP (1), length 84)
8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 2, length 64

The pmacct  version I am running is latest master.
Thank you for your assistance.

Alex


On Mon, Feb 24, 2020 at 6:20 PM Alex K  wrote:

> Hi Paolo,
>
> On Sat, Feb 22, 2020 at 4:18 PM Paolo Lucente  wrote:
>
>>
>> Hi Alex,
>>
>> Is it possible with the new setup - the one where pre_tag_map does not
>> match anything - the traffic is VLAN-tagged (or MPLS-labelled)? If so,
>> you should adjust filters accordingly and add 'vlan and', ie. "vlan and
>> src net 192.168.28.0/24 or vlan and src net 192.168.100.0/24".
>>
> The traffic is not VLAN or MPLS. It is simple one. I confirm I can collect
> traffic when removing the pretag directives. Also when stopping uacctd, I
> can capture traffic at nflog:1 interface.
> I simplified the configuration as below:
>
> !
> daemonize: true
> promisc:   false
> uacctd_group: 1
> !
> pre_tag_map: pretag2.map
> pre_tag_filter[print_wan0_in]: 1
> pre_tag_filter[print_wan0_out]: 2
> !
> !-
> plugins: print[print_wan0_in], print[print_wan0_out]
> print_refresh_time: 10
> print_history: 15m
> print_output_file_append: true
> !
> print_output[print_wan0_in]: csv
> print_output[print_wan0_out]: csv
> print_output_file[print_wan0_in]: traffic-wan0-in.csv
> print_output_file[print_wan0_out]: traffic-wan0-out.csv
> !
> aggregate[print_wan0_in]: tag, src_host, dst_host, src_port, dst_port,
> proto
> aggregate[print_wan0_out]: tag, src_host, dst_host, src_port, dst_port,
> proto
> !
>
> with pretag2.map
> set_tag=1 filter='src net 192.168.28.0/24'
> set_tag=2 filter='dst net 192.168.28.0/24'
>
> As soon as I enable the pretag directives as below, I do not see any
> traffic being collected from uacctd at NFLOG goup 1
>
> pre_tag_map: pretag2.map
> pre_tag_filter[print_wan0_in]: 1
> pre_tag_filter[print_wan0_out]: 2
>
> I am running pmacct 1.7.4.
>
>
>> Paolo
>>
>> On Fri, Feb 21, 2020 at 01:04:25PM +0200, Alex K wrote:
>> > Working further on this, it seems that for pmacct is sufficient to
>> filter
>> > traffic using only the pre_tag_filter, thus no need for the aggregation
>> > filters.
>> > The issue with this setup though is that I loose the information of the
>> > pre_nat source IP address when monitoring at the WAN interfaces. Due to
>> > this I am switching to uacctd as following:
>> >
>> > !
>> > daemonize: true
>> > promisc:   false
>> > uacctd_group: 1
>> > !networks_file: networks.lst
>> > !ports_file: ports.lst
>> > !
>> > pre_tag_map: pretag2.map
>> > pre_tag_filter[print_wan0_in]: 1
>> > pre_tag_filter[print_wan0_out]: 2
>> > pre_tag_filter[wan0_in]: 1
>> > pre_tag_filter[wan0_out]: 2
>> > !
>> > plugins: print[print_wan0_in], print[print_wan0_out], mysql[wan0_in],
>> > mysql[wan0_out]
>> > plugin_pipe_size[wan0_in]: 1024000
>> > plugin_pipe_size[wan0_out]: 1024000
>> > 

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-24 Thread Alex K
Hi Paolo,

On Sat, Feb 22, 2020 at 4:18 PM Paolo Lucente  wrote:

>
> Hi Alex,
>
> Is it possible with the new setup - the one where pre_tag_map does not
> match anything - the traffic is VLAN-tagged (or MPLS-labelled)? If so,
> you should adjust filters accordingly and add 'vlan and', ie. "vlan and
> src net 192.168.28.0/24 or vlan and src net 192.168.100.0/24".
>
The traffic is not VLAN or MPLS. It is simple one. I confirm I can collect
traffic when removing the pretag directives. Also when stopping uacctd, I
can capture traffic at nflog:1 interface.
I simplified the configuration as below:

!
daemonize: true
promisc:   false
uacctd_group: 1
!
pre_tag_map: pretag2.map
pre_tag_filter[print_wan0_in]: 1
pre_tag_filter[print_wan0_out]: 2
!
!-
plugins: print[print_wan0_in], print[print_wan0_out]
print_refresh_time: 10
print_history: 15m
print_output_file_append: true
!
print_output[print_wan0_in]: csv
print_output[print_wan0_out]: csv
print_output_file[print_wan0_in]: traffic-wan0-in.csv
print_output_file[print_wan0_out]: traffic-wan0-out.csv
!
aggregate[print_wan0_in]: tag, src_host, dst_host, src_port, dst_port, proto
aggregate[print_wan0_out]: tag, src_host, dst_host, src_port, dst_port,
proto
!

with pretag2.map
set_tag=1 filter='src net 192.168.28.0/24'
set_tag=2 filter='dst net 192.168.28.0/24'

As soon as I enable the pretag directives as below, I do not see any
traffic being collected from uacctd at NFLOG goup 1

pre_tag_map: pretag2.map
pre_tag_filter[print_wan0_in]: 1
pre_tag_filter[print_wan0_out]: 2

I am running pmacct 1.7.4.


> Paolo
>
> On Fri, Feb 21, 2020 at 01:04:25PM +0200, Alex K wrote:
> > Working further on this, it seems that for pmacct is sufficient to filter
> > traffic using only the pre_tag_filter, thus no need for the aggregation
> > filters.
> > The issue with this setup though is that I loose the information of the
> > pre_nat source IP address when monitoring at the WAN interfaces. Due to
> > this I am switching to uacctd as following:
> >
> > !
> > daemonize: true
> > promisc:   false
> > uacctd_group: 1
> > !networks_file: networks.lst
> > !ports_file: ports.lst
> > !
> > pre_tag_map: pretag2.map
> > pre_tag_filter[print_wan0_in]: 1
> > pre_tag_filter[print_wan0_out]: 2
> > pre_tag_filter[wan0_in]: 1
> > pre_tag_filter[wan0_out]: 2
> > !
> > plugins: print[print_wan0_in], print[print_wan0_out], mysql[wan0_in],
> > mysql[wan0_out]
> > plugin_pipe_size[wan0_in]: 1024000
> > plugin_pipe_size[wan0_out]: 1024000
> > print_refresh_time: 10
> > print_history: 15m
> > print_output_file_append: true
> > !
> > print_output[print_wan0_in]: csv
> > print_output_file[print_wan0_in]: in_traffic.csv
> > print_output[print_wan0_out]: csv
> > print_output_file[print_wan0_out]: out_traffic.csv
> > !
> > aggregate[print_wan0_in]: dst_host, src_port, dst_port, proto
> > aggregate[print_wan0_out]: src_host, src_port, dst_port, proto
> > !
> > sql_table[wan0_in]: traffic_wan0_in_%Y%m%d_%H%M
> > sql_table[wan0_out]: traffic_wan0_out_%Y%m%d_%H%M
> > !
> > sql_table_schema[wan0_in]: traffic_wan0_in.schema
> > sql_table_schema[wan0_out]: traffic_wan0_out.schema
> > !
> > sql_host: localhost
> > sql_db : uacct
> > sql_user : uacct
> > sql_passwd: uacct
> > sql_refresh_time: 30
> > sql_optimize_clauses: true
> > sql_history : 24h
> > sql_history_roundoff: mhd
> > !
> > aggregate[wan0_in]: dst_host, src_port, dst_port, proto
> > aggregate[wan0_out]: src_host, src_port, dst_port, proto
> >
> > Where pretag2.map:
> > set_tag=1 filter='src net 192.168.28.0/24 or src net 192.168.100.0/24'
> > set_tag=2 filter='dst net 192.168.28.0/24 or dst net 192.168.100.0/24'
> >
> > The issue I have with the above config is that no traffic is being
> > collected at all. I confirm that when removing the pre_tag filters,
> traffic
> > is collected, though it is not sorted per direction as I would like to
> > have.
> > Can I use pre_tag_map and pre_tag_filter with uacctd? I don't see any
> > examples for uacctd at
> > https://github.com/pmacct/pmacct/blob/master/examples/pretag.map.example
> .
> >
> > Thanx,
> > Alex
> >
> > On Thu, Feb 20, 2020 at 6:33 PM Alex K  wrote:
> >
> > > Hi all,
> > >
> > > I have a router with multiple interfaces and will need to account
> traffic
> > > at its several WAN interfaces. My purpose is toaccount the traffic
> with the
> > > tuple details and the direction.
> > >
> > > As a test I have compiled the following simple configuration for
> pmacctd:
> > >
> > > !
> > > daemonize: true
> > > plugins: print[wan0_in], print[wan0_out]
> > > print_refresh_time: 10
> > > print_history: 15m
> > > !
> > > print_output[wan0_in]: csv
> > > print_output_file[wan0_in]: in_traffic.csv
> > > print_output[wan0_out]: csv
> > > print_output_file[wan0_out]: out_traffic.csv
> > > !
> > > aggregate[wan0_in]: src_host, dst_host, src_port, dst_port, tag
> > > aggregate[wan0_out]: src_host, dst_host, src_port, dst_port, tag
> > > !
> > > pre_tag_filter[wan0_in]:1
> 

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-22 Thread Paolo Lucente


Hi Alex,

Is it possible with the new setup - the one where pre_tag_map does not
match anything - the traffic is VLAN-tagged (or MPLS-labelled)? If so,
you should adjust filters accordingly and add 'vlan and', ie. "vlan and
src net 192.168.28.0/24 or vlan and src net 192.168.100.0/24".

Paolo
 
On Fri, Feb 21, 2020 at 01:04:25PM +0200, Alex K wrote:
> Working further on this, it seems that for pmacct is sufficient to filter
> traffic using only the pre_tag_filter, thus no need for the aggregation
> filters.
> The issue with this setup though is that I loose the information of the
> pre_nat source IP address when monitoring at the WAN interfaces. Due to
> this I am switching to uacctd as following:
> 
> !
> daemonize: true
> promisc:   false
> uacctd_group: 1
> !networks_file: networks.lst
> !ports_file: ports.lst
> !
> pre_tag_map: pretag2.map
> pre_tag_filter[print_wan0_in]: 1
> pre_tag_filter[print_wan0_out]: 2
> pre_tag_filter[wan0_in]: 1
> pre_tag_filter[wan0_out]: 2
> !
> plugins: print[print_wan0_in], print[print_wan0_out], mysql[wan0_in],
> mysql[wan0_out]
> plugin_pipe_size[wan0_in]: 1024000
> plugin_pipe_size[wan0_out]: 1024000
> print_refresh_time: 10
> print_history: 15m
> print_output_file_append: true
> !
> print_output[print_wan0_in]: csv
> print_output_file[print_wan0_in]: in_traffic.csv
> print_output[print_wan0_out]: csv
> print_output_file[print_wan0_out]: out_traffic.csv
> !
> aggregate[print_wan0_in]: dst_host, src_port, dst_port, proto
> aggregate[print_wan0_out]: src_host, src_port, dst_port, proto
> !
> sql_table[wan0_in]: traffic_wan0_in_%Y%m%d_%H%M
> sql_table[wan0_out]: traffic_wan0_out_%Y%m%d_%H%M
> !
> sql_table_schema[wan0_in]: traffic_wan0_in.schema
> sql_table_schema[wan0_out]: traffic_wan0_out.schema
> !
> sql_host: localhost
> sql_db : uacct
> sql_user : uacct
> sql_passwd: uacct
> sql_refresh_time: 30
> sql_optimize_clauses: true
> sql_history : 24h
> sql_history_roundoff: mhd
> !
> aggregate[wan0_in]: dst_host, src_port, dst_port, proto
> aggregate[wan0_out]: src_host, src_port, dst_port, proto
> 
> Where pretag2.map:
> set_tag=1 filter='src net 192.168.28.0/24 or src net 192.168.100.0/24'
> set_tag=2 filter='dst net 192.168.28.0/24 or dst net 192.168.100.0/24'
> 
> The issue I have with the above config is that no traffic is being
> collected at all. I confirm that when removing the pre_tag filters, traffic
> is collected, though it is not sorted per direction as I would like to
> have.
> Can I use pre_tag_map and pre_tag_filter with uacctd? I don't see any
> examples for uacctd at
> https://github.com/pmacct/pmacct/blob/master/examples/pretag.map.example.
> 
> Thanx,
> Alex
> 
> On Thu, Feb 20, 2020 at 6:33 PM Alex K  wrote:
> 
> > Hi all,
> >
> > I have a router with multiple interfaces and will need to account traffic
> > at its several WAN interfaces. My purpose is toaccount the traffic with the
> > tuple details and the direction.
> >
> > As a test I have compiled the following simple configuration for pmacctd:
> >
> > !
> > daemonize: true
> > plugins: print[wan0_in], print[wan0_out]
> > print_refresh_time: 10
> > print_history: 15m
> > !
> > print_output[wan0_in]: csv
> > print_output_file[wan0_in]: in_traffic.csv
> > print_output[wan0_out]: csv
> > print_output_file[wan0_out]: out_traffic.csv
> > !
> > aggregate[wan0_in]: src_host, dst_host, src_port, dst_port, tag
> > aggregate[wan0_out]: src_host, dst_host, src_port, dst_port, tag
> > !
> > pre_tag_filter[wan0_in]:1
> > pre_tag_filter[wan0_out]:2
> > !
> > pcap_interface: eth0
> > pre_tag_map: pretag.map
> > networks_file: networks.lst
> > ports_file: ports.lst
> > !
> >
> > where pretag.map is:
> > set_tag=1 filter='ether dst 52:54:00:69:a6:0b'
> > set_tag=2 filter='ether src 52:54:00:69:a6:0b'
> >
> > and networks.lst is:
> > 10.100.100.0/24
> >
> > It seems that the details output at the CSV are correctly filtered
> > according to the tag, thus recording the direction also, based on the MAC
> > address of the WAN0 interface.
> >
> > Is this the correct approach to achieve this or is there any other
> > recommended way? Do I need to use aggregate_filters?
> >
> > Also, although I have set a network filter to capture only 10.100.100.0/24,
> > I observe several networks in/out being collected, indicating that the
> > network_file directive is ignored or I have misunderstood its purpose. My
> > purpose it to collect traffic only generated from subnets that belong to
> > configured interfaces of the router.
> >
> > Thanx for your feedback!
> > Alex
> >
> >
> >

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-21 Thread Alex K
Working further on this, it seems that for pmacct is sufficient to filter
traffic using only the pre_tag_filter, thus no need for the aggregation
filters.
The issue with this setup though is that I loose the information of the
pre_nat source IP address when monitoring at the WAN interfaces. Due to
this I am switching to uacctd as following:

!
daemonize: true
promisc:   false
uacctd_group: 1
!networks_file: networks.lst
!ports_file: ports.lst
!
pre_tag_map: pretag2.map
pre_tag_filter[print_wan0_in]: 1
pre_tag_filter[print_wan0_out]: 2
pre_tag_filter[wan0_in]: 1
pre_tag_filter[wan0_out]: 2
!
plugins: print[print_wan0_in], print[print_wan0_out], mysql[wan0_in],
mysql[wan0_out]
plugin_pipe_size[wan0_in]: 1024000
plugin_pipe_size[wan0_out]: 1024000
print_refresh_time: 10
print_history: 15m
print_output_file_append: true
!
print_output[print_wan0_in]: csv
print_output_file[print_wan0_in]: in_traffic.csv
print_output[print_wan0_out]: csv
print_output_file[print_wan0_out]: out_traffic.csv
!
aggregate[print_wan0_in]: dst_host, src_port, dst_port, proto
aggregate[print_wan0_out]: src_host, src_port, dst_port, proto
!
sql_table[wan0_in]: traffic_wan0_in_%Y%m%d_%H%M
sql_table[wan0_out]: traffic_wan0_out_%Y%m%d_%H%M
!
sql_table_schema[wan0_in]: traffic_wan0_in.schema
sql_table_schema[wan0_out]: traffic_wan0_out.schema
!
sql_host: localhost
sql_db : uacct
sql_user : uacct
sql_passwd: uacct
sql_refresh_time: 30
sql_optimize_clauses: true
sql_history : 24h
sql_history_roundoff: mhd
!
aggregate[wan0_in]: dst_host, src_port, dst_port, proto
aggregate[wan0_out]: src_host, src_port, dst_port, proto

Where pretag2.map:
set_tag=1 filter='src net 192.168.28.0/24 or src net 192.168.100.0/24'
set_tag=2 filter='dst net 192.168.28.0/24 or dst net 192.168.100.0/24'

The issue I have with the above config is that no traffic is being
collected at all. I confirm that when removing the pre_tag filters, traffic
is collected, though it is not sorted per direction as I would like to
have.
Can I use pre_tag_map and pre_tag_filter with uacctd? I don't see any
examples for uacctd at
https://github.com/pmacct/pmacct/blob/master/examples/pretag.map.example.

Thanx,
Alex

On Thu, Feb 20, 2020 at 6:33 PM Alex K  wrote:

> Hi all,
>
> I have a router with multiple interfaces and will need to account traffic
> at its several WAN interfaces. My purpose is toaccount the traffic with the
> tuple details and the direction.
>
> As a test I have compiled the following simple configuration for pmacctd:
>
> !
> daemonize: true
> plugins: print[wan0_in], print[wan0_out]
> print_refresh_time: 10
> print_history: 15m
> !
> print_output[wan0_in]: csv
> print_output_file[wan0_in]: in_traffic.csv
> print_output[wan0_out]: csv
> print_output_file[wan0_out]: out_traffic.csv
> !
> aggregate[wan0_in]: src_host, dst_host, src_port, dst_port, tag
> aggregate[wan0_out]: src_host, dst_host, src_port, dst_port, tag
> !
> pre_tag_filter[wan0_in]:1
> pre_tag_filter[wan0_out]:2
> !
> pcap_interface: eth0
> pre_tag_map: pretag.map
> networks_file: networks.lst
> ports_file: ports.lst
> !
>
> where pretag.map is:
> set_tag=1 filter='ether dst 52:54:00:69:a6:0b'
> set_tag=2 filter='ether src 52:54:00:69:a6:0b'
>
> and networks.lst is:
> 10.100.100.0/24
>
> It seems that the details output at the CSV are correctly filtered
> according to the tag, thus recording the direction also, based on the MAC
> address of the WAN0 interface.
>
> Is this the correct approach to achieve this or is there any other
> recommended way? Do I need to use aggregate_filters?
>
> Also, although I have set a network filter to capture only 10.100.100.0/24,
> I observe several networks in/out being collected, indicating that the
> network_file directive is ignored or I have misunderstood its purpose. My
> purpose it to collect traffic only generated from subnets that belong to
> configured interfaces of the router.
>
> Thanx for your feedback!
> Alex
>
>
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists