Re: [pmacct-discussion] [docker-doctors] pmacctd in docker

[Thomas Eckert]

Great, thanks!
That was fast :-)

Regards,
  Thomas

On Tue, May 10, 2022 at 4:29 PM Paolo Lucente  wrote:

>
> Hi Thomas,
>
> I think some confusion may be deriving from docs (to be improved) and
> the fact 1.7.6 is old.
>
> Nevertheless, from the interface indexes from your last output (ie.
> 1872541466, 3698069186, etc.) i can tell that you did configure
> pcap_ifindex to 'hash' (being honored as you can see) in conjunction
> with pcap_interfaces_map.
>
> One issue in the code is for sure the fact to require the definition of
> an ifindex always, even if pcap_ifindex is not set to 'map'. Another
> issue was the silent discarding of pcap_interfaces_map without notifying
> you with a warning. Both of these issues have been addressed in this
> commit that i just passed:
>
>
> https://github.com/pmacct/pmacct/commit/02080179aef3e87527e4d1158700eee729f1a5c3
>
> Paolo
>
>
> On 9/5/22 14:31, Thomas Eckert wrote:
> > Hi Paolo,
> >
> > Thanks for the hint, I gave it a try. I'm observing the exact same
> > behavior between running pmacct in a container & directly on my host in
> > all cases. Tested with
> > * official docker image: 281904b7afd6
> > * official ubuntu 21.10 package: pmacct/impish,now 1.7.6-2 amd64
> >
> > I *think* the problem is with the interfaces' ifindex parameter when
> > using the pcap_interfaces_map config key - everything works fine
> > (capture files are printed) when instead using the pcap_interface key.
> > Whenever I do not specify the 'ifindex' in the file specified as value
> > for the pcap_interfaces_map config key, I do not observe capture files
> > being printed. Vice versa, if I do specify the 'ifindex' parameter, then
> > capture files are printed.
> >
> > In fact, if I do specify 'ifindex' for all interfaces listed when I run
> > "netstat -i", then pmacctd throws errors for my br-* & enx interfaces -
> > which it does not do when I omit 'ifindex' - almost as if it only then
> > realizes that it is supposed to access those interfaces at all. This
> > assumption is also based on the fact that I do see log lines such as
> these
> >  INFO ( default/core ): Reading configuration file
> > '/etc/pmacct/pmacctd.conf'.
> >  INFO ( default/core ): [/etc/pmacct/pcap-itf.conf] (re)loading map.
> >  INFO ( default/core ): [/etc/pmacct/pcap-itf.conf] map successfully
> > (re)loaded.
> >  INFO ( default/core ): [docker0,1872541466] link type is: 1  <=
> >  INFO ( default/core ): [eno2,3698069186] link type is: 1
>  <=
> >  INFO ( default/core ): [lo,2529615826] link type is: 1
> ><=
> >  INFO ( default/core ): [tun0,3990258693] link type is: 12
> <=
> > when specifying 'ifname' whereas the marked (<=) lines are missing
> > whenever I do not.
> >
> > Reading through the config key documentation some more, I found the
> > config key pcap_ifindex. Interestingly enough, using it does not yield
> > any difference in results - neither for value "sys" nor for value "hash"
> > - irrespective of all other settings I played around with.
> >
> > Assuming in pmacctd.conf the config key pcap_interfaces_map is used,
> > then this is what I speculate is effectively happening:
> > * pmacctd ignores config key pcap_ifindex
> > * instead, it expects 'ifindex' to be set in the interface mapping file
> > for each line
> > * each line where 'ifindex' is not set is ignored
> > * if 'ifindex' is missing on all lines, this results in a
> > "no-interface-being-listened-on" case without any warning/error
> > Summary: seems like 'ifname' is a mandatory parameter in the interface
> > mapping file whereas the documentation says "pmacctd: mandatory keys:
> > ifname."
> >
> > My understanding of the documentation for above-mentioned config keys is
> > that the behavior I'm observing is not as intended (e.g. 'ifindex'
> > effectively being required, pcap_ifindex effectively being ignored) . So
> > I'm either making a mistake, e.g. in my config files, misunderstanding
> > the documentation or I'm encountering a bug - which I find difficult to
> > believe given how trivial my setup is.
> >
> > Any Suggestions ?
> >
> > Regards & Thanks,
> >Thomas
> >
> > On Sun, May 8, 2022 at 1:43 PM Paolo Lucente  > > wrote:
> >
> >
> > Hi Thomas,
> >
> > The simplest thing i may recommend is to check it all working
> outside a
> > container - this way you can easily isolate whether the issue is
> > somehow
> > related to the container (config or interaction of pmacctd with the
> > container) or with the pmacct config itself.
> >
> > Paolo
> >
> >
> > On 6/5/22 06:05, Thomas Eckert wrote:
> >  > Hi everyone,
> >  >
> >  > pmacct starter here, trying to get pmacctd working inside of a
> > container
> >  > to listen to the (container's) host's traffic. I suppose this is
> > a, if
> >  > not the, standard use case for pmacctd in a container. So I'm
> > sure it
> >  > works in principle but I'm doing something 

Re: [pmacct-discussion] [docker-doctors] pmacctd in docker

[Paolo Lucente]

Hi Thomas,

I think some confusion may be deriving from docs (to be improved) and 
the fact 1.7.6 is old.


Nevertheless, from the interface indexes from your last output (ie. 
1872541466, 3698069186, etc.) i can tell that you did configure 
pcap_ifindex to 'hash' (being honored as you can see) in conjunction 
with pcap_interfaces_map.


One issue in the code is for sure the fact to require the definition of 
an ifindex always, even if pcap_ifindex is not set to 'map'. Another 
issue was the silent discarding of pcap_interfaces_map without notifying 
you with a warning. Both of these issues have been addressed in this 
commit that i just passed:


https://github.com/pmacct/pmacct/commit/02080179aef3e87527e4d1158700eee729f1a5c3

Paolo


On 9/5/22 14:31, Thomas Eckert wrote:

Hi Paolo,

Thanks for the hint, I gave it a try. I'm observing the exact same 
behavior between running pmacct in a container & directly on my host in 
all cases. Tested with

* official docker image: 281904b7afd6
* official ubuntu 21.10 package: pmacct/impish,now 1.7.6-2 amd64

I *think* the problem is with the interfaces' ifindex parameter when 
using the pcap_interfaces_map config key - everything works fine 
(capture files are printed) when instead using the pcap_interface key. 
Whenever I do not specify the 'ifindex' in the file specified as value 
for the pcap_interfaces_map config key, I do not observe capture files 
being printed. Vice versa, if I do specify the 'ifindex' parameter, then 
capture files are printed.


In fact, if I do specify 'ifindex' for all interfaces listed when I run 
"netstat -i", then pmacctd throws errors for my br-* & enx interfaces - 
which it does not do when I omit 'ifindex' - almost as if it only then 
realizes that it is supposed to access those interfaces at all. This 
assumption is also based on the fact that I do see log lines such as these
     INFO ( default/core ): Reading configuration file 
'/etc/pmacct/pmacctd.conf'.

     INFO ( default/core ): [/etc/pmacct/pcap-itf.conf] (re)loading map.
     INFO ( default/core ): [/etc/pmacct/pcap-itf.conf] map successfully 
(re)loaded.

     INFO ( default/core ): [docker0,1872541466] link type is: 1      <=
     INFO ( default/core ): [eno2,3698069186] link type is: 1           <=
     INFO ( default/core ): [lo,2529615826] link type is: 1  
   <=

     INFO ( default/core ): [tun0,3990258693] link type is: 12          <=
when specifying 'ifname' whereas the marked (<=) lines are missing 
whenever I do not.


Reading through the config key documentation some more, I found the 
config key pcap_ifindex. Interestingly enough, using it does not yield 
any difference in results - neither for value "sys" nor for value "hash" 
- irrespective of all other settings I played around with.


Assuming in pmacctd.conf the config key pcap_interfaces_map is used, 
then this is what I speculate is effectively happening:

* pmacctd ignores config key pcap_ifindex
* instead, it expects 'ifindex' to be set in the interface mapping file 
for each line

* each line where 'ifindex' is not set is ignored
* if 'ifindex' is missing on all lines, this results in a 
"no-interface-being-listened-on" case without any warning/error
Summary: seems like 'ifname' is a mandatory parameter in the interface 
mapping file whereas the documentation says "pmacctd: mandatory keys: 
ifname."


My understanding of the documentation for above-mentioned config keys is 
that the behavior I'm observing is not as intended (e.g. 'ifindex' 
effectively being required, pcap_ifindex effectively being ignored) . So 
I'm either making a mistake, e.g. in my config files, misunderstanding 
the documentation or I'm encountering a bug - which I find difficult to 
believe given how trivial my setup is.


Any Suggestions ?

Regards & Thanks,
   Thomas

On Sun, May 8, 2022 at 1:43 PM Paolo Lucente > wrote:



Hi Thomas,

The simplest thing i may recommend is to check it all working outside a
container - this way you can easily isolate whether the issue is
somehow
related to the container (config or interaction of pmacctd with the
container) or with the pmacct config itself.

Paolo


On 6/5/22 06:05, Thomas Eckert wrote:
 > Hi everyone,
 >
 > pmacct starter here, trying to get pmacctd working inside of a
container
 > to listen to the (container's) host's traffic. I suppose this is
a, if
 > not the, standard use case for pmacctd in a container. So I'm
sure it
 > works in principle but I'm doing something wrong.
 >
 > Command for starting the container:
 >      docker run \
 >          --privileged --network=host \
 >          --name pmacctd \
 >          -v /tmp/pmacctd.conf:/etc/pmacct/pmacctd.conf:ro \
 >          -v /tmp/pcap-itf.conf:/etc/pmacct/pcap-itf.conf:ro \
 >          -v /tmp//captures:/var/pmacct/captures:rw pmacctd-debug \
 >          pmacct/pmacctd:latest
  

Re: [pmacct-discussion] [docker-doctors] pmacctd in docker

[Thomas Eckert]

Hi Paolo,

Thanks for the hint, I gave it a try. I'm observing the exact same behavior
between running pmacct in a container & directly on my host in all cases.
Tested with
* official docker image: 281904b7afd6
* official ubuntu 21.10 package: pmacct/impish,now 1.7.6-2 amd64

I *think* the problem is with the interfaces' ifindex parameter when using
the pcap_interfaces_map config key - everything works fine (capture files
are printed) when instead using the pcap_interface key. Whenever I do not
specify the 'ifindex' in the file specified as value for the
pcap_interfaces_map config key, I do not observe capture files being
printed. Vice versa, if I do specify the 'ifindex' parameter, then capture
files are printed.

In fact, if I do specify 'ifindex' for all interfaces listed when I run
"netstat -i", then pmacctd throws errors for my br-* & enx interfaces -
which it does not do when I omit 'ifindex' - almost as if it only then
realizes that it is supposed to access those interfaces at all. This
assumption is also based on the fact that I do see log lines such as these
INFO ( default/core ): Reading configuration file
'/etc/pmacct/pmacctd.conf'.
INFO ( default/core ): [/etc/pmacct/pcap-itf.conf] (re)loading map.
INFO ( default/core ): [/etc/pmacct/pcap-itf.conf] map successfully
(re)loaded.
INFO ( default/core ): [docker0,1872541466] link type is: 1  <=
INFO ( default/core ): [eno2,3698069186] link type is: 1   <=
INFO ( default/core ): [lo,2529615826] link type is: 1<=
INFO ( default/core ): [tun0,3990258693] link type is: 12  <=
when specifying 'ifname' whereas the marked (<=) lines are missing whenever
I do not.

Reading through the config key documentation some more, I found the config
key pcap_ifindex. Interestingly enough, using it does not yield any
difference in results - neither for value "sys" nor for value "hash" -
irrespective of all other settings I played around with.

Assuming in pmacctd.conf the config key pcap_interfaces_map is used, then
this is what I speculate is effectively happening:
* pmacctd ignores config key pcap_ifindex
* instead, it expects 'ifindex' to be set in the interface mapping file for
each line
* each line where 'ifindex' is not set is ignored
* if 'ifindex' is missing on all lines, this results in a
"no-interface-being-listened-on" case without any warning/error
Summary: seems like 'ifname' is a mandatory parameter in the interface
mapping file whereas the documentation says "pmacctd: mandatory keys:
ifname."

My understanding of the documentation for above-mentioned config keys is
that the behavior I'm observing is not as intended (e.g. 'ifindex'
effectively being required, pcap_ifindex effectively being ignored) . So
I'm either making a mistake, e.g. in my config files, misunderstanding the
documentation or I'm encountering a bug - which I find difficult to believe
given how trivial my setup is.

Any Suggestions ?

Regards & Thanks,
  Thomas

On Sun, May 8, 2022 at 1:43 PM Paolo Lucente  wrote:

>
> Hi Thomas,
>
> The simplest thing i may recommend is to check it all working outside a
> container - this way you can easily isolate whether the issue is somehow
> related to the container (config or interaction of pmacctd with the
> container) or with the pmacct config itself.
>
> Paolo
>
>
> On 6/5/22 06:05, Thomas Eckert wrote:
> > Hi everyone,
> >
> > pmacct starter here, trying to get pmacctd working inside of a container
> > to listen to the (container's) host's traffic. I suppose this is a, if
> > not the, standard use case for pmacctd in a container. So I'm sure it
> > works in principle but I'm doing something wrong.
> >
> > Command for starting the container:
> >  docker run \
> >  --privileged --network=host \
> >  --name pmacctd \
> >  -v /tmp/pmacctd.conf:/etc/pmacct/pmacctd.conf:ro \
> >  -v /tmp/pcap-itf.conf:/etc/pmacct/pcap-itf.conf:ro \
> >  -v /tmp//captures:/var/pmacct/captures:rw pmacctd-debug \
> >  pmacct/pmacctd:latest
> >
> > Contents of pmacctd.conf:
> >  daemonize: false
> >  snaplen: 1000
> >  pcap_interfaces_map: /etc/pmacct/pcap-itf.conf
> >  aggregate: src_host, dst_host, src_port, dst_port, proto, class
> >  plugins: print
> >  print_output: json
> >  print_output_file: /var/pmacct/captures/capture-%Y%m%d_%H%M.txt
> >  print_output_file_append: true
> >  print_history: 1m
> >  print_history_roundoff: m
> >  print_refresh_time: 5
> >
> > pcap-itf.conf contains all interfaces of the host (as per netstat -i) in
> > the form
> >  ifname=eno2
> > One line each, no other keys/values other than ifname.
> > Possibly important note: There's a VPN (openconnect) constantly running
> > on the host. The VPN's interface is listed in netstat -i and, as such,
> > included in pcap-itf.conf.
> >
> > Starting the container yields this output:
> >  INFO ( default/core ): Promiscuous Mode 

Re: [pmacct-discussion] [docker-doctors] pmacctd in docker

[Paolo Lucente]

Hi Thomas,

The simplest thing i may recommend is to check it all working outside a 
container - this way you can easily isolate whether the issue is somehow 
related to the container (config or interaction of pmacctd with the 
container) or with the pmacct config itself.


Paolo


On 6/5/22 06:05, Thomas Eckert wrote:

Hi everyone,

pmacct starter here, trying to get pmacctd working inside of a container 
to listen to the (container's) host's traffic. I suppose this is a, if 
not the, standard use case for pmacctd in a container. So I'm sure it 
works in principle but I'm doing something wrong.


Command for starting the container:
     docker run \
         --privileged --network=host \
         --name pmacctd \
         -v /tmp/pmacctd.conf:/etc/pmacct/pmacctd.conf:ro \
         -v /tmp/pcap-itf.conf:/etc/pmacct/pcap-itf.conf:ro \
         -v /tmp//captures:/var/pmacct/captures:rw pmacctd-debug \
         pmacct/pmacctd:latest

Contents of pmacctd.conf:
     daemonize: false
     snaplen: 1000
     pcap_interfaces_map: /etc/pmacct/pcap-itf.conf
     aggregate: src_host, dst_host, src_port, dst_port, proto, class
     plugins: print
     print_output: json
     print_output_file: /var/pmacct/captures/capture-%Y%m%d_%H%M.txt
     print_output_file_append: true
     print_history: 1m
     print_history_roundoff: m
     print_refresh_time: 5

pcap-itf.conf contains all interfaces of the host (as per netstat -i) in 
the form

     ifname=eno2
One line each, no other keys/values other than ifname.
Possibly important note: There's a VPN (openconnect) constantly running 
on the host. The VPN's interface is listed in netstat -i and, as such, 
included in pcap-itf.conf.


Starting the container yields this output:
     INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd 
1.7.7-git (20211107-0 (ef37a415))
     INFO ( default/core ):  '--enable-mysql' '--enable-pgsql' 
'--enable-sqlite3' '--enable-kafka' '--enable-geoipv2' 
'--enable-jansson' '--enable-rabbitmq' '--enable-nflog' '--enable-ndpi' 
'--enable-zmq' '--enable-avro' '--enable-serdes' '--enable-redis' 
'--enable-gnutls' 'AVRO_CFLAGS=-I/usr/local/avro/include' 
'AVRO_LIBS=-L/usr/local/avro/lib -lavro' '--enable-l2' 
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' 
'--enable-st-bins'
     INFO ( default/core ): Reading configuration file 
'/etc/pmacct/pmacctd.conf'.

     INFO ( default/core ): [/etc/pmacct/pcap-itf.conf] (re)loading map.
     INFO ( default/core ): [/etc/pmacct/pcap-itf.conf] map successfully 
(re)loaded.
     INFO ( default_print/print ): cache entries=16411 base cache 
memory=67875896 bytes

     INFO ( default_print/print ): JSON: setting object handlers.
     INFO ( default_print/print ): *** Purging cache - START (PID: 7) ***
     INFO ( default_print/print ): *** Purging cache - END (PID: 7, QN: 
0/0, ET: X) ***


Now, the problem is there are no files showing up in the 'captures' 
directory at all.


I tried these things  (as well as combinations thereof) to try to 
understand what's going on:
* change the time related settings in pmacct.conf: to dump data 
more/less often - also waited (increasingly) long, at times up to 20 minutes
* change 'snaplen' in pmacct.conf up & down - just to make sure I'm not 
running into buffering problems (just guessing, haven't read pmacct/d 
sources)
* change pcap-itf.conf to contain all interfaces or only the (host's) 
LAN + VPN interfaces (removing all others like docker's internal 'docker0')
* check permission settings of the 'captures' directory - this should be 
fine because a simple "touch /var/pmacct/captures/foobar" works and the 
file does exist as observed in the directory on the host itself
* run the container _not_ in host-sniffing mode, so just inside its own 
network-bubble, then cause traffic against it and observe it writing 
data to the 'captures' directory - works!


Because I started to doubt my own sanity I asked one of our Docker/K8S 
experts to check my docker setup and he found no problem looking over 
it, including via "docker inspect pmacct". So I'm fairly sure my mistake 
is somewhere in the configuration of pmacctd but I cannot figure out 
what is. Would someone please point it out to me ?


Regards & Thanks,
   Thomas

PS: It's been almost 10 years since I've posted to a mailing list. 
Please forgive any conventions/best-practices missteps.



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists