Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !
Alessandro, Since conntrack -D -p udp does fix the issue, it's clear conntrack cache is incorrect. The conjecture here is that pmacct docker container is started (or probably, restarted) with the UDP traffic flowing. Linux's connection tracker (conntrack) keeps track on connections, and also acts as a cache in the kernel. Since when this happens docker container is still in the process of being launched, and not all iptables rules are pushed (sort of a race condition), some packets set the conntrack state incorrectly, and it stays until you manually flush them. Is this happening randomly, or is the container (or some container in general) started/restarted before this happens? I see there are some commits in https://github.com/moby/moby that try to address s/t like this [1]. I don't see any other commit relevant to this issue, but it might be worth to try latest docker CE version (and new kernel). Let me know under which conditions this happens, and if you can reproduce it with a newer OS/docker version, and we can take it from there. As a _very last_ resort, and if this happens randomly (which I wouldn't understand why..) one could flush UDP conntrack info regularly, if a) you can afford the perf penalty of doing so, and possibly some frames lost b) you can afford up to X seconds of records not being processed, where X is the periodicity of the flush... ugly. Marc [1] ``` commit 1c4286bcffcdc6668f84570a2754c78cccbbf7e1 Author: Flavio Crisciani Date: Mon Apr 10 17:12:14 2017 -0700 Adding test for docker/docker#8795 When a container was being destroyed was possible to have flows in conntrack left behind on the host. If a flow is present into the conntrack table, the packet processing will skip the POSTROUTING table of iptables and will use the information in conntrack to do the translation. For this reason is possible that long lived flows created towards a container that is destroyed, will actually affect new flows incoming to the host, creating erroneous conditions where traffic cannot reach new containers. The fix takes care of cleaning them up when a container is destroyed. The test of this commit is actually reproducing the condition where an UDP flow is established towards a container that is then destroyed. The test verifies that the flow established is gone after the container is destroyed. Signed-off-by: Flavio Crisciani ``` Issue trying to be fixed: https://github.com/moby/moby/issues/8795. But this is a 2017 commit... I doubt your docker version doesn't have it. I see kubernetes has bug reports as of 2020 a similar problem, which they are obviously fixing in their container mgmt: https://github.com/kubernetes/kubernetes/issues/102559 > > Marc, > > The system is a fresh installed ubuntu20.04 , really nothing installed in the > host, it's a minimal + sshd + docker ... nothing else, no crons, no tasks > running, no daemons. > > For the two lines you noticed swapped > > MASQUERADE all -- 192.168.200.0/24 anywhere > MASQUERADE all -- 172.17.0.0/16anywhere > > I don't think there is any problem in swapping them, because source nets are > different , the first is the docker-bridge and second is docker-0 (unused) > Anyway let's swap them > > !! problem just happened ... let's check with tcpdump > > # docker exec -ti open-nti_nfacct_1 apt install tcpdump>/dev/null && docker > exec -ti open-nti_nfacct_1 tcpdump -n "udp port 20013" > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes > 22:49:37.294518 IP 192.168.200.1.33956 > 192.168.200.2.20013: UDP, length 380 > 22:49:37.295657 IP 192.168.200.1.33956 > 192.168.200.2.20013: UDP, length 380 > 22:49:37.296836 IP 192.168.200.1.33956 > 192.168.200.2.20013: UDP, length 380 > 22:49:37.298055 IP 192.168.200.1.33956 > 192.168.200.2.20013: UDP, length 380 > 22:49:37.299242 IP 192.168.200.1.33956 > 192.168.200.2.20013: UDP, length 380 > 22:49:37.300450 IP 192.168.200.1.33956 > 192.168.200.2.20013: UDP, length 290 > ^C > 6 packets captured > 6 packets received by filter > 0 packets dropped by kernel > > !! what iptables says > > # iptables -t nat -vL POSTROUTING --line-numbers > Chain POSTROUTING (policy ACCEPT 776 packets, 157K bytes) > num pkts bytes target prot opt in out source > destination > 19 540 MASQUERADE all -- any!br-0b2348db16f3 > 192.168.200.0/24 anywhere > 20 0 MASQUERADE all -- any!docker0 172.17.0.0/16 > anywhere > 30 0 MASQUERADE udp -- anyany 192.168.200.2 > 192.168.200.2udp dpt:20013 > 40 0 MASQUERADE tcp -- anyany 192.168.200.3 > 192.168.200.3tcp dpt:8086 > 50 0 MASQUERADE tcp -- anyany 192.168.200.4 > 192.168.200.4tcp dpt:3000 > 60
Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !
Dusan, I'm new in this docker world, I don't know swarm It's think it's a normal docker-compose version 1.29.2, build 5becea4c -- AlexIT ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !
Dusan, Thanks. I seemed to have misunderstood yo before. That sounds like it, yes. After reading through most, this might be _the_ issue: https://github.com/moby/moby/issues/16720#issuecomment-435637740 https://github.com/moby/moby/issues/16720#issuecomment-444862701 Alessandro, can you try the suggested once the container is in failed state? conntrack -D -p udp Marc Missatge de Dusan Pajin del dia dc., 9 de juny 2021 a les 21:54: > > Hi, > > Alessandro, do you use docker-compose or docker swarm (docker stack)? > > The behavior I am referring to is described in number of issues on Github, > for example: > https://github.com/moby/moby/issues/16720 > https://github.com/docker/for-linux/issues/182 > https://github.com/moby/moby/issues/18845 > https://github.com/moby/libnetwork/issues/1994 > https://github.com/robcowart/elastiflow/issues/414 > In some of those issues you will find links to other issues and so on. > > I don't have an explanation why this works for you in some situations and > some not. > SInce that is the case, you might try clearing the conntrack table, which is > described in some of the issues above. > Using the host network is certainly not convenient, but it is doable. > > Kind regards, > Dusan > > > > On Wed, Jun 9, 2021 at 7:37 PM Marc Sune wrote: >> >> Dusan, Alessandro, >> >> Let me answer Dusan first. >> >> Missatge de Dusan Pajin del dia dc., 9 de juny >> 2021 a les 18:08: >> > >> > Hi Alessandro, >> > >> > I would say that this is a "known" issue or behavior in docker which is >> > experienced by everyone who ever wanted to receive syslog, netflow, >> > telemetry or any other similar UDP stream from network devices. When you >> > expose ports in your docker-compose file, the docker will create the IP >> > tables rules to steer the traffic to your container in docker's bridge >> > network, but unfortunately also translate the source IP address of the >> > packets. I am not sure what is the reasoning behind such a behavior. If >> > you try to search for solutions for this issue, you will find some >> > proposals, but none of them used to work in my case. >> >> That is not my understanding. I've also double checked with a devops >> Docker guru in my organization. >> >> In the default network docker mode, masquerading only happens for >> egress traffic not ingress. >> >> I actually tried it locally by running an httpd container (apache2) >> and redirect 8080 on the "host" to port 80 on the container. Container >> is on the docker range, LAN on my laptop is 192.168.1.36, .33 being >> another client in my LAN. >> >> root@d64c65384e87:/usr/local/apache2# tcpdump -l -n >> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode >> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes >> 17:21:49.546067 IP 192.168.1.33.46595 > 172.17.0.3.80: Flags [F.], seq >> 2777556344, ack 4139714538, win 172, options [nop,nop,TS val 21290101 >> ecr 3311681356], length 0 >> 17:21:49.546379 IP 192.168.1.33.46591 > 172.17.0.3.80: Flags [F.], seq >> 3001175791, ack 61192428, win 172, options [nop,nop,TS val 21290101 >> ecr 3311686360], length 0 >> 17:21:49.546402 IP 172.17.0.3.80 > 192.168.1.33.46591: Flags [.], ack >> 1, win 236, options [nop,nop,TS val 3311689311 ecr 21290101], length 0 >> 17:21:49.546845 IP 172.17.0.3.80 > 192.168.1.33.46595: Flags [F.], seq >> 1, ack 1, win 227, options [nop,nop,TS val 3311689311 ecr 21290101], >> length 0 >> 17:21:49.550993 IP 192.168.1.33.46595 > 172.17.0.3.80: Flags [.], ack >> 2, win 172, options [nop,nop,TS val 21290110 ecr 3311689311], length 0 >> >> That works as expected, showing the real 1.33 address. >> >> Mind that there is a lot of confusion, because firewall services in >> the system's OS can interfere with the rules set by the docker daemon >> itself: >> >> https://stackoverflow.com/a/47913950/9321563 >> >> Alessandro, >> >> I need to analyse in detail your rules, but what is clear is that >> "something" is modifying them (see the two first rules)... whether >> these two lines in particular are causing the issue, I am not sure: >> >> Pre: >> >> Chain POSTROUTING (policy ACCEPT) >> target prot opt source destination >> MASQUERADE all -- 192.168.200.0/24 anywhere >> MASQUERADE all -- 172.17.0.0/16anywhere >> MASQUERADE tcp -- 192.168.200.3192.168.200.3tcp dpt:8086 >> MASQUERADE tcp -- 192.168.200.5192.168.200.5tcp dpt:3000 >> MASQUERADE udp -- 192.168.200.9192.168.200.9udp dpt:5 >> MASQUERADE tcp -- 192.168.200.11 192.168.200.11 tcp dpt:9092 >> MASQUERADE udp -- 192.168.200.4192.168.200.4udp dpt:50005 >> MASQUERADE udp -- 192.168.200.8192.168.200.8udp dpt:5600 >> MASQUERADE tcp -- 192.168.200.8192.168.200.8tcp dpt:bgp >> MASQUERADE udp -- 192.168.200.2192.168.200.2udp dpt:20013 >> >> Post: >> >> Chain POSTROUTING
Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !
Dusan, Alessandro, Let me answer Dusan first. Missatge de Dusan Pajin del dia dc., 9 de juny 2021 a les 18:08: > > Hi Alessandro, > > I would say that this is a "known" issue or behavior in docker which is > experienced by everyone who ever wanted to receive syslog, netflow, telemetry > or any other similar UDP stream from network devices. When you expose ports > in your docker-compose file, the docker will create the IP tables rules to > steer the traffic to your container in docker's bridge network, but > unfortunately also translate the source IP address of the packets. I am not > sure what is the reasoning behind such a behavior. If you try to search for > solutions for this issue, you will find some proposals, but none of them used > to work in my case. That is not my understanding. I've also double checked with a devops Docker guru in my organization. In the default network docker mode, masquerading only happens for egress traffic not ingress. I actually tried it locally by running an httpd container (apache2) and redirect 8080 on the "host" to port 80 on the container. Container is on the docker range, LAN on my laptop is 192.168.1.36, .33 being another client in my LAN. root@d64c65384e87:/usr/local/apache2# tcpdump -l -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 17:21:49.546067 IP 192.168.1.33.46595 > 172.17.0.3.80: Flags [F.], seq 2777556344, ack 4139714538, win 172, options [nop,nop,TS val 21290101 ecr 3311681356], length 0 17:21:49.546379 IP 192.168.1.33.46591 > 172.17.0.3.80: Flags [F.], seq 3001175791, ack 61192428, win 172, options [nop,nop,TS val 21290101 ecr 3311686360], length 0 17:21:49.546402 IP 172.17.0.3.80 > 192.168.1.33.46591: Flags [.], ack 1, win 236, options [nop,nop,TS val 3311689311 ecr 21290101], length 0 17:21:49.546845 IP 172.17.0.3.80 > 192.168.1.33.46595: Flags [F.], seq 1, ack 1, win 227, options [nop,nop,TS val 3311689311 ecr 21290101], length 0 17:21:49.550993 IP 192.168.1.33.46595 > 172.17.0.3.80: Flags [.], ack 2, win 172, options [nop,nop,TS val 21290110 ecr 3311689311], length 0 That works as expected, showing the real 1.33 address. Mind that there is a lot of confusion, because firewall services in the system's OS can interfere with the rules set by the docker daemon itself: https://stackoverflow.com/a/47913950/9321563 Alessandro, I need to analyse in detail your rules, but what is clear is that "something" is modifying them (see the two first rules)... whether these two lines in particular are causing the issue, I am not sure: Pre: Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.200.0/24 anywhere MASQUERADE all -- 172.17.0.0/16anywhere MASQUERADE tcp -- 192.168.200.3192.168.200.3tcp dpt:8086 MASQUERADE tcp -- 192.168.200.5192.168.200.5tcp dpt:3000 MASQUERADE udp -- 192.168.200.9192.168.200.9udp dpt:5 MASQUERADE tcp -- 192.168.200.11 192.168.200.11 tcp dpt:9092 MASQUERADE udp -- 192.168.200.4192.168.200.4udp dpt:50005 MASQUERADE udp -- 192.168.200.8192.168.200.8udp dpt:5600 MASQUERADE tcp -- 192.168.200.8192.168.200.8tcp dpt:bgp MASQUERADE udp -- 192.168.200.2192.168.200.2udp dpt:20013 Post: Chain POSTROUTING (policy ACCEPT 4799 packets, 1170K bytes) pkts bytes target prot opt in out source destination 340 20392 MASQUERADE all -- any!br-d662f1cf56fa 192.168.200.0/24 anywhere <-- 453 28712 MASQUERADE all -- any!docker0 172.17.0.0/16 anywhere <-- 0 0 MASQUERADE tcp -- anyany 192.168.200.3 192.168.200.3tcp dpt:8086 0 0 MASQUERADE tcp -- anyany 192.168.200.5 192.168.200.5tcp dpt:3000 0 0 MASQUERADE udp -- anyany 192.168.200.9 192.168.200.9udp dpt:5 0 0 MASQUERADE tcp -- anyany 192.168.200.11 192.168.200.11 tcp dpt:9092 0 0 MASQUERADE udp -- anyany 192.168.200.4 192.168.200.4udp dpt:50005 0 0 MASQUERADE udp -- anyany 192.168.200.8 192.168.200.8udp dpt:5600 0 0 MASQUERADE tcp -- anyany 192.168.200.8 192.168.200.8tcp dpt:bgp 0 0 MASQUERADE udp -- anyany 192.168.200.2 192.168.200.2udp dpt:20013 Which OS are you using in the host? A bit of a moonshot, when the problem occurs can you try manually (using iptabes) to remove the first two rules and set them exactly as in the PRE scenario. Use iptables -t nat -I which allows you to add it in a specific position. I think the problem might be somewhere else though. marc > > What definitely works is not to expose specific ports, but to configure
Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !
Hi Dusan, A know issue? and nobody can solve it ! with udp packets it's a real problem. And in many situations, it-s not possibile di ricetly-attach to host network. For scalability I was thinking to have many istances of the same collector (with docker -scale) , and an nginx as udp-load-balancer, which uses internal-docker-dns to distribute incoming udp packets (in roundrobin) to the istances I'm already doing this thing with sflow and telemetry streams, but there it's working fine, because source-ip is not used. Anyway, why it'is not constant and not predictable , for example 15:21:59.646861 IP xxx.157.228.yyy.50101 > 192.168.200.2.20013: UDP, length 380 15:21:59.648041 IP xxx.157.228.yyy.50101 > 192.168.200.2.20013: UDP, length 380 15:21:59.649240 IP xxx.157.228.yyy.50101 > 192.168.200.2.20013: UDP, length 380 15:21:59.650439 IP xxx.157.228.yyy.50101 > 192.168.200.2.20013: UDP, length 380 15:21:59.651653 IP xxx.157.228.yyy.50101 > 192.168.200.2.20013: UDP, length 380 15:21:59.652839 IP xxx.157.228.yyy.50101 > 192.168.200.2.20013: UDP, length 380 15:21:59.654055 IP xxx.157.228.yyy.50101 > 192.168.200.2.20013: UDP, length 380 15:21:59.655251 IP xxx.157.228.yyy.50101 > 192.168.200.2.20013: UDP, length 380 15:46:12.818232 IP 192.168.200.1.24229 > 192.168.200.2.20013: UDP, length 380 15:46:12.819363 IP 192.168.200.1.24229 > 192.168.200.2.20013: UDP, length 380 15:46:12.820573 IP 192.168.200.1.24229 > 192.168.200.2.20013: UDP, length 380 15:46:12.821750 IP 192.168.200.1.24229 > 192.168.200.2.20013: UDP, length 380 15:46:12.823020 IP 192.168.200.1.24229 > 192.168.200.2.20013: UDP, length 380 15:46:12.824177 IP 192.168.200.1.24229 > 192.168.200.2.20013: UDP, length 380 15:46:12.825387 IP 192.168.200.1.24229 > 192.168.200.2.20013: UDP, length 380 15:46:12.826567 IP 192.168.200.1.24229 > 192.168.200.2.20013: UDP, length 380 What does it cause this two different behaviors? Cheers. -- AlexIT ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !
Hi Alessandro, I would say that this is a "known" issue or behavior in docker which is experienced by everyone who ever wanted to receive syslog, netflow, telemetry or any other similar UDP stream from network devices. When you expose ports in your docker-compose file, the docker will create the IP tables rules to steer the traffic to your container in docker's bridge network, but unfortunately also translate the source IP address of the packets. I am not sure what is the reasoning behind such a behavior. If you try to search for solutions for this issue, you will find some proposals, but none of them used to work in my case. What definitely works is not to expose specific ports, but to configure your container in docker-compose to be attached directly to the host network. In that case, there will be no translation rules and no source NAT and container will be directly connected to all host's network interfaces. In such case, be aware that Docker DNS will not work, so to export information from pmacct container further to kafka, you would need to send it to "localhost", if the kafka container is running on the same host and not to "kafka". This shouldn't be a big problem in your setup. Btw, I am using docker swarm and not docker-compose, although they both use docker-compose files with similar syntax, but I don't think there is difference in their behavior. Hope this helps Kind regards, Dusan On Wed, Jun 9, 2021 at 3:29 PM Paolo Lucente wrote: > > Hi Alessandro, > > (thanks for the kind words, first and foremost) > > Indeed, the test that Marc proposes is very sound, ie. check the actual > packets coming in "on the wire" with tcpdump: do they really change > sender IP address? > > Let me also confirm that what is used to populate peer_ip_src is the > sender IP address coming straight from the socket (Marc's question) and, > contrary to sFlow, there is typically there is no other way to infer > such info (Alessandro's question). > > Paolo > > > On 9/6/21 14:51, Marc Sune wrote: > > Alessandro, > > > > inline > > > > Missatge de Alessandro Montano | FIBERTELECOM > > del dia dc., 9 de juny 2021 a les 10:12: > >> > >> Hi Paolo (and Marc), > >> > >> this is my first post here ... first of all THANKS FOR YOU GREAT JOB :) > >> > >> I'm using pmacct/nfacctd container from docker-hub > (+kafka+telegraf+influxdb+grafana) and it's really a powerfull tool > >> > >> The sender are JUNIPER MX204 routers, using j-flow (extended netflow) > >> > >> NFACCTD VERSION: > >> NetFlow Accounting Daemon, nfacctd 1.7.6-git [20201226-0 (7ad9d1b)] > >> '--enable-mysql' '--enable-pgsql' '--enable-sqlite3' '--enable-kafka' > '--enable-geoipv2' '--enable-jansson' '--enable-rabbitmq' '--enable-nflog' > '--enable-ndpi' '--enable-zmq' '--enable-avro' '--enable-serdes' > '--enable-redis' '--enable-gnutls' 'AVRO_CFLAGS=-I/usr/local/avro/include' > 'AVRO_LIBS=-L/usr/local/avro/lib -lavro' '--enable-l2' > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > '--enable-st-bins' > >> > >> SYSTEM: > >> Linux 76afde386f6f 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 > UTC 2021 x86_64 GNU/Linux > >> > >> CONFIG: > >> debug: false > >> daemonize: false > >> pidfile: /var/run/nfacctd.pid > >> logfile: /var/log/pmacct/nfacctd.log > >> nfacctd_renormalize: true > >> nfacctd_port: 20013 > >> aggregate[k]: peer_src_ip, peer_dst_ip, in_iface, out_iface, vlan, > sampling_direction, etype, src_as, dst_as, as_path, proto, src_net, > src_mask, dst_net, dst_mask, flows > >> nfacctd_time_new: true > >> plugins: kafka[k] > >> kafka_output[k]: json > >> kafka_topic[k]: nfacct > >> kafka_broker_host[k]: kafka > >> kafka_broker_port[k]: 9092 > >> kafka_refresh_time[k]: 60 > >> kafka_history[k]: 1m > >> kafka_history_roundoff[k]: m > >> kafka_max_writers[k]: 1 > >> kafka_markers[k]: true > >> networks_file_no_lpm: true > >> use_ip_next_hop: true > >> > >> DOCKER-COMPOSE: > >> #Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2 > >> #docker-compose version 1.29.2, build 5becea4c > >> version: "3.9" > >> services: > >>nfacct: > >> networks: > >>- ingress > >> image: pmacct/nfacctd > >> restart: on-failure > >> ports: > >>- "20013:20013/udp" > >> volumes: > >>- /etc/localtime:/etc/localtime > >>- ./nfacct/etc:/etc/pmacct > >>- ./nfacct/lib:/var/lib/pmacct > >>- ./nfacct/log:/var/log/pmacct > >> networks: > >>ingress: > >> name: ingress > >> ipam: > >>config: > >>- subnet: 192.168.200.0/24 > >> > >> My problem is the value of field PEER_IP_SRC ... at start everything > is correct, and it works well for a (long) while ... hours ... days ... > >> I have ten routers so "peer_ip_src": "151.157.228.xxx" where xxx can > easily identify the sender. Perfect. > >> > >> Suddenly ... "peer_ip_src": "192.168.200.1" for all records (and I > loose the sender info!!!) ... > >> > >> It seems that docker-proxy decide to do nat/masquerading and
Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !
Hi Alessandro, (thanks for the kind words, first and foremost) Indeed, the test that Marc proposes is very sound, ie. check the actual packets coming in "on the wire" with tcpdump: do they really change sender IP address? Let me also confirm that what is used to populate peer_ip_src is the sender IP address coming straight from the socket (Marc's question) and, contrary to sFlow, there is typically there is no other way to infer such info (Alessandro's question). Paolo On 9/6/21 14:51, Marc Sune wrote: Alessandro, inline Missatge de Alessandro Montano | FIBERTELECOM del dia dc., 9 de juny 2021 a les 10:12: Hi Paolo (and Marc), this is my first post here ... first of all THANKS FOR YOU GREAT JOB :) I'm using pmacct/nfacctd container from docker-hub (+kafka+telegraf+influxdb+grafana) and it's really a powerfull tool The sender are JUNIPER MX204 routers, using j-flow (extended netflow) NFACCTD VERSION: NetFlow Accounting Daemon, nfacctd 1.7.6-git [20201226-0 (7ad9d1b)] '--enable-mysql' '--enable-pgsql' '--enable-sqlite3' '--enable-kafka' '--enable-geoipv2' '--enable-jansson' '--enable-rabbitmq' '--enable-nflog' '--enable-ndpi' '--enable-zmq' '--enable-avro' '--enable-serdes' '--enable-redis' '--enable-gnutls' 'AVRO_CFLAGS=-I/usr/local/avro/include' 'AVRO_LIBS=-L/usr/local/avro/lib -lavro' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins' SYSTEM: Linux 76afde386f6f 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64 GNU/Linux CONFIG: debug: false daemonize: false pidfile: /var/run/nfacctd.pid logfile: /var/log/pmacct/nfacctd.log nfacctd_renormalize: true nfacctd_port: 20013 aggregate[k]: peer_src_ip, peer_dst_ip, in_iface, out_iface, vlan, sampling_direction, etype, src_as, dst_as, as_path, proto, src_net, src_mask, dst_net, dst_mask, flows nfacctd_time_new: true plugins: kafka[k] kafka_output[k]: json kafka_topic[k]: nfacct kafka_broker_host[k]: kafka kafka_broker_port[k]: 9092 kafka_refresh_time[k]: 60 kafka_history[k]: 1m kafka_history_roundoff[k]: m kafka_max_writers[k]: 1 kafka_markers[k]: true networks_file_no_lpm: true use_ip_next_hop: true DOCKER-COMPOSE: #Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2 #docker-compose version 1.29.2, build 5becea4c version: "3.9" services: nfacct: networks: - ingress image: pmacct/nfacctd restart: on-failure ports: - "20013:20013/udp" volumes: - /etc/localtime:/etc/localtime - ./nfacct/etc:/etc/pmacct - ./nfacct/lib:/var/lib/pmacct - ./nfacct/log:/var/log/pmacct networks: ingress: name: ingress ipam: config: - subnet: 192.168.200.0/24 My problem is the value of field PEER_IP_SRC ... at start everything is correct, and it works well for a (long) while ... hours ... days ... I have ten routers so "peer_ip_src": "151.157.228.xxx" where xxx can easily identify the sender. Perfect. Suddenly ... "peer_ip_src": "192.168.200.1" for all records (and I loose the sender info!!!) ... It seems that docker-proxy decide to do nat/masquerading and translate source_ip for the udp stream. The only way for me to have the correct behavior again is to stop/start the container. How can I fix it? Or, is there an alternative way to obtain the same info (router ip) from inside the netflow stream, and not from the udp packet. Paolo is definitely the right person to answer how "peer_ip_src" is populated. However, there is something that I don't fully understand. To the best of my knowledge, even when binding ports, docker (actually the kernel, configured by docker) shouldn't masquerade traffic at all - if masquerade is truly what happens. And certainly that wouldn't happen "randomly" in the middle of the execution. My first thought would be that this is something related to pmacct itself, and that records are incorrectly generated but traffic is ok. I doubt the linux kernel iptables rules would randomly change the way traffic is manipulated, unless of course, something else on that machine/server is reloading iptables, and the resulting ruleset is _slightly different_ for the traffic flowing towards the docker container, effectively modifying the streams that go to pmacct (e.g. rule priority reording). That _could_ explain why restarting the daemon suddenly works, as order would be fixed. Some more info would be needed to discard an iptables/docker issue: * Dump the iptables -L and iptables -t nat -L before and after the issue and compare. * Use iptables -vL and iptables -t nat -vL to monitor counters, before and after the issue, specially in the NAT table. * Get inside the running container (https://github.com/pmacct/pmacct/blob/master/docs/DOCKER.md#opening-a-shell-on-a-running-container), install tcpdump, and write the pcap to a file, before and after the incident. Since these dumps might contain sensitive data, you can send them anonymized or in private. Hopefully
Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !
Alessandro, inline Missatge de Alessandro Montano | FIBERTELECOM del dia dc., 9 de juny 2021 a les 10:12: > > Hi Paolo (and Marc), > > this is my first post here ... first of all THANKS FOR YOU GREAT JOB :) > > I'm using pmacct/nfacctd container from docker-hub > (+kafka+telegraf+influxdb+grafana) and it's really a powerfull tool > > The sender are JUNIPER MX204 routers, using j-flow (extended netflow) > > NFACCTD VERSION: > NetFlow Accounting Daemon, nfacctd 1.7.6-git [20201226-0 (7ad9d1b)] > '--enable-mysql' '--enable-pgsql' '--enable-sqlite3' '--enable-kafka' > '--enable-geoipv2' '--enable-jansson' '--enable-rabbitmq' '--enable-nflog' > '--enable-ndpi' '--enable-zmq' '--enable-avro' '--enable-serdes' > '--enable-redis' '--enable-gnutls' 'AVRO_CFLAGS=-I/usr/local/avro/include' > 'AVRO_LIBS=-L/usr/local/avro/lib -lavro' '--enable-l2' > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > '--enable-st-bins' > > SYSTEM: > Linux 76afde386f6f 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC > 2021 x86_64 GNU/Linux > > CONFIG: > debug: false > daemonize: false > pidfile: /var/run/nfacctd.pid > logfile: /var/log/pmacct/nfacctd.log > nfacctd_renormalize: true > nfacctd_port: 20013 > aggregate[k]: peer_src_ip, peer_dst_ip, in_iface, out_iface, vlan, > sampling_direction, etype, src_as, dst_as, as_path, proto, src_net, src_mask, > dst_net, dst_mask, flows > nfacctd_time_new: true > plugins: kafka[k] > kafka_output[k]: json > kafka_topic[k]: nfacct > kafka_broker_host[k]: kafka > kafka_broker_port[k]: 9092 > kafka_refresh_time[k]: 60 > kafka_history[k]: 1m > kafka_history_roundoff[k]: m > kafka_max_writers[k]: 1 > kafka_markers[k]: true > networks_file_no_lpm: true > use_ip_next_hop: true > > DOCKER-COMPOSE: > #Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2 > #docker-compose version 1.29.2, build 5becea4c > version: "3.9" > services: > nfacct: > networks: > - ingress > image: pmacct/nfacctd > restart: on-failure > ports: > - "20013:20013/udp" > volumes: > - /etc/localtime:/etc/localtime > - ./nfacct/etc:/etc/pmacct > - ./nfacct/lib:/var/lib/pmacct > - ./nfacct/log:/var/log/pmacct > networks: > ingress: > name: ingress > ipam: > config: > - subnet: 192.168.200.0/24 > > My problem is the value of field PEER_IP_SRC ... at start everything is > correct, and it works well for a (long) while ... hours ... days ... > I have ten routers so "peer_ip_src": "151.157.228.xxx" where xxx can easily > identify the sender. Perfect. > > Suddenly ... "peer_ip_src": "192.168.200.1" for all records (and I loose the > sender info!!!) ... > > It seems that docker-proxy decide to do nat/masquerading and translate > source_ip for the udp stream. > The only way for me to have the correct behavior again is to stop/start the > container. > > How can I fix it? Or, is there an alternative way to obtain the same info > (router ip) from inside the netflow stream, and not from the udp packet. Paolo is definitely the right person to answer how "peer_ip_src" is populated. However, there is something that I don't fully understand. To the best of my knowledge, even when binding ports, docker (actually the kernel, configured by docker) shouldn't masquerade traffic at all - if masquerade is truly what happens. And certainly that wouldn't happen "randomly" in the middle of the execution. My first thought would be that this is something related to pmacct itself, and that records are incorrectly generated but traffic is ok. I doubt the linux kernel iptables rules would randomly change the way traffic is manipulated, unless of course, something else on that machine/server is reloading iptables, and the resulting ruleset is _slightly different_ for the traffic flowing towards the docker container, effectively modifying the streams that go to pmacct (e.g. rule priority reording). That _could_ explain why restarting the daemon suddenly works, as order would be fixed. Some more info would be needed to discard an iptables/docker issue: * Dump the iptables -L and iptables -t nat -L before and after the issue and compare. * Use iptables -vL and iptables -t nat -vL to monitor counters, before and after the issue, specially in the NAT table. * Get inside the running container (https://github.com/pmacct/pmacct/blob/master/docs/DOCKER.md#opening-a-shell-on-a-running-container), install tcpdump, and write the pcap to a file, before and after the incident. Since these dumps might contain sensitive data, you can send them anonymized or in private. Hopefully with this info we will see if it's an iptables issue or we have to look somewhere else. Regards marc > > Thanks for your support. > > Cheers. > > -- > AlexIT > -- > docker-doctors mailing list > docker-doct...@pmacct.net > http://acaraje.pmacct.net/cgi-bin/mailman/listinfo/docker-doctors ___ pmacct-discussion mailing list
