Hi Alessandro, I would say that this is a "known" issue or behavior in docker which is experienced by everyone who ever wanted to receive syslog, netflow, telemetry or any other similar UDP stream from network devices. When you expose ports in your docker-compose file, the docker will create the IP tables rules to steer the traffic to your container in docker's bridge network, but unfortunately also translate the source IP address of the packets. I am not sure what is the reasoning behind such a behavior. If you try to search for solutions for this issue, you will find some proposals, but none of them used to work in my case.
What definitely works is not to expose specific ports, but to configure your container in docker-compose to be attached directly to the host network. In that case, there will be no translation rules and no source NAT and container will be directly connected to all host's network interfaces. In such case, be aware that Docker DNS will not work, so to export information from pmacct container further to kafka, you would need to send it to "localhost", if the kafka container is running on the same host and not to "kafka". This shouldn't be a big problem in your setup. Btw, I am using docker swarm and not docker-compose, although they both use docker-compose files with similar syntax, but I don't think there is difference in their behavior. Hope this helps Kind regards, Dusan On Wed, Jun 9, 2021 at 3:29 PM Paolo Lucente <pa...@pmacct.net> wrote: > > Hi Alessandro, > > (thanks for the kind words, first and foremost) > > Indeed, the test that Marc proposes is very sound, ie. check the actual > packets coming in "on the wire" with tcpdump: do they really change > sender IP address? > > Let me also confirm that what is used to populate peer_ip_src is the > sender IP address coming straight from the socket (Marc's question) and, > contrary to sFlow, there is typically there is no other way to infer > such info (Alessandro's question). > > Paolo > > > On 9/6/21 14:51, Marc Sune wrote: > > Alessandro, > > > > inline > > > > Missatge de Alessandro Montano | FIBERTELECOM > > <a.mont...@fibertelecom.it> del dia dc., 9 de juny 2021 a les 10:12: > >> > >> Hi Paolo (and Marc), > >> > >> this is my first post here ... first of all THANKS FOR YOU GREAT JOB :) > >> > >> I'm using pmacct/nfacctd container from docker-hub > (+kafka+telegraf+influxdb+grafana) and it's really a powerfull tool > >> > >> The sender are JUNIPER MX204 routers, using j-flow (extended netflow) > >> > >> NFACCTD VERSION: > >> NetFlow Accounting Daemon, nfacctd 1.7.6-git [20201226-0 (7ad9d1b)] > >> '--enable-mysql' '--enable-pgsql' '--enable-sqlite3' '--enable-kafka' > '--enable-geoipv2' '--enable-jansson' '--enable-rabbitmq' '--enable-nflog' > '--enable-ndpi' '--enable-zmq' '--enable-avro' '--enable-serdes' > '--enable-redis' '--enable-gnutls' 'AVRO_CFLAGS=-I/usr/local/avro/include' > 'AVRO_LIBS=-L/usr/local/avro/lib -lavro' '--enable-l2' > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > '--enable-st-bins' > >> > >> SYSTEM: > >> Linux 76afde386f6f 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 > UTC 2021 x86_64 GNU/Linux > >> > >> CONFIG: > >> debug: false > >> daemonize: false > >> pidfile: /var/run/nfacctd.pid > >> logfile: /var/log/pmacct/nfacctd.log > >> nfacctd_renormalize: true > >> nfacctd_port: 20013 > >> aggregate[k]: peer_src_ip, peer_dst_ip, in_iface, out_iface, vlan, > sampling_direction, etype, src_as, dst_as, as_path, proto, src_net, > src_mask, dst_net, dst_mask, flows > >> nfacctd_time_new: true > >> plugins: kafka[k] > >> kafka_output[k]: json > >> kafka_topic[k]: nfacct > >> kafka_broker_host[k]: kafka > >> kafka_broker_port[k]: 9092 > >> kafka_refresh_time[k]: 60 > >> kafka_history[k]: 1m > >> kafka_history_roundoff[k]: m > >> kafka_max_writers[k]: 1 > >> kafka_markers[k]: true > >> networks_file_no_lpm: true > >> use_ip_next_hop: true > >> > >> DOCKER-COMPOSE: > >> #Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2 > >> #docker-compose version 1.29.2, build 5becea4c > >> version: "3.9" > >> services: > >> nfacct: > >> networks: > >> - ingress > >> image: pmacct/nfacctd > >> restart: on-failure > >> ports: > >> - "20013:20013/udp" > >> volumes: > >> - /etc/localtime:/etc/localtime > >> - ./nfacct/etc:/etc/pmacct > >> - ./nfacct/lib:/var/lib/pmacct > >> - ./nfacct/log:/var/log/pmacct > >> networks: > >> ingress: > >> name: ingress > >> ipam: > >> config: > >> - subnet: 192.168.200.0/24 > >> > >> My problem is the value of field PEER_IP_SRC ... at start everything > is correct, and it works well for a (long) while ... hours ... days ... > >> I have ten routers so "peer_ip_src": "151.157.228.xxx" where xxx can > easily identify the sender. Perfect. > >> > >> Suddenly ... "peer_ip_src": "192.168.200.1" for all records (and I > loose the sender info!!!) ... > >> > >> It seems that docker-proxy decide to do nat/masquerading and translate > source_ip for the udp stream. > >> The only way for me to have the correct behavior again is to stop/start > the container. > >> > >> How can I fix it? Or, is there an alternative way to obtain the same > info (router ip) from inside the netflow stream, and not from the udp > packet. > > > > Paolo is definitely the right person to answer how "peer_ip_src" is > populated. > > > > However, there is something that I don't fully understand. To the best > > of my knowledge, even when binding ports, docker (actually the kernel, > > configured by docker) shouldn't masquerade traffic at all - if > > masquerade is truly what happens. And certainly that wouldn't happen > > "randomly" in the middle of the execution. > > > > My first thought would be that this is something related to pmacct > > itself, and that records are incorrectly generated but traffic is ok. > > > > I doubt the linux kernel iptables rules would randomly change the way > > traffic is manipulated, unless of course, something else on that > > machine/server is reloading iptables, and the resulting ruleset is > > _slightly different_ for the traffic flowing towards the docker > > container, effectively modifying the streams that go to pmacct (e.g. > > rule priority reording). That _could_ explain why restarting the > > daemon suddenly works, as order would be fixed. > > > > Some more info would be needed to discard an iptables/docker issue: > > > > * Dump the iptables -L and iptables -t nat -L before and after the > > issue and compare. > > * Use iptables -vL and iptables -t nat -vL to monitor counters, before > > and after the issue, specially in the NAT table. > > * Get inside the running container > > ( > https://github.com/pmacct/pmacct/blob/master/docs/DOCKER.md#opening-a-shell-on-a-running-container > ), > > install tcpdump, and write the pcap to a file, before and after the > > incident. > > > > Since these dumps might contain sensitive data, you can send them > > anonymized or in private. > > > > Hopefully with this info we will see if it's an iptables issue or we > > have to look somewhere else. > > > > Regards > > marc > > > >> > >> Thanks for your support. > >> > >> Cheers. > >> > >> -- > >> AlexIT > >> -- > >> docker-doctors mailing list > >> docker-doct...@pmacct.net > >> http://acaraje.pmacct.net/cgi-bin/mailman/listinfo/docker-doctors > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists >
_______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
