Re: [pmacct-discussion] [PATCH 0/2] patchset/questions to support pcap_setdirection and ifindex handling in nfprobe
Hi guys, Would be great if someone of the maintainers could comment on my patches, and give some feedback on whether they make sense or propose and alternative/better configuration to achieve a similar setup. Thanks a lot, Mikhail On Fri, 17 Jan 2020 at 13:02, Mikhail Sennikovsky wrote: > > Hi all, > > I had experienced some issues with configuring multiple pcap interfaces with > pmacctd and wanted to clarify them with you and potentially ask for a > better/alternative solution/configuration. > The two patches submitted here actually illustrate the problems I encountered > and the way I had to fix them. > In my setup I have a "firewall entity" which forwards all traffic between two > network interfaces, doing the necessary traffic firewalling/filtering, i.e. > something like: > +--+ > <---> | InterfaceA <->(Firewall)<-> InterfaceB | <---> > +--+ > now I want to generate a NetFlow v9 / IPFIX data for the traffic the Firewall > instance allows in both directions, which means for all egress traffic for > InterfaceA and all egress traffic for InterfaceB. > > So I tried to use pmacctd + its nfprobe plugin for doing that (configs are > listed below).I configured pmacctd with multiple pcap interfaces, specifying > pcap_interfaces_map and the nfprobe plugin (configs are listed below). > The first problem I encountered is that pmacctd does not configure pcap > direction. > I.e. I can not use pcap filter, because I do not have any specific traffic > pattern to determine the egress traffic for both interfaces (i.e. all traffic > is being forwarded between those two interfaces, potentially filtered with > some "firewall rules" in place.) > The pcap direction configuration was explicitly disabled in > 81fe649917036b9ef1ed4b3ea521befcaf36496b, however even before that commit it > apparently did not work, because the pcap_setdirection must be called after > pcap_activate, not before it. > So my first patch actually makes it possible to configure pmacctd to do > pcap_setdirection by introducing a new config variable, pcap_set_direction. > > Now after I ended up with pcap direction working, and configured pmacctd to > listen for the egress traffic on both InterfaceA and InterfaceB, I faced the > second problem, which seems to be a real bug in nfprobe plugin. > nfprobe flow tree does not take interface index into consideration when > searching/aggregating the flow data. This means that for the case multiple > pcap interfaces are being monitored and same src/dst ip/port traffic pattern > is being handled over several of those interfaces, this will all land in the > same FLOW entry. This leads to the issues that flows being handled by one > network interface are actually reported via NetFlow (via Flow InputInt and > OutputInt fields) as being handled by another network interface (held by the > FLOW entry originally created for matching the given src/dst ip/port traffic > pattern). > This perhaps could be worked around by having different nfprobe plugin > instances handling InterfaceA and InterfaceB traffic, but I want to get the > NetFlow data for both interfaces simultaneously, and I might also need to > dynamically add/remove such interface pairs w/o restarting the pmacctd. > The latter could be easily achieved with updating the pcap_interfaces_map > file and sending SIGUSR2 to pmacctd. The same would not be possible however > if I had to add/remove nfprobe plugin instance configurations in pmacctd.conf > So my second patch fixes the above issue by making it possible to configure > nfprobe to take flow interface indexes into consideration when > matching/searching for the FLOW entries in the flow cache tree. The > nfprobe_per_interface_flows config is introduced for that. > > Would be great if someone could have a look into these two patches to see if > they make sense, and/or give some hints on a better/proper way of making the > similar NetFlow configuration. > > Here are my configs for the reference: > pmacctd.conf: > > daemonize: false > pidfile: /var/run/pmacctd.pid > syslog: daemon > > pcap_interfaces_map: /path/to/pcap_interfaces.map > pcap_ifindex: map > ! newly introduced config to tell pmacctd to actually do pcap_setdirection > pcap_set_direction: true > promisc: true > > pmacctd_flow_buffer_buckets: 65536 > pmacctd_flow_buffer_size: 128Mb > > plugins: nfprobe[filtered], print[filtered_p] > plugin_pipe_size: 1048576000 > plugin_buffer_size: 10485760 > > aggregate: src_host, dst_host, src_port, dst_port, in_iface, out_iface > aggregate[filtered]: src_host, dst_host, src_port, dst_port, in_iface, > out_iface > aggregate[filtered_p]: src_host, dst_host, src_port, dst_port, in_iface, > out_iface > > pmacctd_as: file > > refresh_maps: true > pre_tag_map: /path/to/pretag.map > > pre_tag_filter[filtered]: -666 > pre_tag_filter[filtered_p]: -666 > > nfprobe_source_ip:
[pmacct-discussion] [PATCH 0/2] patchset/questions to support pcap_setdirection and ifindex handling in nfprobe
Hi all, I had experienced some issues with configuring multiple pcap interfaces with pmacctd and wanted to clarify them with you and potentially ask for a better/alternative solution/configuration. The two patches submitted here actually illustrate the problems I encountered and the way I had to fix them. In my setup I have a "firewall entity" which forwards all traffic between two network interfaces, doing the necessary traffic firewalling/filtering, i.e. something like: +--+ <---> | InterfaceA <->(Firewall)<-> InterfaceB | <---> +--+ now I want to generate a NetFlow v9 / IPFIX data for the traffic the Firewall instance allows in both directions, which means for all egress traffic for InterfaceA and all egress traffic for InterfaceB. So I tried to use pmacctd + its nfprobe plugin for doing that (configs are listed below).I configured pmacctd with multiple pcap interfaces, specifying pcap_interfaces_map and the nfprobe plugin (configs are listed below). The first problem I encountered is that pmacctd does not configure pcap direction. I.e. I can not use pcap filter, because I do not have any specific traffic pattern to determine the egress traffic for both interfaces (i.e. all traffic is being forwarded between those two interfaces, potentially filtered with some "firewall rules" in place.) The pcap direction configuration was explicitly disabled in 81fe649917036b9ef1ed4b3ea521befcaf36496b, however even before that commit it apparently did not work, because the pcap_setdirection must be called after pcap_activate, not before it. So my first patch actually makes it possible to configure pmacctd to do pcap_setdirection by introducing a new config variable, pcap_set_direction. Now after I ended up with pcap direction working, and configured pmacctd to listen for the egress traffic on both InterfaceA and InterfaceB, I faced the second problem, which seems to be a real bug in nfprobe plugin. nfprobe flow tree does not take interface index into consideration when searching/aggregating the flow data. This means that for the case multiple pcap interfaces are being monitored and same src/dst ip/port traffic pattern is being handled over several of those interfaces, this will all land in the same FLOW entry. This leads to the issues that flows being handled by one network interface are actually reported via NetFlow (via Flow InputInt and OutputInt fields) as being handled by another network interface (held by the FLOW entry originally created for matching the given src/dst ip/port traffic pattern). This perhaps could be worked around by having different nfprobe plugin instances handling InterfaceA and InterfaceB traffic, but I want to get the NetFlow data for both interfaces simultaneously, and I might also need to dynamically add/remove such interface pairs w/o restarting the pmacctd. The latter could be easily achieved with updating the pcap_interfaces_map file and sending SIGUSR2 to pmacctd. The same would not be possible however if I had to add/remove nfprobe plugin instance configurations in pmacctd.conf So my second patch fixes the above issue by making it possible to configure nfprobe to take flow interface indexes into consideration when matching/searching for the FLOW entries in the flow cache tree. The nfprobe_per_interface_flows config is introduced for that. Would be great if someone could have a look into these two patches to see if they make sense, and/or give some hints on a better/proper way of making the similar NetFlow configuration. Here are my configs for the reference: pmacctd.conf: daemonize: false pidfile: /var/run/pmacctd.pid syslog: daemon pcap_interfaces_map: /path/to/pcap_interfaces.map pcap_ifindex: map ! newly introduced config to tell pmacctd to actually do pcap_setdirection pcap_set_direction: true promisc: true pmacctd_flow_buffer_buckets: 65536 pmacctd_flow_buffer_size: 128Mb plugins: nfprobe[filtered], print[filtered_p] plugin_pipe_size: 1048576000 plugin_buffer_size: 10485760 aggregate: src_host, dst_host, src_port, dst_port, in_iface, out_iface aggregate[filtered]: src_host, dst_host, src_port, dst_port, in_iface, out_iface aggregate[filtered_p]: src_host, dst_host, src_port, dst_port, in_iface, out_iface pmacctd_as: file refresh_maps: true pre_tag_map: /path/to/pretag.map pre_tag_filter[filtered]: -666 pre_tag_filter[filtered_p]: -666 nfprobe_source_ip: 10.11.12.23/24 nfprobe_receiver: 10.11.12.15:2055 nfprobe_version: 9 nfprobe_timeouts: maxlife=10:general=10:icmp=10:expint=10 nfprobe_maxflows: 512000 ! newly introduced config to tell nfprobe plugin to also match flow interface indexes ! when matching/searching for the FLOW entries in the flow cache tree nfprobe_per_interface_flows: true pcap_interfaces.map: ifindex=100 ifname=InterfaceA direction=out ifindex=200 ifname=InterfaceB
