Hi Jonathan,
This is actually great troubleshooting, thanks very much as it's
going to help a lot solving the issue - much appreciated. Can you
please send me privately a trace of a few NetFlow packets containing
offending flows (also the NetFlow v9 template must be part of the
trace)?
I will use it to replay the issue in lab as that "MPLS Label
Stack Length: 101" on one hand does not look right, while on
the other should not impact the collector. When following-up,
can you also indicate urgency?
Cheers,
Paolo
On Mon, Sep 09, 2013 at 12:20:49AM +0000, Jonathan Thorpe wrote:
> Hi Paolo,
>
> I've done a bit more testing with some packet captures and can see that the
> issue almost certainly appears to be coming from the handling of the
> pre_tag_map handling.
>
> Specifically, with the original configuration, I get exported NetFlow packets
> with the following (complete capture of an offending flow attached - I can
> send you a complete packet capture directly if you'd like):
>
> Source Mac Address: Vmware_b9:1c:a2 (00:50:56:b9:1c:a2)
> Destination Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
> Vlan Id: 2576
> MPLS Label Stack Length: 101
>
> I assume the "MPLS Label Stack Length" shown here is the tag set by the
> pre_tag_filter.
>
> Remembering that my pretag.map contains the following:
> ---
> ! Inputs
> id=100 filter='ether src 00:10:db:ff:25:10'
> id=101 filter='ether src 00:10:db:ff:25:11'
>
> ! Outputs
> id=200 filter='ether dst 00:10:db:ff:25:10'
> id=201 filter='ether dst 00:10:db:ff:25:11'
> ---
>
> It's almost as if it's getting the Source and Destination MACs confused by
> assigning 101 to it.
>
> The "SourceID" sent with the packet containing the 3 flowsets is 12, which
> matches this:
> nfprobe_engine[inbound]: 0:12
>
> There's a lot of traffic on this network and not all source/destinations end
> up in this erroneous state. What's special about this particular flow is that
> the flow of traffic is as follows:
>
> 00:50:56:b9:1c:a2 (xxxx.yyyy.67.235) VLAN 2576 -> Firewall Inside
> (00:10:db:ff:25:11) Forwards (L3) Traffic -> xxxx.yyyy.64.227
> (00:50:56:b9:34:bb) VLAN 2505.
>
> The traffic for both VLAN 2576 and VLAN 2505 are both visible on the eth0
> mirrored interface (but not eth1 when I run the probe on there as well).
>
> Reviewing the data I've collected for 00:50:56:b9:34:bb, I see:
>
> Inbound Table (as expected):
> 1. Data recorded for mac_dst 00:50:56:b9:34:bb, mac_src
> 00:10:db:ff:25:11
> 2. No data recorded for mac_src 00:50:56:b9:34:bb, mac_dst
> 00:10:db:ff:25:11
>
> Outbound Table (as expected):
> 1. Data recorded for mac_src 00:50:56:b9:34:bb, mac_dst
> 00:10:db:ff:25:11
> 2. No data recorded for mac_dst 00:50:56:b9:34:bb, mac_src
> 00:10:db:ff:25:11
>
>
> Is there anything else I can provide that may give some clue to what's
> happening here?
>
> Kind Regards,
> Jonathan
>
>
>
>
>
> Hi Jonathan,
>
> Very good test. That should rule out the possibility i had in mind
> this could be data between xx:xx:xx:ff:25:10 and xx:xx:xx:ff:25:11.
>
> You can do another test, to progress this one step further, to see
> whether the issue is on the nfprobe plugin or with nfacctd somehow
> mis-classifying traffic: in parallel to nfacctd, on the collector
> box, capture NetFlow traffic with Wireshark and stop it as soon as
> you verify some mis-placed flow: analyzing captured data, helping
> yourself also with filtering capabilities of Wireshark, you should
> be able to check if the NetFlow export is allright.
>
> If this gives a hint to solve the issue, good. Otherwise i'd have
> sufficient information to reproduce the scenario in lab as a next
> step.
>
> Cheers,
> Paolo
>
> On Fri, Sep 06, 2013 at 06:04:06AM +0000, Jonathan Thorpe wrote:
> > Hi Paolo,
> >
> > Thanks for the reply.
> >
> > If I do the following with the probe:
> >
> > 1. Run the pmacctd config with the memory plugin as follows (disabling the
> > nfprobe plugin in my original probe config):
> >
> > plugins: memory[inbound], memory[outbound]
> > imt_path[inbound]: /tmp/in.pipe
> > imt_path[outbound]: /tmp/out.pipe
> >
> > 2. Run the following command to see if the ins/outs are mixed shows that
> > traffic is collected and separated into in/out as expected.
> >
> > pmacct -s -p /tmp/in.pipe | awk '{ if ($3 ~ /00:10:db:ff:25:.*/) print }'
> > pmacct -s -p /tmp/out.pipe | awk '{ if ($2 ~ /00:10:db:ff:25:.*/) print }'
> >
> > This just tests to make sure 00:10:db:ff:25:xx does not appear on the
> > inbound
> > as a destination address and 00:10:db:ff:25:xx does not appear on the
> > outbound as a source address.
> >
> > The tag IDs are set correctly as per the pre_tag_map, so this leaves the
> > nfprobe configuration in the pmacctd probe or the classification in the
> > nfacctd (or its associated pre_tag_map).
> >
> > Thoughts?
> >
> > Kind Regards,
> > Jonathan
> >
> >
> >
> >
> >
> > Hi Jonathan,
> >
> > Since the four tags in pre_tag_map are mutual exclusive and it
> > anyway works first-match-wins fashion, remove the 'return=true'
> > part from the equation. Let me know if this solves anything. If
> > not, it would be beneficial to know whether it's the probe which
> > is mistaken tagging (which is my bet) or the collector mistaken
> > classifying then we can take it from there.
> >
> > Cheers,
> > Paolo
> >
> > On Thu, Sep 05, 2013 at 04:32:37AM +0000, Jonathan Thorpe wrote:
> > > Hi All,
> > >
> > > I have a host with two Ethernet interfaces (eth0 and eth1) which are
> > > attached
> > > to mirrored ports that I'd like to collect data from and store in a MySQL
> > > database for collection with separate in and out tables.
> > >
> > > There are a large number of VLANs in the mirrored ports and I'm able to
> > > work
> > > out the direction of the traffic by looking at the source and destination
> > > MACs.
> > >
> > > For example, if the src_mac is xx:xx:xx:ff:25:10 or xx:xx:xx:ff:25:11, I
> > > consider the traffic as inbound to the hosts. If the dst_mac is is
> > > xx:xx:xx:ff:25:10 or xx:xx:xx:ff:25:11, I consider the traffic to be
> > > outbound
> > > to the hosts I'm collecting for.
> > >
> > > My configuration uses two pre_tag_maps - one for the pmacctd process to
> > > assign a tag based on the src/dst MAC. Inbound and Outbound traffic each
> > > gets
> > > a different nfprobe_engine ID as a result.
> > >
> > > The second pre_tag_map is for the nfacctd process and looks at the
> > > engine
> > > ID
> > > to determine whether the traffic is to be counted on inbound or outbound.
> > >
> > > This works for the most part, however I'm seeing a few packets in the
> > > Inbound
> > > table that contain a Destination MAC of or xx:xx:xx:ff:25:11 which should
> > > actually be in the Outbound table.
> > >
> > > Clearly I have misunderstood the how the pre_tag_map and pre_tag_filter
> > > works.
> > >
> > > I've tried this on both 0.14.3 and 1.5rc1 with the same results.
> > >
> > > My configuration is below.
> > >
> > > I'd greatly appreciate some assistance in understanding where I've gone
> > > wrong.
> > >
> > > Kind Regards,
> > > Jonathan
> > >
> > >
> > >
> > >
> > >
> > > Pmacctd process for eth0
> > > ----
> > > interface: eth0
> > > plugins: nfprobe[inbound], nfprobe[outbound]
> > >
> > > nfprobe_version: 9
> > > nfprobe_receiver: 127.0.0.1:2100
> > > nfprobe_source_ip: 127.0.0.1
> > >
> > > nfprobe_engine[inbound]: 0:12
> > > nfprobe_engine[outbound]: 0:13
> > >
> > > aggregate[inbound]: dst_host, src_host, src_mac, dst_mac, vlan, proto,
> > > dst_port, src_port, tag
> > > aggregate[outbound]: dst_host, src_host, src_mac, dst_mac, vlan, proto,
> > > dst_port, src_port, tag
> > >
> > > pre_tag_map: /etc/pmacct/pretag.map
> > > refresh_maps: true
> > > pre_tag_map_entries: 3840
> > >
> > > pre_tag_filter[inbound]: 100-110
> > > pre_tag_filter[outbound]: 200-210
> > > ---
> > >
> > >
> > > Pmacctd process for eth1 is exactly the same, except for:
> > > ----
> > > interface: eth1
> > > nfprobe_engine[inbound]: 0:10
> > > nfprobe_engine[outbound]: 0:11
> > > ----
> > >
> > > The referenced pretag map is as follows:
> > > /etc/pmacct/pretag.map
> > > ----
> > > ! Inputs
> > > id=100 filter='ether src xx:xx:xx:ff:25:10' return=true
> > > id=101 filter='ether src xx:xx:xx:ff:25:11' return=true
> > >
> > > ! Outputs
> > > id=200 filter='ether dst xx:xx:xx:ff:25:10' return=true
> > > id=201 filter='ether dst xx:xx:xx:ff:25:11' return=true
> > > ----
> > >
> > >
> > > Then there's the nfacctd configuration file:
> > > ----
> > > nfacctd_ip: 127.0.0.1
> > > nfacctd_port: 2100
> > >
> > >
> > > nfacctd_time_new: true
> > > plugins: mysql[inbound], mysql[outbound]
> > >
> > > aggregate[inbound]: src_mac, dst_mac, vlan, tag
> > > aggregate[outbound]: src_mac, dst_mac, vlan, tag
> > >
> > > pre_tag_filter[inbound]: 100-110
> > > pre_tag_filter[outbound]: 200-210
> > >
> > > <snipped sql configuration>
> > >
> > > pre_tag_map: /etc/pmacct/pretag-netflow.map
> > >
> > > refresh_maps: true
> > > pre_tag_map_entries: 3840
> > > ----
> > >
> > > The above referenced /etc/pmacct/pretag-netflow.map is as follows:
> > >
> > > ----
> > > id=100 ip=127.0.0.1 engine_type=0 engine_id=10
> > > id=101 ip=127.0.0.1 engine_type=0 engine_id=12
> > >
> > > id=200 ip=127.0.0.1 engine_type=0 engine_id=11
> > > id=201 ip=127.0.0.1 engine_type=0 engine_id=13
> > > ----
> > >
> > > _______________________________________________
> > > pmacct-discussion mailing list
> > > http://www.pmacct.net/#mailinglists
> >
> > _______________________________________________
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> No. Time Source Destination Protocol
> Length Info
> 24 0.003170 127.0.0.1 127.0.0.1 CFLOW 550
> total: 8 (v9) records
>
> Frame 24: 550 bytes on wire (4400 bits), 550 bytes captured (4400 bits)
> Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst:
> 00:00:00_00:00:00 (00:00:00:00:00:00)
> Internet Protocol Version 4, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1
> (127.0.0.1)
> User Datagram Protocol, Src Port: 46270 (46270), Dst Port: amiganetfs (2100)
> Source port: 46270 (46270)
> Destination port: amiganetfs (2100)
> Length: 516
> Checksum: 0x0018 [validation disabled]
> [Good Checksum: False]
> [Bad Checksum: False]
> Cisco NetFlow/IPFIX
> Version: 9
> Count: 8
> SysUptime: 2872
> Timestamp: Sep 9, 2013 08:19:22.000000000 AUS Eastern Standard Time
> CurrentSecs: 1378678762
> FlowSequence: 1
> SourceId: 12
> FlowSet 1
> FlowSet Id: Data Template (V9) (0)
> FlowSet Length: 80
> Template (Id = 1024, Count = 18)
> Template Id: 1024
> Field Count: 18
> Field (1/18): LAST_SWITCHED
> Type: LAST_SWITCHED (21)
> Length: 4
> Field (2/18): FIRST_SWITCHED
> Type: FIRST_SWITCHED (22)
> Length: 4
> Field (3/18): BYTES
> Type: BYTES (1)
> Length: 4
> Field (4/18): PKTS
> Type: PKTS (2)
> Length: 4
> Field (5/18): IP_PROTOCOL_VERSION
> Type: IP_PROTOCOL_VERSION (60)
> Length: 1
> Field (6/18): INPUT_SNMP
> Type: INPUT_SNMP (10)
> Length: 2
> Field (7/18): OUTPUT_SNMP
> Type: OUTPUT_SNMP (14)
> Length: 2
> Field (8/18): DIRECTION
> Type: DIRECTION (61)
> Length: 1
> Field (9/18): IP_SRC_ADDR
> Type: IP_SRC_ADDR (8)
> Length: 4
> Field (10/18): IP_DST_ADDR
> Type: IP_DST_ADDR (12)
> Length: 4
> Field (11/18): L4_SRC_PORT
> Type: L4_SRC_PORT (7)
> Length: 2
> Field (12/18): L4_DST_PORT
> Type: L4_DST_PORT (11)
> Length: 2
> Field (13/18): TCP_FLAGS
> Type: TCP_FLAGS (6)
> Length: 1
> Field (14/18): PROTOCOL
> Type: PROTOCOL (4)
> Length: 1
> Field (15/18): SRC_MAC
> Type: SRC_MAC (56)
> Length: 6
> Field (16/18): DESTINATION_MAC
> Type: DESTINATION_MAC (80)
> Length: 6
> Field (17/18): SRC_VLAN
> Type: SRC_VLAN (58)
> Length: 2
> Field (18/18): MPLS_LABEL_STACK_OCTETS
> Type: MPLS_LABEL_STACK_OCTETS (201)
> Length: 4
> FlowSet 2
> FlowSet Id: Data Template (V9) (0)
> FlowSet Length: 80
> Template (Id = 1025, Count = 18)
> Template Id: 1025
> Field Count: 18
> Field (1/18): LAST_SWITCHED
> Type: LAST_SWITCHED (21)
> Length: 4
> Field (2/18): FIRST_SWITCHED
> Type: FIRST_SWITCHED (22)
> Length: 4
> Field (3/18): BYTES
> Type: BYTES (1)
> Length: 4
> Field (4/18): PKTS
> Type: PKTS (2)
> Length: 4
> Field (5/18): IP_PROTOCOL_VERSION
> Type: IP_PROTOCOL_VERSION (60)
> Length: 1
> Field (6/18): INPUT_SNMP
> Type: INPUT_SNMP (10)
> Length: 2
> Field (7/18): OUTPUT_SNMP
> Type: OUTPUT_SNMP (14)
> Length: 2
> Field (8/18): DIRECTION
> Type: DIRECTION (61)
> Length: 1
> Field (9/18): IP_SRC_ADDR
> Type: IP_SRC_ADDR (8)
> Length: 4
> Field (10/18): IP_DST_ADDR
> Type: IP_DST_ADDR (12)
> Length: 4
> Field (11/18): L4_SRC_PORT
> Type: L4_SRC_PORT (7)
> Length: 2
> Field (12/18): L4_DST_PORT
> Type: L4_DST_PORT (11)
> Length: 2
> Field (13/18): TCP_FLAGS
> Type: TCP_FLAGS (6)
> Length: 1
> Field (14/18): PROTOCOL
> Type: PROTOCOL (4)
> Length: 1
> Field (15/18): SOURCE_MAC
> Type: SOURCE_MAC (81)
> Length: 6
> Field (16/18): DST_MAC
> Type: DST_MAC (57)
> Length: 6
> Field (17/18): DST_VLAN
> Type: DST_VLAN (59)
> Length: 2
> Field (18/18): MPLS_LABEL_STACK_OCTETS
> Type: MPLS_LABEL_STACK_OCTETS (201)
> Length: 4
> FlowSet 3
> FlowSet Id: (Data) (1024)
> FlowSet Length: 328
> Flow 1
> [Duration: 0.000000000 seconds]
> StartTime: 0.268000000 seconds
> EndTime: 0.268000000 seconds
> Octets: 40
> Packets: 1
> IPVersion: 04
> InputInt: 0
> OutputInt: 0
> Direction: Ingress (0)
> SrcAddr: xxxx.zzzz.54.216 (xxxx.zzzz.54.216)
> DstAddr: xxx.yyyy.66.166 (xxx.yyyy.66.166)
> SrcPort: 52896
> DstPort: 80
> TCP Flags: 0x04
> Protocol: 6
> Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
> Destination Mac Address: Vmware_b9:00:78 (00:50:56:b9:00:78)
> Vlan Id: 2552
> MPLS Label Stack Length: 101
> Flow 2
> [Duration: 0.428000000 seconds]
> StartTime: 0.557000000 seconds
> EndTime: 0.985000000 seconds
> Octets: 80
> Packets: 2
> IPVersion: 04
> InputInt: 0
> OutputInt: 0
> Direction: Ingress (0)
> SrcAddr: 165.69.90.8 (165.69.90.8)
> DstAddr: xxx.yyyy.64.84 (xxx.yyyy.64.84)
> SrcPort: 30783
> DstPort: 443
> TCP Flags: 0x14
> Protocol: 6
> Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
> Destination Mac Address: Supermic_c6:3e:ba (00:30:48:c6:3e:ba)
> Vlan Id: 2502
> MPLS Label Stack Length: 101
> Flow 3
> [Duration: 0.003000000 seconds]
> StartTime: 0.656000000 seconds
> EndTime: 0.659000000 seconds
> Octets: 2623
> Packets: 4
> IPVersion: 04
> InputInt: 0
> OutputInt: 0
> Direction: Ingress (0)
> SrcAddr: xxxx.zzzz.181.115 (xxxx.zzzz.181.115)
> DstAddr: xxx.yyyy.64.227 (xxx.yyyy.64.227)
> SrcPort: 443
> DstPort: 40489
> TCP Flags: 0x1e
> Protocol: 6
> Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
> Destination Mac Address: Vmware_b9:34:bb (00:50:56:b9:34:bb)
> Vlan Id: 2505
> MPLS Label Stack Length: 101
> Flow 4
> [Duration: 0.001000000 seconds]
> StartTime: 0.679000000 seconds
> EndTime: 0.680000000 seconds
> Octets: 112
> Packets: 2
> IPVersion: 04
> InputInt: 0
> OutputInt: 0
> Direction: Ingress (0)
> SrcAddr: xxx.yyyy.64.227 (xxx.yyyy.64.227)
> DstAddr: xxx.yyyy.67.235 (xxx.yyyy.67.235)
> SrcPort: 50899
> DstPort: 995
> TCP Flags: 0x16
> Protocol: 6
> Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
> Destination Mac Address: Vmware_b9:1c:a2 (00:50:56:b9:1c:a2)
> Vlan Id: 2576
> MPLS Label Stack Length: 101
> Flow 5
> [Duration: 0.001000000 seconds]
> StartTime: 0.679000000 seconds
> EndTime: 0.680000000 seconds
> Octets: 60
> Packets: 1
> IPVersion: 04
> InputInt: 0
> OutputInt: 0
> Direction: Ingress (0)
> SrcAddr: xxx.yyyy.67.235 (xxx.yyyy.67.235)
> DstAddr: xxx.yyyy.64.227 (xxx.yyyy.64.227)
> SrcPort: 995
> DstPort: 50899
> TCP Flags: 0x12
> Protocol: 6
> Source Mac Address: Vmware_b9:1c:a2 (00:50:56:b9:1c:a2)
> Destination Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
> Vlan Id: 2576
> MPLS Label Stack Length: 101
> Flow 6
> [Duration: 0.001000000 seconds]
> StartTime: 0.984000000 seconds
> EndTime: 0.985000000 seconds
> Octets: 80
> Packets: 2
> IPVersion: 04
> InputInt: 0
> OutputInt: 0
> Direction: Ingress (0)
> SrcAddr: zzzz.yyyy.90.8 (zzzz.yyyy.90.8)
> DstAddr: xxx.yyyy.64.84 (xxx.yyyy.64.84)
> SrcPort: 30834
> DstPort: 443
> TCP Flags: 0x14
> Protocol: 6
> Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
> Destination Mac Address: Supermic_c6:3e:ba (00:30:48:c6:3e:ba)
> Vlan Id: 2502
> MPLS Label Stack Length: 101
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
