Hi Paolo,
I've done a bit more testing with some packet captures and can see that the
issue almost certainly appears to be coming from the handling of the
pre_tag_map handling.
Specifically, with the original configuration, I get exported NetFlow packets
with the following (complete capture of an offending flow attached - I can send
you a complete packet capture directly if you'd like):
Source Mac Address: Vmware_b9:1c:a2 (00:50:56:b9:1c:a2)
Destination Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
Vlan Id: 2576
MPLS Label Stack Length: 101
I assume the "MPLS Label Stack Length" shown here is the tag set by the
pre_tag_filter.
Remembering that my pretag.map contains the following:
---
! Inputs
id=100 filter='ether src 00:10:db:ff:25:10'
id=101 filter='ether src 00:10:db:ff:25:11'
! Outputs
id=200 filter='ether dst 00:10:db:ff:25:10'
id=201 filter='ether dst 00:10:db:ff:25:11'
---
It's almost as if it's getting the Source and Destination MACs confused by
assigning 101 to it.
The "SourceID" sent with the packet containing the 3 flowsets is 12, which
matches this:
nfprobe_engine[inbound]: 0:12
There's a lot of traffic on this network and not all source/destinations end up
in this erroneous state. What's special about this particular flow is that the
flow of traffic is as follows:
00:50:56:b9:1c:a2 (xxxx.yyyy.67.235) VLAN 2576 -> Firewall Inside
(00:10:db:ff:25:11) Forwards (L3) Traffic -> xxxx.yyyy.64.227
(00:50:56:b9:34:bb) VLAN 2505.
The traffic for both VLAN 2576 and VLAN 2505 are both visible on the eth0
mirrored interface (but not eth1 when I run the probe on there as well).
Reviewing the data I've collected for 00:50:56:b9:34:bb, I see:
Inbound Table (as expected):
1. Data recorded for mac_dst 00:50:56:b9:34:bb, mac_src
00:10:db:ff:25:11
2. No data recorded for mac_src 00:50:56:b9:34:bb, mac_dst
00:10:db:ff:25:11
Outbound Table (as expected):
1. Data recorded for mac_src 00:50:56:b9:34:bb, mac_dst
00:10:db:ff:25:11
2. No data recorded for mac_dst 00:50:56:b9:34:bb, mac_src
00:10:db:ff:25:11
Is there anything else I can provide that may give some clue to what's
happening here?
Kind Regards,
Jonathan
Hi Jonathan,
Very good test. That should rule out the possibility i had in mind
this could be data between xx:xx:xx:ff:25:10 and xx:xx:xx:ff:25:11.
You can do another test, to progress this one step further, to see
whether the issue is on the nfprobe plugin or with nfacctd somehow
mis-classifying traffic: in parallel to nfacctd, on the collector
box, capture NetFlow traffic with Wireshark and stop it as soon as
you verify some mis-placed flow: analyzing captured data, helping
yourself also with filtering capabilities of Wireshark, you should
be able to check if the NetFlow export is allright.
If this gives a hint to solve the issue, good. Otherwise i'd have
sufficient information to reproduce the scenario in lab as a next
step.
Cheers,
Paolo
On Fri, Sep 06, 2013 at 06:04:06AM +0000, Jonathan Thorpe wrote:
> Hi Paolo,
>
> Thanks for the reply.
>
> If I do the following with the probe:
>
> 1. Run the pmacctd config with the memory plugin as follows (disabling the
> nfprobe plugin in my original probe config):
>
> plugins: memory[inbound], memory[outbound]
> imt_path[inbound]: /tmp/in.pipe
> imt_path[outbound]: /tmp/out.pipe
>
> 2. Run the following command to see if the ins/outs are mixed shows that
> traffic is collected and separated into in/out as expected.
>
> pmacct -s -p /tmp/in.pipe | awk '{ if ($3 ~ /00:10:db:ff:25:.*/) print }'
> pmacct -s -p /tmp/out.pipe | awk '{ if ($2 ~ /00:10:db:ff:25:.*/) print }'
>
> This just tests to make sure 00:10:db:ff:25:xx does not appear on the inbound
> as a destination address and 00:10:db:ff:25:xx does not appear on the
> outbound as a source address.
>
> The tag IDs are set correctly as per the pre_tag_map, so this leaves the
> nfprobe configuration in the pmacctd probe or the classification in the
> nfacctd (or its associated pre_tag_map).
>
> Thoughts?
>
> Kind Regards,
> Jonathan
>
>
>
>
>
> Hi Jonathan,
>
> Since the four tags in pre_tag_map are mutual exclusive and it
> anyway works first-match-wins fashion, remove the 'return=true'
> part from the equation. Let me know if this solves anything. If
> not, it would be beneficial to know whether it's the probe which
> is mistaken tagging (which is my bet) or the collector mistaken
> classifying then we can take it from there.
>
> Cheers,
> Paolo
>
> On Thu, Sep 05, 2013 at 04:32:37AM +0000, Jonathan Thorpe wrote:
> > Hi All,
> >
> > I have a host with two Ethernet interfaces (eth0 and eth1) which are
> > attached
> > to mirrored ports that I'd like to collect data from and store in a MySQL
> > database for collection with separate in and out tables.
> >
> > There are a large number of VLANs in the mirrored ports and I'm able to
> > work
> > out the direction of the traffic by looking at the source and destination
> > MACs.
> >
> > For example, if the src_mac is xx:xx:xx:ff:25:10 or xx:xx:xx:ff:25:11, I
> > consider the traffic as inbound to the hosts. If the dst_mac is is
> > xx:xx:xx:ff:25:10 or xx:xx:xx:ff:25:11, I consider the traffic to be
> > outbound
> > to the hosts I'm collecting for.
> >
> > My configuration uses two pre_tag_maps - one for the pmacctd process to
> > assign a tag based on the src/dst MAC. Inbound and Outbound traffic each
> > gets
> > a different nfprobe_engine ID as a result.
> >
> > The second pre_tag_map is for the nfacctd process and looks at the engine
> > ID
> > to determine whether the traffic is to be counted on inbound or outbound.
> >
> > This works for the most part, however I'm seeing a few packets in the
> > Inbound
> > table that contain a Destination MAC of or xx:xx:xx:ff:25:11 which should
> > actually be in the Outbound table.
> >
> > Clearly I have misunderstood the how the pre_tag_map and pre_tag_filter
> > works.
> >
> > I've tried this on both 0.14.3 and 1.5rc1 with the same results.
> >
> > My configuration is below.
> >
> > I'd greatly appreciate some assistance in understanding where I've gone
> > wrong.
> >
> > Kind Regards,
> > Jonathan
> >
> >
> >
> >
> >
> > Pmacctd process for eth0
> > ----
> > interface: eth0
> > plugins: nfprobe[inbound], nfprobe[outbound]
> >
> > nfprobe_version: 9
> > nfprobe_receiver: 127.0.0.1:2100
> > nfprobe_source_ip: 127.0.0.1
> >
> > nfprobe_engine[inbound]: 0:12
> > nfprobe_engine[outbound]: 0:13
> >
> > aggregate[inbound]: dst_host, src_host, src_mac, dst_mac, vlan, proto,
> > dst_port, src_port, tag
> > aggregate[outbound]: dst_host, src_host, src_mac, dst_mac, vlan, proto,
> > dst_port, src_port, tag
> >
> > pre_tag_map: /etc/pmacct/pretag.map
> > refresh_maps: true
> > pre_tag_map_entries: 3840
> >
> > pre_tag_filter[inbound]: 100-110
> > pre_tag_filter[outbound]: 200-210
> > ---
> >
> >
> > Pmacctd process for eth1 is exactly the same, except for:
> > ----
> > interface: eth1
> > nfprobe_engine[inbound]: 0:10
> > nfprobe_engine[outbound]: 0:11
> > ----
> >
> > The referenced pretag map is as follows:
> > /etc/pmacct/pretag.map
> > ----
> > ! Inputs
> > id=100 filter='ether src xx:xx:xx:ff:25:10' return=true
> > id=101 filter='ether src xx:xx:xx:ff:25:11' return=true
> >
> > ! Outputs
> > id=200 filter='ether dst xx:xx:xx:ff:25:10' return=true
> > id=201 filter='ether dst xx:xx:xx:ff:25:11' return=true
> > ----
> >
> >
> > Then there's the nfacctd configuration file:
> > ----
> > nfacctd_ip: 127.0.0.1
> > nfacctd_port: 2100
> >
> >
> > nfacctd_time_new: true
> > plugins: mysql[inbound], mysql[outbound]
> >
> > aggregate[inbound]: src_mac, dst_mac, vlan, tag
> > aggregate[outbound]: src_mac, dst_mac, vlan, tag
> >
> > pre_tag_filter[inbound]: 100-110
> > pre_tag_filter[outbound]: 200-210
> >
> > <snipped sql configuration>
> >
> > pre_tag_map: /etc/pmacct/pretag-netflow.map
> >
> > refresh_maps: true
> > pre_tag_map_entries: 3840
> > ----
> >
> > The above referenced /etc/pmacct/pretag-netflow.map is as follows:
> >
> > ----
> > id=100 ip=127.0.0.1 engine_type=0 engine_id=10
> > id=101 ip=127.0.0.1 engine_type=0 engine_id=12
> >
> > id=200 ip=127.0.0.1 engine_type=0 engine_id=11
> > id=201 ip=127.0.0.1 engine_type=0 engine_id=13
> > ----
> >
> > _______________________________________________
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
No. Time Source Destination Protocol Length
Info
24 0.003170 127.0.0.1 127.0.0.1 CFLOW 550
total: 8 (v9) records
Frame 24: 550 bytes on wire (4400 bits), 550 bytes captured (4400 bits)
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00
(00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1 (127.0.0.1), Dst: 127.0.0.1
(127.0.0.1)
User Datagram Protocol, Src Port: 46270 (46270), Dst Port: amiganetfs (2100)
Source port: 46270 (46270)
Destination port: amiganetfs (2100)
Length: 516
Checksum: 0x0018 [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
Cisco NetFlow/IPFIX
Version: 9
Count: 8
SysUptime: 2872
Timestamp: Sep 9, 2013 08:19:22.000000000 AUS Eastern Standard Time
CurrentSecs: 1378678762
FlowSequence: 1
SourceId: 12
FlowSet 1
FlowSet Id: Data Template (V9) (0)
FlowSet Length: 80
Template (Id = 1024, Count = 18)
Template Id: 1024
Field Count: 18
Field (1/18): LAST_SWITCHED
Type: LAST_SWITCHED (21)
Length: 4
Field (2/18): FIRST_SWITCHED
Type: FIRST_SWITCHED (22)
Length: 4
Field (3/18): BYTES
Type: BYTES (1)
Length: 4
Field (4/18): PKTS
Type: PKTS (2)
Length: 4
Field (5/18): IP_PROTOCOL_VERSION
Type: IP_PROTOCOL_VERSION (60)
Length: 1
Field (6/18): INPUT_SNMP
Type: INPUT_SNMP (10)
Length: 2
Field (7/18): OUTPUT_SNMP
Type: OUTPUT_SNMP (14)
Length: 2
Field (8/18): DIRECTION
Type: DIRECTION (61)
Length: 1
Field (9/18): IP_SRC_ADDR
Type: IP_SRC_ADDR (8)
Length: 4
Field (10/18): IP_DST_ADDR
Type: IP_DST_ADDR (12)
Length: 4
Field (11/18): L4_SRC_PORT
Type: L4_SRC_PORT (7)
Length: 2
Field (12/18): L4_DST_PORT
Type: L4_DST_PORT (11)
Length: 2
Field (13/18): TCP_FLAGS
Type: TCP_FLAGS (6)
Length: 1
Field (14/18): PROTOCOL
Type: PROTOCOL (4)
Length: 1
Field (15/18): SRC_MAC
Type: SRC_MAC (56)
Length: 6
Field (16/18): DESTINATION_MAC
Type: DESTINATION_MAC (80)
Length: 6
Field (17/18): SRC_VLAN
Type: SRC_VLAN (58)
Length: 2
Field (18/18): MPLS_LABEL_STACK_OCTETS
Type: MPLS_LABEL_STACK_OCTETS (201)
Length: 4
FlowSet 2
FlowSet Id: Data Template (V9) (0)
FlowSet Length: 80
Template (Id = 1025, Count = 18)
Template Id: 1025
Field Count: 18
Field (1/18): LAST_SWITCHED
Type: LAST_SWITCHED (21)
Length: 4
Field (2/18): FIRST_SWITCHED
Type: FIRST_SWITCHED (22)
Length: 4
Field (3/18): BYTES
Type: BYTES (1)
Length: 4
Field (4/18): PKTS
Type: PKTS (2)
Length: 4
Field (5/18): IP_PROTOCOL_VERSION
Type: IP_PROTOCOL_VERSION (60)
Length: 1
Field (6/18): INPUT_SNMP
Type: INPUT_SNMP (10)
Length: 2
Field (7/18): OUTPUT_SNMP
Type: OUTPUT_SNMP (14)
Length: 2
Field (8/18): DIRECTION
Type: DIRECTION (61)
Length: 1
Field (9/18): IP_SRC_ADDR
Type: IP_SRC_ADDR (8)
Length: 4
Field (10/18): IP_DST_ADDR
Type: IP_DST_ADDR (12)
Length: 4
Field (11/18): L4_SRC_PORT
Type: L4_SRC_PORT (7)
Length: 2
Field (12/18): L4_DST_PORT
Type: L4_DST_PORT (11)
Length: 2
Field (13/18): TCP_FLAGS
Type: TCP_FLAGS (6)
Length: 1
Field (14/18): PROTOCOL
Type: PROTOCOL (4)
Length: 1
Field (15/18): SOURCE_MAC
Type: SOURCE_MAC (81)
Length: 6
Field (16/18): DST_MAC
Type: DST_MAC (57)
Length: 6
Field (17/18): DST_VLAN
Type: DST_VLAN (59)
Length: 2
Field (18/18): MPLS_LABEL_STACK_OCTETS
Type: MPLS_LABEL_STACK_OCTETS (201)
Length: 4
FlowSet 3
FlowSet Id: (Data) (1024)
FlowSet Length: 328
Flow 1
[Duration: 0.000000000 seconds]
StartTime: 0.268000000 seconds
EndTime: 0.268000000 seconds
Octets: 40
Packets: 1
IPVersion: 04
InputInt: 0
OutputInt: 0
Direction: Ingress (0)
SrcAddr: xxxx.zzzz.54.216 (xxxx.zzzz.54.216)
DstAddr: xxx.yyyy.66.166 (xxx.yyyy.66.166)
SrcPort: 52896
DstPort: 80
TCP Flags: 0x04
Protocol: 6
Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
Destination Mac Address: Vmware_b9:00:78 (00:50:56:b9:00:78)
Vlan Id: 2552
MPLS Label Stack Length: 101
Flow 2
[Duration: 0.428000000 seconds]
StartTime: 0.557000000 seconds
EndTime: 0.985000000 seconds
Octets: 80
Packets: 2
IPVersion: 04
InputInt: 0
OutputInt: 0
Direction: Ingress (0)
SrcAddr: 165.69.90.8 (165.69.90.8)
DstAddr: xxx.yyyy.64.84 (xxx.yyyy.64.84)
SrcPort: 30783
DstPort: 443
TCP Flags: 0x14
Protocol: 6
Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
Destination Mac Address: Supermic_c6:3e:ba (00:30:48:c6:3e:ba)
Vlan Id: 2502
MPLS Label Stack Length: 101
Flow 3
[Duration: 0.003000000 seconds]
StartTime: 0.656000000 seconds
EndTime: 0.659000000 seconds
Octets: 2623
Packets: 4
IPVersion: 04
InputInt: 0
OutputInt: 0
Direction: Ingress (0)
SrcAddr: xxxx.zzzz.181.115 (xxxx.zzzz.181.115)
DstAddr: xxx.yyyy.64.227 (xxx.yyyy.64.227)
SrcPort: 443
DstPort: 40489
TCP Flags: 0x1e
Protocol: 6
Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
Destination Mac Address: Vmware_b9:34:bb (00:50:56:b9:34:bb)
Vlan Id: 2505
MPLS Label Stack Length: 101
Flow 4
[Duration: 0.001000000 seconds]
StartTime: 0.679000000 seconds
EndTime: 0.680000000 seconds
Octets: 112
Packets: 2
IPVersion: 04
InputInt: 0
OutputInt: 0
Direction: Ingress (0)
SrcAddr: xxx.yyyy.64.227 (xxx.yyyy.64.227)
DstAddr: xxx.yyyy.67.235 (xxx.yyyy.67.235)
SrcPort: 50899
DstPort: 995
TCP Flags: 0x16
Protocol: 6
Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
Destination Mac Address: Vmware_b9:1c:a2 (00:50:56:b9:1c:a2)
Vlan Id: 2576
MPLS Label Stack Length: 101
Flow 5
[Duration: 0.001000000 seconds]
StartTime: 0.679000000 seconds
EndTime: 0.680000000 seconds
Octets: 60
Packets: 1
IPVersion: 04
InputInt: 0
OutputInt: 0
Direction: Ingress (0)
SrcAddr: xxx.yyyy.67.235 (xxx.yyyy.67.235)
DstAddr: xxx.yyyy.64.227 (xxx.yyyy.64.227)
SrcPort: 995
DstPort: 50899
TCP Flags: 0x12
Protocol: 6
Source Mac Address: Vmware_b9:1c:a2 (00:50:56:b9:1c:a2)
Destination Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
Vlan Id: 2576
MPLS Label Stack Length: 101
Flow 6
[Duration: 0.001000000 seconds]
StartTime: 0.984000000 seconds
EndTime: 0.985000000 seconds
Octets: 80
Packets: 2
IPVersion: 04
InputInt: 0
OutputInt: 0
Direction: Ingress (0)
SrcAddr: zzzz.yyyy.90.8 (zzzz.yyyy.90.8)
DstAddr: xxx.yyyy.64.84 (xxx.yyyy.64.84)
SrcPort: 30834
DstPort: 443
TCP Flags: 0x14
Protocol: 6
Source Mac Address: Netscreen_ff:25:11 (00:10:db:ff:25:11)
Destination Mac Address: Supermic_c6:3e:ba (00:30:48:c6:3e:ba)
Vlan Id: 2502
MPLS Label Stack Length: 101
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
