Re: netflow srcip and dstip reversed for redirected traffic
On Sat, 31 May 2014 20:01:25 +0200 Sebastian Benoit wrote: > The simple answer: It's complicated. > > The complicated answer: the pf state is used to keep track of both > directions of the traffic flow. When the state times out, _two_ flows > are created, one for each direction of traffic, you can see this in > copy_flow_ipfix_4_data() in /usr/src/sys/net/if_pflow.c. > > For NAT/RDR its a bit more complicated, so what you are seeing might > be 'normal' or a problem. > > nfdump should be able to show you both directions of this traffic. > Please check what in and out interface is recorded for each flow, ie > grep for 178.148.77.73 but dont restrict on the interface. > > Also, please show a dmesg - we need to know what version you are > running. > > /Benno > I have enabled pflow for outbound traffic on $int_if and $ext_if first, and it appears that in this setup no redirected traffic is recorded by nfdump, either entering $ext_if and leaving $int_if on arrival, or entering $int_if and leaving $ext_if on return. Other kinds of traffic appear to be recorded correctly by pflow, including NAT traffic. Next, I enabled pflow for one additional inbound redirected rule: pass in on $if_ext inet proto tcp from any to $pub_srv port 1002 \ rdr-to $priv_srv keep state (pflow) In this setup flows appear to be recorded by nfdump fine on $int_if, both leaving it on arrival and entering it on return. Direction is correct. % nfdump -R 2014 -s srcip/bytes 'out if 5 and port 1002' Src IP AddrFlows(%) Packets(%) Bytes(%) 212.200.65.243 3678(34.9)24554(36.0)2.1 M(35.2) 212.200.65.244 2393(22.7)15331(22.5)1.4 M(23.3) 212.200.65.241 2457(23.3)15488(22.7)1.3 M(22.5) 212.200.65.242 2025(19.2)12765(18.7)1.1 M(19.0) % nfdump -R 2014 -s dstip/bytes 'in if 5 and port 1002' Dst IP AddrFlows(%) Packets(%) Bytes(%) 212.200.65.243 3678(34.9)20699(34.9)1.0 M(36.3) 212.200.65.241 2457(23.3)13572(22.9) 638520(22.5) 212.200.65.244 2393(22.7)13590(22.9) 619420(21.9) 212.200.65.242 2025(19.2)11496(19.4) 547616(19.3) However, on external interface the direction appears to be reversed (notice I need to request '$ext_if outbound srcip' in order to get '$ext_if outbound dstip': % nfdump -R 2014 -s srcip/bytes 'out if 4 and port 1002' Src IP AddrFlows(%) Packets(%) Bytes(%) 212.200.65.243 4051(35.0)26862(36.4)2.3 M(35.7) 212.200.65.244 2654(23.0)16771(22.7)1.5 M(23.4) 212.200.65.241 2683(23.2)16731(22.7)1.4 M(22.4) 212.200.65.242 2175(18.8)13475(18.2)1.2 M(18.5) Also I need to request '$ext_if inbound dstip' in order to get '$ext_if inbound srcip': % nfdump -R 2014 -s dstip/bytes 'in if 4 and port 1002' Dst IP AddrFlows(%) Packets(%) Bytes(%) 212.200.65.243 4051(35.0)22767(35.0)1.1 M(36.5) 212.200.65.241 2683(23.2)14756(22.7) 692652(22.4) 212.200.65.244 2654(23.0)15024(23.1) 683824(22.1) 212.200.65.242 2175(18.8)12409(19.1) 586820(19.0) I am using quite recent snapshot: OpenBSD 5.5-current (GENERIC.MP) #150: Mon May 26 11:50:31 MDT 2014 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2128887808 (2030MB) avail mem = 2063499264 (1967MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xee000 (69 entries) bios0: vendor HP version "P58" date 05/02/2011 bios0: HP ProLiant DL360 G5 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP SPCR MCFG HPET SPMI ERST APIC BERT HEST SSDT acpi0: wakeup devices PCI0(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimcfg0 at acpi0 addr 0xe000, bus 0-255 acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.38 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF cpu0: 6MB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 333MHz cpu0: mwait min=64, max=64, C-substates=0.2.2.2.0, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2000.08 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF cpu1: 6MB 64b/line 16-way L2 cache cpu1: smt 0, core 2, package 0 cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.09 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
Re: netflow srcip and dstip reversed for redirected traffic
Marko Cupa??(marko.cu...@mimar.rs) on 2014.05.31 13:03:18 +0200: > Hi, > > I'm trying to understand and measure traffic on relatively large and > complicated pf firewall, and for this purpose I am exporting netflow > data with pflow to nfsen/nfdump. > > For the time being, I have set pflow on external interface in outbound > direction: > > pass out on $if_ext inet all keep state (pflow) > > On collector (nfsen), I want to see interface numbers so i can create > interface filter: > > % nfdump -R 2014 -s if/bytes > Top 10 In/Out If ordered by bytes: > If Flows(%) Packets(%) Bytes(%) pps bps bpp > 519396(100.0) 300683(100.0) 186.7 M(100.0)316984 620 > 719109(98.5)299769(99.7)186.6 M(100.0)316976 622 > 0 287( 1.5) 914( 0.3)83170( 0.0) 0 33090 > > Another mailing list member told me I can find about interface numbers > with snmpwalk: > > % snmpwalk -v2c -c community -On IP.ADD.RE.SS > .1.3.6.1.2.1.2.2.1.2.5 = STRING: bnx1 > .1.3.6.1.2.1.2.2.1.2.7 = STRING: carp2 > > Ok, now I know interface 5 is bnx1 ($if_ext), and I want to know what > comes in: > > % nfdump -R 2014 -s dstip/bytes 'in if 5' > Top 10 Dst IP Addr ordered by bytes: > Dst IP AddrFlows(%) Packets(%) Bytes(%) > 10.20.0.1510754(62.9) 323834(52.9) 324.9 M(63.7) > 10.20.4.99 462( 2.7)10496( 1.7)9.4 M( 1.8) > 178.148.77.734( 0.0) 6681( 1.1)7.7 M( 1.5) > > First two addresses really are on my internal network, and I know first > one is return web traffic to my proxy, and the second one return web > traffic to another internal host. > > But the last address is not on my network. Let's see records for this > address: > > nfdump -R 2014 -n 10 -s record/bytes 'in if 5' | grep 178.148.77.73 > TCP 193.53.106.35:443 -> 178.148.77.73:49193 56067.6 M > TCP 193.53.106.35:443 -> 178.148.77.73:49191 31395342 > TCP 193.53.106.35:443 -> 178.148.77.73:49192 40418674 > TCP 193.53.106.35:443 -> 178.148.77.73:49190 35816798 > > Ok, these are redirected incoming requests to HTTPS server on my > internal network: > > pass in on $if_ext inet proto tcp from any to $pub_web port { 80 443 } \ > rdr-to $priv_web keep state > > But source and destination IP adresses are reversed! > > Here's what pf's state table shows: > $ sudo pfctl -ss | grep 178.148.77.73 > all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49377 > all tcp 178.148.77.73:49377 -> 10.20.0.36:443 > all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49378 > all tcp 178.148.77.73:49378 -> 10.20.0.36:443 > all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49379 > all tcp 178.148.77.73:49379 -> 10.20.0.36:443 > all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49380 > all tcp 178.148.77.73:49380 -> 10.20.0.36:443 > > How could this be corrected? Am I configuring pf incorrectly? Or is > there a problem with how pflow exports data? Or is pfdump parsing the > data incorrectly? The simple answer: It's complicated. The complicated answer: the pf state is used to keep track of both directions of the traffic flow. When the state times out, _two_ flows are created, one for each direction of traffic, you can see this in copy_flow_ipfix_4_data() in /usr/src/sys/net/if_pflow.c. For NAT/RDR its a bit more complicated, so what you are seeing might be 'normal' or a problem. nfdump should be able to show you both directions of this traffic. Please check what in and out interface is recorded for each flow, ie grep for 178.148.77.73 but dont restrict on the interface. Also, please show a dmesg - we need to know what version you are running. /Benno
netflow srcip and dstip reversed for redirected traffic
Hi, I'm trying to understand and measure traffic on relatively large and complicated pf firewall, and for this purpose I am exporting netflow data with pflow to nfsen/nfdump. For the time being, I have set pflow on external interface in outbound direction: pass out on $if_ext inet all keep state (pflow) On collector (nfsen), I want to see interface numbers so i can create interface filter: % nfdump -R 2014 -s if/bytes Top 10 In/Out If ordered by bytes: If Flows(%) Packets(%) Bytes(%) pps bps bpp 519396(100.0) 300683(100.0) 186.7 M(100.0)316984 620 719109(98.5)299769(99.7)186.6 M(100.0)316976 622 0 287( 1.5) 914( 0.3)83170( 0.0) 0 33090 Another mailing list member told me I can find about interface numbers with snmpwalk: % snmpwalk -v2c -c community -On IP.ADD.RE.SS .1.3.6.1.2.1.2.2.1.2.5 = STRING: bnx1 .1.3.6.1.2.1.2.2.1.2.7 = STRING: carp2 Ok, now I know interface 5 is bnx1 ($if_ext), and I want to know what comes in: % nfdump -R 2014 -s dstip/bytes 'in if 5' Top 10 Dst IP Addr ordered by bytes: Dst IP AddrFlows(%) Packets(%) Bytes(%) 10.20.0.1510754(62.9) 323834(52.9) 324.9 M(63.7) 10.20.4.99 462( 2.7)10496( 1.7)9.4 M( 1.8) 178.148.77.734( 0.0) 6681( 1.1)7.7 M( 1.5) First two addresses really are on my internal network, and I know first one is return web traffic to my proxy, and the second one return web traffic to another internal host. But the last address is not on my network. Let's see records for this address: nfdump -R 2014 -n 10 -s record/bytes 'in if 5' | grep 178.148.77.73 TCP 193.53.106.35:443 -> 178.148.77.73:49193 56067.6 M TCP 193.53.106.35:443 -> 178.148.77.73:49191 31395342 TCP 193.53.106.35:443 -> 178.148.77.73:49192 40418674 TCP 193.53.106.35:443 -> 178.148.77.73:49190 35816798 Ok, these are redirected incoming requests to HTTPS server on my internal network: pass in on $if_ext inet proto tcp from any to $pub_web port { 80 443 } \ rdr-to $priv_web keep state But source and destination IP adresses are reversed! Here's what pf's state table shows: $ sudo pfctl -ss | grep 178.148.77.73 all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49377 all tcp 178.148.77.73:49377 -> 10.20.0.36:443 all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49378 all tcp 178.148.77.73:49378 -> 10.20.0.36:443 all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49379 all tcp 178.148.77.73:49379 -> 10.20.0.36:443 all tcp 10.20.0.36:443 (193.53.106.35:443) <- 178.148.77.73:49380 all tcp 178.148.77.73:49380 -> 10.20.0.36:443 How could this be corrected? Am I configuring pf incorrectly? Or is there a problem with how pflow exports data? Or is pfdump parsing the data incorrectly? Thank you in advance, -- Marko Cupać