Re: netflow srcip and dstip reversed for redirected traffic
On Sat, 31 May 2014 20:01:25 +0200 Sebastian Benoit benoit-li...@fb12.de wrote: The simple answer: It's complicated. The complicated answer: the pf state is used to keep track of both directions of the traffic flow. When the state times out, _two_ flows are created, one for each direction of traffic, you can see this in copy_flow_ipfix_4_data() in /usr/src/sys/net/if_pflow.c. For NAT/RDR its a bit more complicated, so what you are seeing might be 'normal' or a problem. nfdump should be able to show you both directions of this traffic. Please check what in and out interface is recorded for each flow, ie grep for 178.148.77.73 but dont restrict on the interface. Also, please show a dmesg - we need to know what version you are running. /Benno I have enabled pflow for outbound traffic on $int_if and $ext_if first, and it appears that in this setup no redirected traffic is recorded by nfdump, either entering $ext_if and leaving $int_if on arrival, or entering $int_if and leaving $ext_if on return. Other kinds of traffic appear to be recorded correctly by pflow, including NAT traffic. Next, I enabled pflow for one additional inbound redirected rule: pass in on $if_ext inet proto tcp from any to $pub_srv port 1002 \ rdr-to $priv_srv keep state (pflow) In this setup flows appear to be recorded by nfdump fine on $int_if, both leaving it on arrival and entering it on return. Direction is correct. % nfdump -R 2014 -s srcip/bytes 'out if 5 and port 1002' Src IP AddrFlows(%) Packets(%) Bytes(%) 212.200.65.243 3678(34.9)24554(36.0)2.1 M(35.2) 212.200.65.244 2393(22.7)15331(22.5)1.4 M(23.3) 212.200.65.241 2457(23.3)15488(22.7)1.3 M(22.5) 212.200.65.242 2025(19.2)12765(18.7)1.1 M(19.0) % nfdump -R 2014 -s dstip/bytes 'in if 5 and port 1002' Dst IP AddrFlows(%) Packets(%) Bytes(%) 212.200.65.243 3678(34.9)20699(34.9)1.0 M(36.3) 212.200.65.241 2457(23.3)13572(22.9) 638520(22.5) 212.200.65.244 2393(22.7)13590(22.9) 619420(21.9) 212.200.65.242 2025(19.2)11496(19.4) 547616(19.3) However, on external interface the direction appears to be reversed (notice I need to request '$ext_if outbound srcip' in order to get '$ext_if outbound dstip': % nfdump -R 2014 -s srcip/bytes 'out if 4 and port 1002' Src IP AddrFlows(%) Packets(%) Bytes(%) 212.200.65.243 4051(35.0)26862(36.4)2.3 M(35.7) 212.200.65.244 2654(23.0)16771(22.7)1.5 M(23.4) 212.200.65.241 2683(23.2)16731(22.7)1.4 M(22.4) 212.200.65.242 2175(18.8)13475(18.2)1.2 M(18.5) Also I need to request '$ext_if inbound dstip' in order to get '$ext_if inbound srcip': % nfdump -R 2014 -s dstip/bytes 'in if 4 and port 1002' Dst IP AddrFlows(%) Packets(%) Bytes(%) 212.200.65.243 4051(35.0)22767(35.0)1.1 M(36.5) 212.200.65.241 2683(23.2)14756(22.7) 692652(22.4) 212.200.65.244 2654(23.0)15024(23.1) 683824(22.1) 212.200.65.242 2175(18.8)12409(19.1) 586820(19.0) I am using quite recent snapshot: OpenBSD 5.5-current (GENERIC.MP) #150: Mon May 26 11:50:31 MDT 2014 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2128887808 (2030MB) avail mem = 2063499264 (1967MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xee000 (69 entries) bios0: vendor HP version P58 date 05/02/2011 bios0: HP ProLiant DL360 G5 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP SPCR MCFG HPET SPMI ERST APIC BERT HEST SSDT acpi0: wakeup devices PCI0(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimcfg0 at acpi0 addr 0xe000, bus 0-255 acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.38 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF cpu0: 6MB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 333MHz cpu0: mwait min=64, max=64, C-substates=0.2.2.2.0, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2000.08 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF cpu1: 6MB 64b/line 16-way L2 cache cpu1: smt 0, core 2, package 0 cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.09 MHz cpu2:
netflow srcip and dstip reversed for redirected traffic
Hi,
I'm trying to understand and measure traffic on relatively large and
complicated pf firewall, and for this purpose I am exporting netflow
data with pflow to nfsen/nfdump.
For the time being, I have set pflow on external interface in outbound
direction:
pass out on $if_ext inet all keep state (pflow)
On collector (nfsen), I want to see interface numbers so i can create
interface filter:
% nfdump -R 2014 -s if/bytes
Top 10 In/Out If ordered by bytes:
If Flows(%) Packets(%) Bytes(%) pps bps bpp
519396(100.0) 300683(100.0) 186.7 M(100.0)316984 620
719109(98.5)299769(99.7)186.6 M(100.0)316976 622
0 287( 1.5) 914( 0.3)83170( 0.0) 0 33090
Another mailing list member told me I can find about interface numbers
with snmpwalk:
% snmpwalk -v2c -c community -On IP.ADD.RE.SS
.1.3.6.1.2.1.2.2.1.2.5 = STRING: bnx1
.1.3.6.1.2.1.2.2.1.2.7 = STRING: carp2
Ok, now I know interface 5 is bnx1 ($if_ext), and I want to know what
comes in:
% nfdump -R 2014 -s dstip/bytes 'in if 5'
Top 10 Dst IP Addr ordered by bytes:
Dst IP AddrFlows(%) Packets(%) Bytes(%)
10.20.0.1510754(62.9) 323834(52.9) 324.9 M(63.7)
10.20.4.99 462( 2.7)10496( 1.7)9.4 M( 1.8)
178.148.77.734( 0.0) 6681( 1.1)7.7 M( 1.5)
First two addresses really are on my internal network, and I know first
one is return web traffic to my proxy, and the second one return web
traffic to another internal host.
But the last address is not on my network. Let's see records for this
address:
nfdump -R 2014 -n 10 -s record/bytes 'in if 5' | grep 178.148.77.73
TCP 193.53.106.35:443 - 178.148.77.73:49193 56067.6 M
TCP 193.53.106.35:443 - 178.148.77.73:49191 31395342
TCP 193.53.106.35:443 - 178.148.77.73:49192 40418674
TCP 193.53.106.35:443 - 178.148.77.73:49190 35816798
Ok, these are redirected incoming requests to HTTPS server on my
internal network:
pass in on $if_ext inet proto tcp from any to $pub_web port { 80 443 } \
rdr-to $priv_web keep state
But source and destination IP adresses are reversed!
Here's what pf's state table shows:
$ sudo pfctl -ss | grep 178.148.77.73
all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49377
all tcp 178.148.77.73:49377 - 10.20.0.36:443
all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49378
all tcp 178.148.77.73:49378 - 10.20.0.36:443
all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49379
all tcp 178.148.77.73:49379 - 10.20.0.36:443
all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49380
all tcp 178.148.77.73:49380 - 10.20.0.36:443
How could this be corrected? Am I configuring pf incorrectly? Or is
there a problem with how pflow exports data? Or is pfdump parsing the
data incorrectly?
Thank you in advance,
--
Marko Cupać
Re: netflow srcip and dstip reversed for redirected traffic
Marko Cupa??(marko.cu...@mimar.rs) on 2014.05.31 13:03:18 +0200:
Hi,
I'm trying to understand and measure traffic on relatively large and
complicated pf firewall, and for this purpose I am exporting netflow
data with pflow to nfsen/nfdump.
For the time being, I have set pflow on external interface in outbound
direction:
pass out on $if_ext inet all keep state (pflow)
On collector (nfsen), I want to see interface numbers so i can create
interface filter:
% nfdump -R 2014 -s if/bytes
Top 10 In/Out If ordered by bytes:
If Flows(%) Packets(%) Bytes(%) pps bps bpp
519396(100.0) 300683(100.0) 186.7 M(100.0)316984 620
719109(98.5)299769(99.7)186.6 M(100.0)316976 622
0 287( 1.5) 914( 0.3)83170( 0.0) 0 33090
Another mailing list member told me I can find about interface numbers
with snmpwalk:
% snmpwalk -v2c -c community -On IP.ADD.RE.SS
.1.3.6.1.2.1.2.2.1.2.5 = STRING: bnx1
.1.3.6.1.2.1.2.2.1.2.7 = STRING: carp2
Ok, now I know interface 5 is bnx1 ($if_ext), and I want to know what
comes in:
% nfdump -R 2014 -s dstip/bytes 'in if 5'
Top 10 Dst IP Addr ordered by bytes:
Dst IP AddrFlows(%) Packets(%) Bytes(%)
10.20.0.1510754(62.9) 323834(52.9) 324.9 M(63.7)
10.20.4.99 462( 2.7)10496( 1.7)9.4 M( 1.8)
178.148.77.734( 0.0) 6681( 1.1)7.7 M( 1.5)
First two addresses really are on my internal network, and I know first
one is return web traffic to my proxy, and the second one return web
traffic to another internal host.
But the last address is not on my network. Let's see records for this
address:
nfdump -R 2014 -n 10 -s record/bytes 'in if 5' | grep 178.148.77.73
TCP 193.53.106.35:443 - 178.148.77.73:49193 56067.6 M
TCP 193.53.106.35:443 - 178.148.77.73:49191 31395342
TCP 193.53.106.35:443 - 178.148.77.73:49192 40418674
TCP 193.53.106.35:443 - 178.148.77.73:49190 35816798
Ok, these are redirected incoming requests to HTTPS server on my
internal network:
pass in on $if_ext inet proto tcp from any to $pub_web port { 80 443 } \
rdr-to $priv_web keep state
But source and destination IP adresses are reversed!
Here's what pf's state table shows:
$ sudo pfctl -ss | grep 178.148.77.73
all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49377
all tcp 178.148.77.73:49377 - 10.20.0.36:443
all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49378
all tcp 178.148.77.73:49378 - 10.20.0.36:443
all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49379
all tcp 178.148.77.73:49379 - 10.20.0.36:443
all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49380
all tcp 178.148.77.73:49380 - 10.20.0.36:443
How could this be corrected? Am I configuring pf incorrectly? Or is
there a problem with how pflow exports data? Or is pfdump parsing the
data incorrectly?
The simple answer: It's complicated.
The complicated answer: the pf state is used to keep track of both
directions of the traffic flow. When the state times out, _two_ flows are
created, one for each direction of traffic, you can see this in
copy_flow_ipfix_4_data() in /usr/src/sys/net/if_pflow.c.
For NAT/RDR its a bit more complicated, so what you are seeing might be
'normal' or a problem.
nfdump should be able to show you both directions of this traffic. Please
check what in and out interface is recorded for each flow, ie grep for
178.148.77.73 but dont restrict on the interface.
Also, please show a dmesg - we need to know what version you are running.
/Benno
