Re: netflow srcip and dstip reversed for redirected traffic

2014-06-01 Thread Marko Cupańá
On Sat, 31 May 2014 20:01:25 +0200
Sebastian Benoit benoit-li...@fb12.de wrote:

 The simple answer: It's complicated.
 
 The complicated answer: the pf state is used to keep track of both
 directions of the traffic flow. When the state times out, _two_ flows
 are created, one for each direction of traffic, you can see this in
 copy_flow_ipfix_4_data() in /usr/src/sys/net/if_pflow.c.
 
 For NAT/RDR its a bit more complicated, so what you are seeing might
 be 'normal' or a problem.
 
 nfdump should be able to show you both directions of this traffic.
 Please check what in and out interface is recorded for each flow, ie
 grep for 178.148.77.73 but dont restrict on the interface.
 
 Also, please show a dmesg - we need to know what version you are
 running.
 
 /Benno
 

I have enabled pflow for outbound traffic on $int_if and $ext_if first,
and it appears that in this setup no redirected traffic is recorded by
nfdump, either entering $ext_if and leaving $int_if on arrival, or
entering $int_if and leaving $ext_if on return. Other kinds of traffic
appear to be recorded correctly by pflow, including NAT traffic.

Next, I enabled pflow for one additional inbound redirected rule:

pass in on $if_ext inet proto tcp from any to $pub_srv port 1002 \
   rdr-to $priv_srv keep state (pflow)

In this setup flows appear to be recorded by nfdump fine on $int_if,
both leaving it on arrival and entering it on return. Direction is
correct.

% nfdump -R 2014 -s srcip/bytes 'out if 5 and port 1002'
   Src IP AddrFlows(%) Packets(%)   Bytes(%)
212.200.65.243 3678(34.9)24554(36.0)2.1 M(35.2)
212.200.65.244 2393(22.7)15331(22.5)1.4 M(23.3)
212.200.65.241 2457(23.3)15488(22.7)1.3 M(22.5)
212.200.65.242 2025(19.2)12765(18.7)1.1 M(19.0)

% nfdump -R 2014 -s dstip/bytes 'in if 5 and port 1002'
   Dst IP AddrFlows(%) Packets(%)   Bytes(%)
212.200.65.243 3678(34.9)20699(34.9)1.0 M(36.3)
212.200.65.241 2457(23.3)13572(22.9)   638520(22.5)
212.200.65.244 2393(22.7)13590(22.9)   619420(21.9)
212.200.65.242 2025(19.2)11496(19.4)   547616(19.3)

However, on external interface the direction appears to be reversed
(notice I need to request '$ext_if outbound srcip' in order to get
'$ext_if outbound dstip':

% nfdump -R 2014 -s srcip/bytes 'out if 4 and port 1002'
   Src IP AddrFlows(%) Packets(%)   Bytes(%)
212.200.65.243 4051(35.0)26862(36.4)2.3 M(35.7)
212.200.65.244 2654(23.0)16771(22.7)1.5 M(23.4)
212.200.65.241 2683(23.2)16731(22.7)1.4 M(22.4)
212.200.65.242 2175(18.8)13475(18.2)1.2 M(18.5)

Also I need to request '$ext_if inbound dstip' in order to get '$ext_if
inbound srcip':

% nfdump -R 2014 -s dstip/bytes 'in if 4 and port 1002'
   Dst IP AddrFlows(%) Packets(%)   Bytes(%)
212.200.65.243 4051(35.0)22767(35.0)1.1 M(36.5)
212.200.65.241 2683(23.2)14756(22.7)   692652(22.4)
212.200.65.244 2654(23.0)15024(23.1)   683824(22.1)
212.200.65.242 2175(18.8)12409(19.1)   586820(19.0)

I am using quite recent snapshot:

OpenBSD 5.5-current (GENERIC.MP) #150: Mon May 26 11:50:31 MDT 2014
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2128887808 (2030MB)
avail mem = 2063499264 (1967MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xee000 (69 entries)
bios0: vendor HP version P58 date 05/02/2011
bios0: HP ProLiant DL360 G5
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPCR MCFG HPET SPMI ERST APIC  BERT HEST
SSDT acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0 addr 0xe000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.38 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF
cpu0: 6MB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 333MHz
cpu0: mwait min=64, max=64, C-substates=0.2.2.2.0, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2000.08 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,LONG,LAHF,PERF
cpu1: 6MB 64b/line 16-way L2 cache cpu1: smt 0, core 2, package 0
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2500.09 MHz
cpu2:

Re: netflow srcip and dstip reversed for redirected traffic

2014-05-31 Thread Sebastian Benoit
Marko Cupa??(marko.cu...@mimar.rs) on 2014.05.31 13:03:18 +0200:
 Hi,
 
 I'm trying to understand and measure traffic on relatively large and
 complicated pf firewall, and for this purpose I am exporting netflow
 data with pflow to nfsen/nfdump.
 
 For the time being, I have set pflow on external interface in outbound
 direction:
 
 pass out on $if_ext inet all keep state (pflow)
 
 On collector (nfsen), I want to see interface numbers so i can create
 interface filter:
 
 % nfdump -R 2014 -s if/bytes
 Top 10 In/Out If ordered by bytes:
 If   Flows(%)   Packets(%)  Bytes(%) pps   bps bpp
 519396(100.0)   300683(100.0)   186.7 M(100.0)316984   620
 719109(98.5)299769(99.7)186.6 M(100.0)316976   622
 0  287( 1.5)   914( 0.3)83170( 0.0)   0  33090
 
 Another mailing list member told me I can find about interface numbers
 with snmpwalk:
 
 % snmpwalk -v2c -c community -On IP.ADD.RE.SS
 .1.3.6.1.2.1.2.2.1.2.5 = STRING: bnx1
 .1.3.6.1.2.1.2.2.1.2.7 = STRING: carp2
 
 Ok, now I know interface 5 is bnx1 ($if_ext), and I want to know what
 comes in:
 
 % nfdump -R 2014 -s dstip/bytes 'in if 5'
 Top 10 Dst IP Addr ordered by bytes:
 Dst IP AddrFlows(%) Packets(%)   Bytes(%)
  10.20.0.1510754(62.9)   323834(52.9)  324.9 M(63.7)
  10.20.4.99  462( 2.7)10496( 1.7)9.4 M( 1.8)
   178.148.77.734( 0.0) 6681( 1.1)7.7 M( 1.5)
 
 First two addresses really are on my internal network, and I know first
 one is return web traffic to my proxy, and the second one return web
 traffic to another internal host.
 
 But the last address is not on my network. Let's see records for this
 address:
 
 nfdump -R 2014 -n 10 -s record/bytes 'in if 5' | grep 178.148.77.73
 TCP  193.53.106.35:443 - 178.148.77.73:49193 56067.6 M
 TCP  193.53.106.35:443 - 178.148.77.73:49191  31395342
 TCP  193.53.106.35:443 - 178.148.77.73:49192  40418674
 TCP  193.53.106.35:443 - 178.148.77.73:49190  35816798
 
 Ok, these are redirected incoming requests to HTTPS server on my
 internal network:
 
 pass in on $if_ext inet proto tcp from any to $pub_web port { 80 443 } \
  rdr-to $priv_web keep state
 
 But source and destination IP adresses are reversed!
 
 Here's what pf's state table shows:
 $ sudo pfctl -ss | grep 178.148.77.73  
 all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49377
 all tcp 178.148.77.73:49377 - 10.20.0.36:443
 all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49378
 all tcp 178.148.77.73:49378 - 10.20.0.36:443
 all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49379
 all tcp 178.148.77.73:49379 - 10.20.0.36:443
 all tcp 10.20.0.36:443 (193.53.106.35:443) - 178.148.77.73:49380
 all tcp 178.148.77.73:49380 - 10.20.0.36:443
 
 How could this be corrected? Am I configuring pf incorrectly? Or is
 there a problem with how pflow exports data? Or is pfdump parsing the
 data incorrectly?

The simple answer: It's complicated.

The complicated answer: the pf state is used to keep track of both
directions of the traffic flow. When the state times out, _two_ flows are
created, one for each direction of traffic, you can see this in
copy_flow_ipfix_4_data() in /usr/src/sys/net/if_pflow.c.

For NAT/RDR its a bit more complicated, so what you are seeing might be
'normal' or a problem.

nfdump should be able to show you both directions of this traffic. Please
check what in and out interface is recorded for each flow, ie grep for
178.148.77.73 but dont restrict on the interface.

Also, please show a dmesg - we need to know what version you are running.

/Benno