I had this gateway with NAT working fine until I added another for
load balancing using carp. So now I've been slowly discovering the ins
and outs of carp in PF rules. Namely that packets seem to be going in
and out of the physical interfaces, but in on the carp interfaces at
the same time. Only a detail I've noted with tcpdump.
vic0 is management only, works fine.
vic1 is pfsync only, set skip
vic2 is external public ipv4 address, seen here below as XX.XX.XX.59.
vic3 is the first internal network 10.221.181.0/24
vic4 is the second internal network 10.221.182.0/24
carp0 is vic2 on both machines, there is no IP-information on the
physical carpdev's, only on the carps.
carp1 is vic3
carp2 is vic4
There are no pfsync update errors in syslog, I can see carp traffic
pass between the carpdevs using proto carp.
Both loadbalancers are identical in configuration and I use git,
public ssh keys and bash scripts to update the configuration on the
git-server, update it on both loadbalancers and then run pfctl -vf
/etc/pf.conf on both.
Here is output of pfctl -vvsr, please excuse the mail formatting. I'm
hoping this will shed light on my ruleset.
@0 block drop on vic0 all
[ Evaluations: 353 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@1 pass in quick on vic0 inet proto tcp from any to 10.220.100.0/24
port = 1022 flags S/SA keep state label "PassInMGMTSSH"
[ Evaluations: 347 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@2 pass in quick on vic0 inet proto tcp from any to 10.220.100.0/24
port = ssh flags S/SA keep state label "PassInMGMTSSH" [ Evaluations:
0 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@3 pass out quick on vic0 inet proto tcp from 10.220.100.0/24 to any
port = 1022 flags S/SA keep state label "PassOutMGMTSSH"
[ Evaluations: 16 Packets: 136 Bytes: 19978 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 1 ]
@4 pass out quick on vic0 inet proto tcp from 10.220.100.0/24 to any
port = ssh flags S/SA keep state label "PassOutMGMTSSH"
[ Evaluations: 15 Packets: 1020 Bytes: 441024 States: 4
]
[ Inserted: uid 0 pid 3940 State Creations: 30 ]
@5 pass on vic0 proto udp from any to any port = domain keep state
label "PassMGMTDNS"
[ Evaluations: 331 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@6 pass on vic0 inet proto icmp all icmp-type echorep keep state label
"PassMGMTICMP"
[ Evaluations: 331 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@7 pass on vic0 inet proto icmp all icmp-type echoreq keep state label
"PassMGMTICMP"
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@8 pass on vic0 inet proto icmp all icmp-type unreach keep state label
"PassMGMTICMP"
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@9 pass quick on vic2 proto carp all keep state label "CUST-PassCarp"
[ Evaluations: 337 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@10 pass quick on vic3 proto carp all keep state label "CUST-PassCarp"
[ Evaluations: 332 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@11 pass quick on vic4 proto carp all keep state label "CUST-PassCarp"
[ Evaluations: 331 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@12 block drop log all
[ Evaluations: 337 Packets: 335 Bytes: 36237 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@13 pass log inet proto icmp from any to XX.XX.XX.59 icmp-type echoreq
keep state label "CUST-PingOutPhys"
[ Evaluations: 337 Packets: 1722 Bytes: 144648 States: 1
]
[ Inserted: uid 0 pid 3940 State Creations: 1 ]
@14 pass log inet proto icmp from any to XX.XX.XX.59 icmp-type echorep
keep state label "CUST-PingOutPhys"
[ Evaluations: 1 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@15 pass log inet proto icmp from any to XX.XX.XX.59 icmp-type unreach
keep state label "CUST-PingOutPhys"
[ Evaluations: 1 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@16 pass log inet proto icmp from any to 10.221.181.10 icmp-type
echoreq keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 884 Bytes: 74256 States: 1
]
[ Inserted: uid 0 pid 3940 State Creations: 2 ]
@17 pass log inet proto icmp from any to 10.221.182.10 icmp-type
echoreq keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@18 pass log inet proto icmp from any to 10.221.181.10 icmp-type
echorep keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@19 pass log inet proto icmp from any to 10.221.182.10 icmp-type
echorep keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@20 pass log inet proto icmp from any to 10.221.181.10 icmp-type
unreach keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@21 pass log inet proto icmp from any to 10.221.182.10 icmp-type
unreach keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@22 pass log inet proto icmp from 10.221.181.10 to any icmp-type
echoreq keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@23 pass log inet proto icmp from 10.221.182.10 to any icmp-type
echoreq keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@24 pass log inet proto icmp from 10.221.181.10 to any icmp-type
echorep keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@25 pass log inet proto icmp from 10.221.182.10 to any icmp-type
echorep keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@26 pass log inet proto icmp from 10.221.181.10 to any icmp-type
unreach keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@27 pass log inet proto icmp from 10.221.182.10 to any icmp-type
unreach keep state label "CUST-PingInternalGW"
[ Evaluations: 2 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@28 match out log on vic2 from ! (egress:2) to any label
"CUST-MatchOutNAT" nat-to (egress:2) round-robin
[ Evaluations: 337 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@29 pass out log inet proto tcp from XX.XX.XX.59 to any port = www
flags S/SA keep state label "CUST-PassOutPorts"
[ Evaluations: 332 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@30 pass out log inet proto tcp from XX.XX.XX.59 to any port = https
flags S/SA keep state label "CUST-PassOutPorts"
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@31 pass out log inet proto tcp from XX.XX.XX.59 to any port = smtp
flags S/SA keep state label "CUST-PassOutPorts"
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@32 pass out log inet proto tcp from XX.XX.XX.59 to any port = ssh
flags S/SA keep state label "CUST-PassOutPorts"
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@33 pass out log inet proto tcp from XX.XX.XX.59 to any port = 1022
flags S/SA keep state label "CUST-PassOutPorts"
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@34 pass in log on vic3 inet proto tcp from 10.221.181.0/24 to any
flags S/SA keep state
[ Evaluations: 337 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@35 pass in log on vic3 inet proto tcp from 10.221.182.0/24 to any
flags S/SA keep state
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@36 pass in log on vic4 inet proto tcp from 10.221.181.0/24 to any
flags S/SA keep state
[ Evaluations: 336 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@37 pass in log on vic4 inet proto tcp from 10.221.182.0/24 to any
flags S/SA keep state
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@38 pass out log on vic2 inet proto tcp from 10.221.181.0/24 to any
flags S/SA keep state
[ Evaluations: 336 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@39 pass out log on vic2 inet proto tcp from 10.221.182.0/24 to any
flags S/SA keep state
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@40 anchor "ftp-proxy/*" all
[ Evaluations: 337 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
@41 pass in log on vic3 inet proto tcp from any to any port = ftp
flags S/SA keep state label "CUST-PassInRDRFTP" rdr-to 127.0.0.1 port
8021
[ Evaluations: 337 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 3940 State Creations: 0 ]
My real issue is with connections going out from the LAN to remote
destinations on the internet.
>From a LAN server I do nc -vv 212.211.132.250 80 # (this is
security.debian.org) and get the following on pflog0.
Oct 17 07:47:30.793687 rule 34/(match) pass in on vic3:
10.221.181.21.45667 > 212.211.132.250.80: S 3729652686:3729652686(0)
win 5840 <mss 1460,sackOK,timestamp 85822017 0,nop,wscale 7> (DF)
Oct 17 07:47:30.793786 rule 28/(match) match out on vic2:
XX.XX.XX.59.57433 > 212.211.132.250.80: S 3729652686:3729652686(0) win
5840 <mss 1460,sackOK,timestamp 85822017 0,nop,wscale 7> (DF)
Oct 17 07:47:30.793800 rule 29/(match) pass out on vic2:
XX.XX.XX.59.57433 > 212.211.132.250.80: S 3729652686:3729652686(0) win
5840 <mss 1460,sackOK,timestamp 85822017 0,nop,wscale 7> (DF)
So this tells me that NAT is being done, and it is even sending
packets out on the external carpdev (vic2) to the remote destination.
But nothing comes back and the connection times out. I am not a
networking guy but TCP is a streaming protocol, I know that much, am I
to allow packets coming back through that stream? Or even packets
coming back to establish the stream?
Even removing 'tcp' from that tcpdump command doesn't give me any new
clues as to what is happening to the rest of the stream. If I do nc
-vv 212.211.132.250 80 from the shell on the gateway it works fine.
Strangely enough the same command from lb02 does not work, it triggers
rule 29 in tcpdump of pflog0 on both systems but on lb02 no connection
is established.
lb01 $
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 01:00:5e:00:01:01
description: Customer External Public Loadbalancer
priority: 0
carp: carpdev vic2 advbase 1 balancing ip
state MASTER vhid 1 advskew 0
state MASTER vhid 2 advskew 100
groups: carp egress
status: master
inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x8
inet XX.XX.XX.59 netmask 0xffffff00 broadcast XX.XX.XX.255
vic2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 00:50:56:8e:00:63
description: Customer External
priority: 0
media: Ethernet autoselect
status: active
inet6 fe80::250:56ff:fe8e:63%vic2 prefixlen 64 scopeid 0x3
lb02 $
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 01:00:5e:00:01:01
description: Customer External Public Loadbalancer
priority: 0
carp: carpdev vic2 advbase 1 balancing ip
state MASTER vhid 1 advskew 100
state MASTER vhid 2 advskew 0
groups: carp egress
status: master
inet6 fe80::200:5eff:fe00:102%carp0 prefixlen 64 scopeid 0x8
Connection to 10.220.100.53 closed.
inet XX.XX.XX.59 netmask 0xffffff00 broadcast XX.XX.XX.255
vic2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 00:50:56:8e:00:5e
description: Customer External
priority: 0
media: Ethernet autoselect
status: active
Connection to 10.220.100.53 closed.
inet6 fe80::250:56ff:fe8e:5e%vic2 prefixlen 64 scopeid 0x3
Now there is actually another issue here that is of lower priority,
the fact that I can ping the external IP of the gateway without issue
but I can't have my LAN servers ping their gateways, meaning the
IP-addresses of carp1 and carp2. In that case I can see packets going
in on the physical interface behind carp1, but no reply comes back.
Oct 17 07:45:48.710962 rule 16/(match) pass in on vic3: 10.221.181.21
> 10.221.181.10: icmp: echo request (DF)
This is strange to me as the rule to allow ICMP on the external
interface is nearly identical to the ones to allow on the internal
interfaces.
--
Med vdnliga hdlsningar / With kind regards
Stefan Midjich
