On Tue, Mar 22, 2022 at 09:56:49AM -0500, rea...@catastrophe.net wrote:
>Rules on both sides are:
>
># server-east
>--------------
>pass in proto udp from any to self port { isakmp, ipsec-nat-t } keep state
>pass out proto udp from any to any port { isakmp, ipsec-nat-t } keep state
>
>pass in proto { esp, ah } from any to vio0 keep state
>pass out proto { esp, ah } from vio0 to any keep state
>
>pass on log enc0 keep state (if-bound) tagged VPN.LAX
>pass on log enc0 keep state (if-bound)
[..]
It definitely looks like my issue is the ipsec flows and not pf.
I start a ping on server-east after the ipsec flows are established and
traffic sourcing from server-east vio0 (100.64.1.92) take the default route
and do not get encapsulated when destined for enc0 on server-west
(10.255.255.1):
Flows:
server-east# ipsecctl -sa | grep 10.255.255
flow esp in from 10.255.255.0/24 to 10.254.255.0/24 peer 203.0.113.50 srcid
FQDN/server-east.example.com dstid FQDN/server-west.example.com type require
flow esp in from 10.255.255.0/24 to 100.64.1.92 peer 203.0.113.50 srcid
FQDN/server-east.example.com dstid FQDN/server-west.example.com type require
flow esp out from 10.254.255.0/24 to 10.255.255.0/24 peer 203.0.113.50 srcid
FQDN/server-east.example.com dstid FQDN/server-west.example.com type require
flow esp out from 100.64.1.92 to 10.255.255.0/24 peer 203.0.113.50 srcid
FQDN/server-east.example.com dstid FQDN/server-west.example.com type require
Start generating traffic:
server-east# ping 10.255.255.1 &
[1] 55341
server-east# PING 10.255.255.1 (10.255.255.1): 56 data bytes
Capture packets:
server-east# tcpdump -ni vio0 host 10.255.255.1
tcpdump: listening on vio0, link-type EN10MB
14:58:42.514298 100.64.1.92 > 10.255.255.1: icmp: echo request
14:58:43.514396 100.64.1.92 > 10.255.255.1: icmp: echo request
14:58:44.514326 100.64.1.92 > 10.255.255.1: icmp: echo request
14:58:45.514310 100.64.1.92 > 10.255.255.1: icmp: echo request
14:58:46.514340 100.64.1.92 > 10.255.255.1: icmp: echo request
14:58:47.514263 100.64.1.92 > 10.255.255.1: icmp: echo request
^C
29 packets received by filter
0 packets dropped by kernel
server-east# fg
ping 10.255.255.1
^C
--- 10.255.255.1 ping statistics ---
19 packets transmitted, 0 packets received, 100.0% packet loss
If I source from server-east enc0 (10.254.255.1), then the traffic goes over
the ipsec tunnel:
server-east# ping -c 5 -I 10.254.255.1 10.255.255.1
PING 10.255.255.1 (10.255.255.1): 56 data bytes
64 bytes from 10.255.255.1: icmp_seq=0 ttl=255 time=26.369 ms
64 bytes from 10.255.255.1: icmp_seq=1 ttl=255 time=26.412 ms
64 bytes from 10.255.255.1: icmp_seq=2 ttl=255 time=26.519 ms
64 bytes from 10.255.255.1: icmp_seq=3 ttl=255 time=26.426 ms
64 bytes from 10.255.255.1: icmp_seq=4 ttl=255 time=26.575 ms
--- 10.255.255.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 26.369/26.460/26.575/0.075 ms
Flows from em0 on server-west (203.0.113.50) have no problem when sourcing
from either it's enc0 (10.255.255.1) or em0 (203.0.113.50) interfaces toward
10.254.255.1. Those flows look like this, and they're basically a reverse of
that which is on server-east.
server-west# ipsecctl -sa | grep 10.254.255
flow esp in from 10.254.255.0/24 to 10.255.255.0/24 peer 100.64.1.92 srcid
FQDN/server-west.example.com dstid FQDN/server-east.example.com type require
flow esp in from 10.254.255.0/24 to 203.0.113.50 peer 100.64.1.92 srcid
FQDN/server-west.example.com dstid FQDN/server-east.example.com type require
flow esp out from 10.255.255.0/24 to 10.254.255.0/24 peer 100.64.1.92 srcid
FQDN/server-west.example.com dstid FQDN/server-east.example.com type require
flow esp out from 203.0.113.50 to 10.254.255.0/24 peer 100.64.1.92 srcid
FQDN/server-west.example.com dstid FQDN/server-east.example.com type require
