pf.conf "reply-to" routing parameter seemingly not working?

[Joe Crivello]

Hello!

I have a trunk0 interface on a router (#1) that is used for a singular
purpose -- to pass (IPsec protected) traffic for an IPIP tunnel (gif0) to
another router (#2). I have configured PF rules on router #1 that prevent
any other type of traffic from passing on trunk0. There are several routing
table entries that forward to router #2 on gif0.

My objective is to configure an additional pass rule that would allow SSH
traffic destined for router #1 to pass in and out on trunk0.

The problem is that the aforementioned routes on gif0 cause packets sent in
reply to incoming SSH traffic to pass out on gif0 (after passing in on
trunk0). This ends up getting blocked by PF on router #1 because the
state-policy is set to if-bound (which is how I want it). I am trying to
use reply-to to enforce symmetric routing, but it isn't working.

As you will see below, my "reply-to" rule is matched, but the reply is
_still_ routed to gif0:

# tcpdump -nevvpi pflog0 tcp port 22
tcpdump: WARNING: snaplen raised from 116 to 224
tcpdump: listening on pflog0, link-type PFLOG
01:27:46.503040 rule 5/(match) [uid 0, pid 16018] pass in on trunk0: [uid
4294967295, pid 10] [SSH CLIENT IP].57427 > [TRUNK0 IP].22: S [tcp sum
ok] 1707770457:1707770457(0) win 64240  (DF) (ttl 127, id 24244, len 52)
01:27:46.503069 rule 4/(match) [uid 0, pid 16018] block out on gif1: [uid
4294967295, pid 10] [TRUNK0 IP].22 > [SSH CLIENT IP].57427: S [tcp sum
ok] 4100262020:4100262020(0) ack 1707770458 win 16384  (DF) (ttl 64, id 43497, len 52, bad ip
cksum 0! -> d71b)
^C
2 packets received by filter
0 packets dropped by kernel

# pfctl -sr | grep @5
@5 pass in log quick on trunk0 inet proto tcp from any to [TRUNK0 IP] port
= 22 flags S/SA keep state (if-bound) reply-to @trunk0

Router #1 is running OpenBSD 6.2.

Anyone have any idea why this isn't working the way I want it to?

Joe

Re: ICMPv6 Neighbor Advertisement PF Weirdness

[Joe Crivello]

I solved the problem described in my last email.

The problem was that we copy pasted the IPv6 address for each vlan
interface, and then changed part of the address for each interface, but
failed to change the prefix length to 64. This meant that each vlan
interface had a different address, but shared the same subnet with other
interfaces. Obviously this resulted in an "unusual" route table -- but it
is unclear to us why the previously described PF problem manifested in the
way it did -- especially given that the ICMPv6 packet used link-local
addresses, and the pass rule did not filter on interface. Seems like it is
possibly a bug.

Joe

On Mon, Apr 30, 2018 at 12:31 PM, Joe Crivello <josephcrive...@gmail.com>
wrote:

> Hello --
>
> While configuring a new firewall, I noticed that pflog0 was showing that
> some ICMPv6 neighbor advertisement packets were being blocked in on vlan51,
> which is a sub-interface of vmx1 (a vmxnet3 interface using VGT). I added a
> PF rule allowing this traffic to pass. However, even after adding the rule
> and flushing all state, the traffic was still being reported as blocked in
> by pflog0. Thinking that there might be something else wrong with the rule
> set, I made the pass rule "quick" and inserted it as the first pass rule in
> the set. This still didn't work.
>
> See below for (1) the first two rules of the rule set (they are the only
> ones that matter in this example), (2) a tcpdump that was run after the
> rule set was applied and states were flushed, showing the blocked traffic,
> and (3) the interface configurations in question.
>
> # pfctl -sr
> FILTER RULES:
> block drop log all label "Block all"
> pass in quick on any inet6 proto ipv6-icmp all icmp6-type neighbradv
>
> # tcpdump -netvvi pflog0 "icmp6 && ip6[40] == 136"
> tcpdump: WARNING: snaplen raised from 116 to 160
> tcpdump: listening on pflog0, link-type PFLOG
> rule 1/(match) [uid 0, pid 91607] block in on vlan51:
> fe80::1905:23c0:fe7a:dcfe > ff02::1: [|icmp6] (len 32, hlim 255)
> ^C
> 7 packets received by filter
> 0 packets dropped by kernel
>
> # ifconfig vmx1
> vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> lladdr 00:0c:29:50:66:d1
> index 2 priority 0 llprio 3
> media: Ethernet autoselect (10GbaseT)
> status: active
> inet6 fe80::20c:29ff:fe50:66d1%vmx1 prefixlen 64 scopeid 0x2
>
> # ifconfig vlan51
> vlan51: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> lladdr 00:0c:29:50:66:d1
> description: 
> index 7 priority 0 llprio 3
> encap: vnetid 51 parent vmx1
> groups: vlan inetusr resolverusr
> media: Ethernet autoselect (10GbaseT)
> status: active
> inet6 fe80::20c:29ff:fe50:66d1%vlan51 prefixlen 64 scopeid 0x7
> inet 10.84.51.1 netmask 0xff00 broadcast 10.84.51.255
> inet6 ::1 prefixlen 56
>
> The firewall is a VMware ESXi 6.5 virtual machine running
> SecurityRouter.org 6.2-lyra -- which is a distribution based on OpenBSD 6.2.
>
> Thinking that this might be a problem with the vmx(4) driver, we changed
> the network interface to emulated E1000e (em(4)), no change. Also tried
> adding "allow-opts" to the pass rule, no change.
>
> I understand this list isn't meant to support SecurityRouter.org's
> distribution of OpenBSD... but does anyone see something obviously wrong
> with my rule set or my expectations of how it should behave? Are there
> known problems with using VGT on VMware ESXi with vmx(4) and em(4) drivers?
> I reviewed the 6.2 errata and didn't see anything pertinent.
>
> Joe
>
>

ICMPv6 Neighbor Advertisement PF Weirdness

[Joe Crivello]

Hello --

While configuring a new firewall, I noticed that pflog0 was showing that
some ICMPv6 neighbor advertisement packets were being blocked in on vlan51,
which is a sub-interface of vmx1 (a vmxnet3 interface using VGT). I added a
PF rule allowing this traffic to pass. However, even after adding the rule
and flushing all state, the traffic was still being reported as blocked in
by pflog0. Thinking that there might be something else wrong with the rule
set, I made the pass rule "quick" and inserted it as the first pass rule in
the set. This still didn't work.

See below for (1) the first two rules of the rule set (they are the only
ones that matter in this example), (2) a tcpdump that was run after the
rule set was applied and states were flushed, showing the blocked traffic,
and (3) the interface configurations in question.

# pfctl -sr
FILTER RULES:
block drop log all label "Block all"
pass in quick on any inet6 proto ipv6-icmp all icmp6-type neighbradv

# tcpdump -netvvi pflog0 "icmp6 && ip6[40] == 136"
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
rule 1/(match) [uid 0, pid 91607] block in on vlan51:
fe80::1905:23c0:fe7a:dcfe > ff02::1: [|icmp6] (len 32, hlim 255)
^C
7 packets received by filter
0 packets dropped by kernel

# ifconfig vmx1
vmx1: flags=8843 mtu 1500
lladdr 00:0c:29:50:66:d1
index 2 priority 0 llprio 3
media: Ethernet autoselect (10GbaseT)
status: active
inet6 fe80::20c:29ff:fe50:66d1%vmx1 prefixlen 64 scopeid 0x2

# ifconfig vlan51
vlan51: flags=8843 mtu 1500
lladdr 00:0c:29:50:66:d1
description: 
index 7 priority 0 llprio 3
encap: vnetid 51 parent vmx1
groups: vlan inetusr resolverusr
media: Ethernet autoselect (10GbaseT)
status: active
inet6 fe80::20c:29ff:fe50:66d1%vlan51 prefixlen 64 scopeid 0x7
inet 10.84.51.1 netmask 0xff00 broadcast 10.84.51.255
inet6 ::1 prefixlen 56

The firewall is a VMware ESXi 6.5 virtual machine running
SecurityRouter.org 6.2-lyra -- which is a distribution based on OpenBSD 6.2.

Thinking that this might be a problem with the vmx(4) driver, we changed
the network interface to emulated E1000e (em(4)), no change. Also tried
adding "allow-opts" to the pass rule, no change.

I understand this list isn't meant to support SecurityRouter.org's
distribution of OpenBSD... but does anyone see something obviously wrong
with my rule set or my expectations of how it should behave? Are there
known problems with using VGT on VMware ESXi with vmx(4) and em(4) drivers?
I reviewed the 6.2 errata and didn't see anything pertinent.

Joe

What is the future of the multicast routing daemons in OpenBSD?

[Joe Crivello]

I am wondering about the future of the multicast routing daemons in base on
OpenBSD -- is the future in mrouted(8), or dvmrpd(8), or something else? I
am aware that dvmrpd(8) is the newer codebase -- but when I have tried to
use it in practice I have had consistent problems with the daemon crashing
under normal scenarios.

In my most recent dvmrpd test scenario, I set net.inet.ip.mforwarding to 1,
created the following dvmrpd.conf configuration file, ran the daemon in
debug mode and then started the Windows version of the Sonos Controller
application on, in this scenario, VLAN 3 (which immediately makes dvmrpd
crash):

# cat dvmrpd.conf
group {
  interface vlan3
  interface vlan5
}

# dvmrpd -dvf /etc/dvmrpd.conf
startup
kmr_init: interface vlan5
kmr_init: interface vlan3
fsm_if: event 'UP' resulted in action 'START' and changing state for
interface vlan5 from 'DOWN' to 'QUERIER'
fsm_if: event 'UP' resulted in action 'START' and changing state for
interface vlan3 from 'DOWN' to 'QUERIER'
rt_insert: inserting route 10.82.5.0/24
rt_insert: inserting route 10.82.3.0/24
send_igmp_query: interface vlan5
send_igmp_query: interface vlan3
[... START SONOS CONTROLLER ON VLAN 3 ...]
mfc_start_expire_timer: group 239.255.255.250
mfc_start_prune_timer: group 239.255.255.250
fatal in dvmrpe: unknown neighbor to send prune
mrt_add_mfc: interface 13, group 239.255.255.250
fatal in rde: pipe closed
fatal in parent: pipe closed

Initially, the daemon _appears_ to be running fine, until I start the Sonos
Controller application at the point indicated above, when the daemon
immediately outputs the errors shown above and exits. The culprit packet
that appears to cause dvmrpd to crash is a SSDP discovery packet sent by
the Sonos Controller application to 239.255.255.250 (see below):

tcpdump: listening on vlan3, link-type EN10MB
17:45:10.417081 10.82.3.1 > 224.0.0.1: igmp query [tos 0xc0] [ttl 1] (id
23931, len 28)
17:45:10.417103 10.82.3.1 > 224.0.0.4: igmp dvmrp Probe
genid 2257873240 [tos 0xc0] [ttl 1] (id 35152, len 32)
17:45:19.248851 10.82.3.100 > 239.255.255.250: igmp nreport 239.255.255.250
[ttl 1] (id 31513, len 32, optlen=4 IPOPT-148{4})
17:45:20.426728 10.82.3.1 > 224.0.0.4: igmp dvmrp Probe
genid 2257873240 [tos 0xc0] [ttl 1] (id 41676, len 32)
[... START SONOS CONTROLLER ON VLAN 3 ...]
17:45:25.376993 10.82.3.100.1901 > 239.255.255.250.1900: udp 230 (ttl 4, id
31514, len 258)

I have had relatively more success with mrouted(8). It seems to work in
most of the scenarios I have used it in, although in one case I have not
been able to get it to work on one particular piece of hardware for no
discernible reason -- no errors, no warnings found anywhere, it just fails
to route multicast packets. mrouted(8) has a significant limitation though
-- it only supports 32 VIFs on the entire system. If you have more than 32
virtual interfaces it will ignore the 33rd interface and beyond (the system
decides the ordering of the interfaces).

I can't use igmpproxy from ports because I require multiple "upstream"
interfaces.

I have been thinking about jumping into the code to try to address these
issues, but I'm not sure where to direct my efforts. I am inclined to focus
on the crash in dvmrpd -- but it's not clear from anything I have read that
dvmrpd is the future of multicast routing in OpenBSD. I have briefly looked
into the feasibility of increasing the number of VIFs supported by mrouted,
and it looks like that may be non-trivial, and it seems even less likely
that mrouted is the future of multicast routing on OpenBSD. I am also
troubled by the no error, no warning failure of mrouted on the system I
described above -- wouldn't even know where to start with that one.

Thanks in advance for any advice on this subject...

Joe Crivello

Logic Supply Nuvo-5000 / Intel Q170 Chipset

[Joe Crivello]

Does anyone out there on the list have experience with the Intel Q170
chipset on OpenBSD 6.0?

We are looking into using the Logic Supply Nuvo 5000LP (
http://www.logicsupply.com/nuvo-5000lp/#specs) as a high performance router
in some challenging environmental conditions. It would use an Intel Q170
chipset with an Intel i7-6700TE CPU and I210 and I219 GbE controllers.

Thanks in advance for any responses...

Joe Crivello

Re: Recommendation for firewall appliance running of and OpenBSD

[Joe Crivello]

> As far as I know, Halon cuts the number of IPSec tunnels on free version.


You're paying for ease of use and polish. Software developers aren't free.

Re: Recommendation for firewall appliance running of and OpenBSD

[Joe Crivello]

> Can somebody please recommend me a firewall appliance that can run OpenBSD
and
> pf, and can be upgradeable to the latest version? It would be a great plus
if
> the appliance can also be configured as part of CARP firewall group.


http://securityrouter.org/

Great product.

Re: recommendations for 10GBase Ethernet on OpenBSD

[Joe Crivello]

Intel X520 cards seem to work nicely in our shop.

Intel C61X / C22X Chipset Support

[Joe Crivello]

Does anyone have any experience with running OpenBSD on the Intel C61X or
C22X series chipsets? These chipsets are used frequently by Super Micro in
their newer line of very common and (relatively) affordable rack mountable
servers. In particular I am also curious if the SATA ports would be
supported in AHCI mode.

I did a quick grep of sys in 5.7 and current, and it looked to me like
there might be some support for C61X in IDE mode using the pciide driver,
but nothing for C22X. This matches with the conclusions of a few older
threads I found on this list.

Also, does anyone know of any developer activity or interest in either of
these two chipsets?

-Joe

Re: Intel C61X / C22X Chipset Support

[Joe Crivello]

Awesome! Thanks so much.

So C22X gets detected as an Intel 8 series chipset then (which makes
sense). We are initially thinking about using a couple of Super Micro
5018D-MR servers with Intel X520 cards as routers, so the lack of onboard
Ethernet support is not a problem for us. That said, if there are any
developers out there who are interested in looking into the problem, we'd
donate an Intel C224 motherboard.

We'll contribute back a report including a dmesg once it's up and running.

Anyone out there with a C61X chipset?

-Joe

Re: Executable signing - a proposal

[Joe Crivello]

First off, you seem to have set forth a design without first setting forth
it's objective. I suppose in this case it's pretty clear what your implied
objectives are, though.

Traditionally, executable or code signing is used to certify who compiled
a binary, and to prove that it wasn't tampered with since the time it was
compiled by that party. This is an effective and useful way of proving that
a binary hasn't been tampered with in the process of distribution from the
producer of that binary to the user of that binary (via the Internet, or a
CD/DVD and retail distribution, or whatever). In schemes like this, the
binary is verified only once, when it is installed. OpenBSD already has a
system that is similar to this; signify(1).

But your scheme would go farther than this. You seem to be advocating for a
system that also endeavors to protect binaries from tampering while they
are at rest on your system, by reverifying them before every execution.
To me... this would have a dubious benefit at a very, very high performance
cost -- and it goes without saying that you would sign hashes of the
binaries, not the binaries themselves. That's not optional or some kind of
an optimization... it's an intrinsic part of every cryptographic message
signing scheme that I know of. Regardless, verifying a signature with a
public key every time an executable is launched is going to make launching
executables probably *orders of magnitude* slower.

Ok, so you are probably going to say that there are people that would
accept that performance hit in exchange for the putative security
benefits...

So now let's talk about a few of the at rest tampering scenarios you may
be trying to protect against:

1. An attacker modifies the disk of the system while it is halted, to add
malware to the system.

In this case, your scheme provides little value because the attacker could
also trivially disable the signed executable verification feature (probably
by simply editing a text file). A more stealthy attacker could modify the
system's signed executable verification function to run a million tight
loops doing nothing, and then always return OK.

2. An attacker sends a sequence of network packets to the system that
causes it to execute some of the attacker's code (i.e. a remote exploit).
This presupposes the existence of a remote exploit vulnerability in the
system.

In this case, your scheme does nothing to prevent the attacker's code from
running, because it could only verify executable code immediately before a
new process is launched. If the attacker's code is running in a privileged
process, then it can disable the signed executable verification scheme. If
the attacker's code is not running in a privileged process, then the
attacker probably isn't in any position to tamper with a system binary in
the first place, rendering this scenario irrelevant.

3. A rogue administrator surreptitiously installs malware onto a system he
or she manages in coordination with other people.

Obviously in this case the rogue administrator could mount the same attack
as in #1.

I can't think of any other scenarios right now, but I'd be interested to
hear if there is something I'm not thinking of...

-Joe

Re: Executable signing - a proposal

[Joe Crivello]

 To prevent (in theory) various attack vectors (e.g., physical access to
the disk while offline), you need to have the system in a trusted state.

 Somebody has already thought this through, here is the result:

http://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface#Secure_boot

 Such a fully trusted, BSD-licensed OpenBSD boot chain, where I can put my
own keys into the BIOS, would be nice to have. Good luck writing it ;)

Secure Boot only protects the boot loader, and potentially the kernel and
it's drivers *if* the boot loader implements that. Any other system file on
the disk is totally unprotected, even in a OS like Windows 8+ which has
implemented a fully signed boot chain (through the kernel and it's
drivers). This means that even a system using Secure Boot is very easy to
root with offline physical access.

You actually could expand Mr. Nelson's proposal into something more
interesting by starting with a Windows-style fully signed boot chain
(through the kernel) and then also verifying launched binaries in the
manner he suggests. This would ostensibly resist all attempts to tamper
with the kernel or system binaries, even with offline physical access.
However, it wouldn't protect anything else, including configuration files,
which would still leave gaping holes open to the offline physical access
attack. It also could not be configurable and it would need to hard code
the keys, which strikes me as inelegant and inflexible. Seems like an
enormous amount of effort with little return...

FYI... FreeBSD is actually working on implementing Secure Boot right now,
according to their website:

https://wiki.freebsd.org/SecureBoot

Not sure how easy it would be to port to OpenBSD once completed. AFAIK,
OpenBSD doesn't even support UEFI.

I doubt it will ever be possible to really protect a system from an offline
physical access attack. Even if the entire system disk were protected with
an authenticated cipher like AES-GCM, there are plenty of firmware based
attacks that you can prep to launch as soon as the system is turned back on.

Re: GRE over IPv6 not supported by gre(4)?

[Joe Crivello]

I was contacted directly by someone who wanted to suggest that I try gif(4)
instead. I have tried gif(4) in the past, but I ran into a lot of weirdness
when using it in combination with IPsec (on 5.6-stable). After I shared my
experience, he suggested that I send my synopsis to the list... so here it
is.

Be forewarned... this is all from memory and it is from a while ago, so
some details might be slightly inaccurate. If a developer is interested I
will reproduce these scenarios again and report back with exact steps.

A few examples:

1. Consider a scenario where gif0 on a theoretical host is receiving an
ICMP echo destined for its inside address, while the ipencap traffic is
protected by a transport mode IPsec SA. From what I observed with tcpdump,
the encapsulated echo never appears to arrive on either enc0 or gif0, the
decapsulated echo arrives on enc0, and nothing at all arrives on gif0. The
reply appears to be sent on gif0, and then the encapsulated reply is sent
on enc0 (so that side is ok). I observed this while looking into some
really weird PF rule behavior that was clearly not right.

2. For a gif(4) device with an IPv6 inside address and an IPv4 outside
address, while the ipencap traffic is protected by a transport mode IPsec
SA, my host would respond to one IPv6 echo (destined to the gif inside
address) in a sequence of them, and no more. If the gif tunnel was not
protected by IPsec this worked totally fine. Honestly I have no idea why
this was happening, could have been something else with my configuration
but it fits into a pattern of weirdness.

3. For a gif(4) device with IPv4  IPv6 inside addresses and an IPv6
outside address, while the ipencap traffic is protected by a transport mode
IPsec SA, only one inside address or the other would work at a time.

4. Consider a situation in which I create two PF rules pass in on enc0
proto ipencap keep state (if-bound) and pass out on enc0 proto ipencap
keep state (if-bound) (also with appropriate address filters in both
directions). This appears to pass the gif traffic as expected. However, if
I suddenly delete the IPsec SA while the gif tunnel is being used, causing
the gif traffic to suddenly come in on a different interface (other than
enc0), PF continues to pass all the gif traffic until I run pfctl -F
state (WTF?).

For whatever reason, I just could not get gif(4) to play nicely with IPsec
on my 5.6-stable boxes. I should note that it appeared to work fine in the
absence of an IPsec SA, though. I spent a lot of time trying to diagnose
the problem, and then eventually just gave up and reverted to gre(4), which
resolved all the issues I identified.

-Joe

GRE over IPv6 not supported by gre(4)?

[Joe Crivello]

It looks like the gre(4) pseudo-device doesn't support GRE over IPv6:

# ifconfig gre1 tunnel fe80::1%vlan9 fe80::2%vlan9
ifconfig: SIOCSLIFPHYADDR: Address family not supported by protocol family

I believe this is at least within the realm of possibility, and it looks
like someone has done some work on this problem in the Linux world / it is
already supported there:

http://lwn.net/Articles/508786/

I understand that this has no relevance to the current state of affairs
when it comes to OpenBSD, but I just wanted to provide some evidence of
feasibility.

Are there any OpenBSD developers out there that would be interested in
working on this in exchange for a donation of equipment or money?

-Joe

Re: iked and isakmpd in parallel

[Joe Crivello]

We also have a need for this in our environment.

We use transport mode IPSEC to protect gif(4) tunnels between our OpenBSD
routers at our remote sites, and we would also ideally like one of these
routers to act as a Win 7 road warrior IKEv2 gateway. We would just use
iked for both scenarios, but as of 5.6, iked doesn't appear to support
transport mode yet. We have also tried running isakmpd and iked
side-by-side, but we have been unsuccessful in doing so. IIRC, when one
daemon starts after the other it wipes out the other's SAs and encap routes.

-Joe

On Tue, Jan 20, 2015 at 9:17 AM, Fedor Piecka teplav...@gmail.com wrote:

 Hello

 We need to support both IKEv1 and IKEv2 peers in our environment.

 Isakmpd.conf supports Listen-on directive.

 However I haven't found such a thing in iked.conf an iked manual pages.


 My first question now is how to instruct iked to listen only on a selected
 interface.

 The second question is whether the 2 of them wouldn't interfere with each
 other.

 Regards
 Fedor

Re: Problem With Default Route Over IPSEC Site-To-Site Tunnel VPN

[Joe Crivello]

/32  0 0
172.16.5.1/esp/require/out

On Tue, Dec 16, 2014 at 11:05 AM, Joseph Crivello josephcrive...@gmail.com
wrote:

 I was able to resolve my own problem.

 The solution I found was to create a gre tunnel between the two routers
 using the gre(4) pseudo-device. Once I had that working, I used IPSEC
 transport mode to protect the GRE tunnel. This method eliminates all
 default
 encap routes from both routers; which was the root cause of the problem.

 -Joe

 -Original Message-
 From: Joe Crivello [mailto:josephcrive...@gmail.com]
 Sent: Monday, December 15, 2014 2:24 PM
 Subject: Problem With Default Route Over IPSEC Site-To-Site Tunnel VPN

 Hello,

 I am having a problem with a particular aspect of my attempt to establish
 an
 IPSEC site-to-site tunnel between two gateways using ISAKMPD/IKEv1. I seem
 to be doing something wrong, but I have exhausted all of the resources that
 I know of in my quest to fix the problem (MAN pages, OpenBSD.org FAQ,
 Google, etc). I am hoping that someone with more OpenBSD experience than
 myself will be able to help me... either way, thanks so much for your time!

 The routers in question both run OpenBSD 5.6, situated at either end of
 long
 range wifi bridge link. Router 1 also has a interface connecting to an
 ISP
 router, which provides a route to the Internet:

 Internet == Router 1
   172.16.5.1
  ||
 Wifi
  ||
   172.16.5.2
   Router 2 == Local Networks
 (172.16.6.1/24, 172.16.7.1/24)

 The intention is to establish an IPSEC tunnel between Router 1 and Router
 2,
 over which Router 2 should send all traffic not destined for one of it's
 local networks. Accordingly, I set the default route of Router 2 to
 172.16.5.1, and I configured the tunnel like so:

 ## Router 1

 ike passive esp \
 from any to { 172.16.5.2/32, 172.16.6.0/24, 172.16.7.0/24 } \
 local 172.16.5.1 peer 172.16.5.2 \
 main auth hmac-sha2-512 enc aes-256 group modp2048 \
 quick auth hmac-sha2-512 enc aes-256-ctr group modp2048 \
 srcid SNIP: Router 1 \
 dstid SNIP: Router 2

 ## Router 2

 ike active esp \
 from { 172.16.5.2/32, 172.16.6.0/24, 172.16.7.0/24 } to any \
 local 172.16.5.2 peer 172.16.5.1 \
 main auth hmac-sha2-512 enc aes-256 group modp2048 \
 quick auth hmac-sha2-512 enc aes-256-ctr group modp2048 \
 srcid SNIP: Router 2 \
 dstid SNIP: Router 1

 This configuration (correctly) causes six SAs to be established:

 ## Router 1

 # ipsecctl -sa
 FLOWS:
 flow esp in from 172.16.5.2 to 0.0.0.0/0 peer 172.16.5.2 srcid SNIP:
 Router 1 dstid SNIP: Router 2 type use flow esp out from 0.0.0.0/0 to
 172.16.5.2 peer 172.16.5.2 srcid SNIP:
 Router 1 dstid SNIP: Router 2 type require flow esp in from
 172.16.7.0/24
 to 0.0.0.0/0 peer 172.16.5.2 srcid SNIP:
 Router 1 dstid SNIP: Router 2 type use flow esp out from 0.0.0.0/0 to
 172.16.7.0/24 peer 172.16.5.2 srcid SNIP:
 Router 1 dstid SNIP: Router 2 type require flow esp in from
 172.16.6.0/24
 to 0.0.0.0/0 peer 172.16.5.2 srcid SNIP:
 Router 1 dstid SNIP: Router 2 type use flow esp out from 0.0.0.0/0 to
 172.16.6.0/24 peer 172.16.5.2 srcid SNIP:
 Router 1 dstid SNIP: Router 2 type require

 SAD:
 esp tunnel from 172.16.5.2 to 172.16.5.1 spi 0x0eec4a02 auth hmac-sha2-512
 enc aes-256-ctr esp tunnel from 172.16.5.1 to 172.16.5.2 spi 0x1cde0906
 auth
 hmac-sha2-512 enc aes-256-ctr esp tunnel from 172.16.5.2 to 172.16.5.1 spi
 0x6769c99e auth hmac-sha2-512 enc aes-256-ctr esp tunnel from 172.16.5.2 to
 172.16.5.1 spi 0xad29e69c auth hmac-sha2-512 enc aes-256-ctr esp tunnel
 from
 172.16.5.1 to 172.16.5.2 spi 0xaf8c3502 auth hmac-sha2-512 enc aes-256-ctr
 esp tunnel from 172.16.5.1 to 172.16.5.2 spi 0xcdad877e auth hmac-sha2-512
 enc aes-256-ctr

 ## Router 2

 # ipsecctl -sa
 FLOWS:
 flow esp in from 0.0.0.0/0 to 172.16.5.2 peer 172.16.5.1 srcid SNIP:
 Router 2 dstid SNIP: Router 1 type use flow esp out from 172.16.5.2 to
 0.0.0.0/0 peer 172.16.5.1 srcid SNIP:
 Router 2 dstid SNIP: Router 1 type require flow esp in from 0.0.0.0/0
 to
 172.16.7.0/24 peer 172.16.5.1 srcid SNIP:
 Router 2 dstid SNIP: Router 1 type use flow esp out from 172.16.7.0/24
 to
 0.0.0.0/0 peer 172.16.5.1 srcid SNIP:
 Router 2 dstid SNIP: Router 1 type require flow esp in from 0.0.0.0/0
 to
 172.16.6.0/24 peer 172.16.5.1 srcid SNIP:
 Router 2 dstid SNIP: Router 1 type use flow esp out from 172.16.6.0/24
 to
 0.0.0.0/0 peer 172.16.5.1 srcid SNIP:
 Router 2 dstid SNIP: Router 1 type require

 SAD:
 esp tunnel from 172.16.5.2 to 172.16.5.1 spi 0x0eec4a02 auth hmac-sha2-512
 enc aes-256-ctr esp tunnel from 172.16.5.1 to 172.16.5.2 spi 0x1cde0906
 auth
 hmac-sha2-512 enc aes-256-ctr esp tunnel from 172.16.5.2 to 172.16.5.1 spi
 0x6769c99e auth hmac-sha2-512 enc aes-256-ctr esp tunnel from 172.16.5.2 to
 172.16.5.1 spi 0xad29e69c auth hmac-sha2-512 enc aes-256-ctr esp

Problem With Default Route Over IPSEC Site-To-Site Tunnel VPN

[Joe Crivello]

.

For example, if I try to ping 172.16.5.21 (the wireless bridge local to
Router 2) from Router 2, then it should send ICMP echo requests out over
the local interface connecting to the wireless bridge. This works fine
before ISAKMPD establishes the tunnel. After the tunnel is established, it
starts encapsulating the ICMP echo packets and sending them down the
tunnel, where they get blackholed:

# ping 172.16.5.21
PING 172.16.5.21 (172.16.5.21): 56 data bytes
--- 172.16.5.21 ping statistics ---
11 packets transmitted, 0 packets received, 100.0% packet loss

# tcpdump -nvvvpi enc0
tcpdump: listening on enc0, link-type ENC
tcpdump: WARNING: compensating for unaligned libpcap packets
SNIP
13:21:13.960188 (authentic,confidential): SPI 0x67eec1f6: 172.16.5.2 
172.16.5.1: 172.16.5.2  172.16.5.21: icmp: echo request (id:1370 seq:0)
[icmp cksum ok] (ttl 255, id 35507, len 84) (ttl 255, id 35507, len 84)
SNIP
13:21:14.960396 (authentic,confidential): SPI 0x67eec1f6: 172.16.5.2 
172.16.5.1: 172.16.5.2  172.16.5.21: icmp: echo request (id:1370 seq:1)
[icmp cksum ok] (ttl 255, id 39619, len 84) (ttl 64, id 24561, len 104)
SNIP
^C
65 packets received by filter
0 packets dropped by kernel

So, I think I understand the problem, but I don't have any idea what the
solution would be. I tried adding manual bypass flows to the ipsec.conf
of Router 2, to no avail:

flow esp from 172.16.5.0/24 to 172.16.5.0/24 type bypass
flow esp from 172.16.6.0/24 to 172.16.6.0/24 type bypass
flow esp from 172.16.7.0/24 to 172.16.7.0/24 type bypass

What is the correct way to implement this scenario, while still directing
the default route over the IPSEC tunnel?

Thanks so much...!

Joe Crivello