Short version:
I'm trying to set up a "road warrior"-style VPN like the one described
at https://www.openbsd.org/faq/faq17.html but I'm trying to use IPv6 so
I can have globally-routable addresses (so I'm not using NAT). So far
I've gotten the initiator and the responder to set up a security
association (to be precise, according to "ipsecctl -sa" the
initiator/client sees one SA and zero flows, and the responder/server
sees two SA's and two flows). When I try to ping the client's virtual
IP address from the server, tcpdump running on the egress interfaces of
both the client and the server see appropriately-sized "esp" packets
going from the server to the client as expected, but tcpdump doesn't
see any traffic emerging on the "lo1" loopback interface the FAQ told
me to create, nor does it see any evidence of ping-responses being
sent. When I try to ping from the client to the server, I get
"ping6: sendmsg: Permission denied".
Longer version:
The client is running -current and the server is running 7.3.
The server is a VPS hosted in a datacenter somewhere, and my VPS
provider gives me a whole /64 subnet to work with. My VPS provider
doesn't appear to look at neighbor discovery traffic; it routes all
traffic in that /64 to my VirtIO interface, regardless of which IP
address it is, which seems like a handy feature for what I'm trying to
do.
Here is what I want to do: the client should open an IPsec tunnel to
the server. The client should request an IPv6 address from the server's
enormous /64 subnet. Then, applications running on the client should
have the option to use that "virtual" IPv6 address as though it was
available on a local interface on the client machine. The client would
effectively be multihosted: in addition to its regular physical
Internet connection it would also have this IPsec tunnel that also
connects to the global Internet. My plan right now is to use rdomains
to control which applications use the tunnel and which ones use the
regular egress interface, but I'm not attached to that specific
approach.
The application should be obvious: I'm trying to set up a region-faking
VPN, so that I can have applications appear to connect to the Internet
from different continents.
Both client and server are using the same pf.conf that is installed by
default. I tried adding "pass on enc0" to the client's pf.conf and
"pass on enc0 tagged ROADW" to the server's, but it hasn't made a
difference because (I assume) my existing pf.conf rules weren't
blocking much to begin with.
Here are my config files, with IP addresses and server names changed.
But basically you should see the client at 2001:db8:1::1 connecting to
the server at 2001:db8:2::1 and requesting a random IP address in the
server's subnet, e.g. 2001:db8:2::485b:4ac7. I have also put the
appropriate keys in /etc/iked/pubkeys/fqdn/.
=== Client-side configuration ===
$ cat /etc/hostname.lo1
rdomain 1
$ cat /etc/iked.conf
ikev2 'client_config' active esp \
rdomain 1 \
from dynamic to any \
local 2001:db8:1::1 \
peer 2001:db8:2::1 \
srcid client.example.org \
dstid server.example.org \
request address any \
iface lo1
=== Server-side configuration ===
$ cat /etc/sysctl.conf
net.inet6.ip6.forwarding=1
$ cat /etc/iked.conf
ikev2 'server_config' passive esp \
from any to dynamic \
local 2001:db8:2::1 \
peer 2001:db8:1::1 \
srcid server.example.org \
config address 2001:db8:2::/64 \
tag "ROADW"
=== End of configurations ===
Does it work? Sort of. Here's what I see on the server:
=== Checking status on the server ===
# ipsecctl -sa
FLOWS:
flow esp in from 2001:db8:2::485b:4ac7 to ::/0 peer 2001:db8:1::1 srcid
FQDN/server.example.org dstid FQDN/client.example.org type require
flow esp out from ::/0 to 2001:db8:2::485b:4ac7 peer 2001:db8:1::1 srcid
FQDN/server.example.org dstid FQDN/client.example.org type require
SAD:
esp tunnel from 2001:db8:2::1 to 2001:db8:1::1 spi 0x001a3754 enc aes-128-gcm
esp tunnel from 2001:db8:1::1 to 2001:cb8:2::1 spi 0x7634a3b6 enc aes-128-gcm
$ route show
Internet6:
Destination Gateway Flags Refs Use Mtu Prio Iface
default fe80::1234%vio0 UGS 8 578 - 8 vio0
::/96 localhost UGRS 0 0 32768 8 lo0
localhost localhost UHhl 10 90 32768 1 lo0
::ffff:0.0.0.0/96 localhost UGRS 0 0 32768 8 lo0
2001:db8:2::/64 server UCn 3 3630 - 4 vio0
2001:db8:2::485b:4ac7 link#1 UHLc 0 3633 - 3 vio0
...and a whole lot more
=== Checking status on the client ===
# ipsecctl -sa
FLOWS:
No flows
SAD:
esp tunnel from 2001:db8:2::1 to 2600:db8:1::1 spi 0x001a3754 enc aes-128-gcm
$ ifconfig lo1
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> rdomain 1 mtu 32768
index 4 priority 0 llprio 3
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo1 prefixlen 64 scopeid 0x4
inet6 2001:db8:2::485b:4ac7 prefixlen 128
$ route -T1 show
Routing tables
Internet6:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 2001:db8:2::485b:4ac7 UGS 0 1 32768 6 lo1
localhost localhost UHl 0 37 32768 1 lo1
2001:db8:2::485b:4ac7 2001:db8:2::485b:4ac7 UHhl 1 2 32768 1 lo1
fe80::1%lo1 fe80::1%lo1 UHl 0 2893 32768 1 lo1
ff01::%lo1/32 fe80::1%lo1 Um 0 12 32768 4 lo1
ff02::%lo1/32 fe80::1%lo1 Um 0 12 32768 4 lo1
[This is the complete routing table; I didn't delete any rows.]
====
We see that the server sees two security associations (one in each
direction) and two flows (also one in each direction), but the client
sees only one security association (in the server-to-client direction)
and no flows.
I trimmed a lot of irrelevant stuff out of the server's routing table
but it looks correct. The server recognizes itself as the gateway for
the entire 2001:db8:2::/64 subnet (though this has nothing to do with
the IPsec tunnel, and is just a statement that traffic on the local
subnet doesn't need to go through a router, and thus does not require
an external gateway). It also recognizes that traffic to the client's
virtual IP should be sent over "link#1", which I assume is the tunnel.
Meanwhile, the client's routing table seems wrong. The client
recognizes that it owns its own virtual IP address, and it recognizes
itself as its own default gateway. But it doesn't have any references
to "link#n" for any n. That perhaps makes sense bcause there is no
outward flow to route packets to.
We get three interesting results from pings:
=== Test #1 ===
If I run "ping6 2001:db8:2::485b:4ac7" from the server and watch the
egress interfaces on both the client and server and filter on "esp",
I see appropriately-sized packets with "spi 0x001a3754". (If I pass
the "-s" option to ping6, the sizes of the packets change.) If I run
"netstat -s -pesp" on both the client and the server, I see that the
server thinks it has output some number of ESP packets and received
none, and the client htinks it has received the same number of ESP
packets but sent none. So, OK, clearly we have a one-sided ping.
But when I run "tcpdump -ilo1" on the client I get nothing! The
encrypted ESP packets are coming in, and iked.conf says they're
supposed to emerge on lo1 (which ifconfig confirms exists and has
the expected automatically-assigned IP address) but the decrypted
ICMP packets are nowhere to be seen.
I get the same behavior if I run "nc -u 2001:db8:2::485b:4ac7 5000"
from the server---the only differences between this and ping6 are
that I'm not *expecting* any replies, but also I'm sending UDP
packets instead of ICMPv6 packets so it seems like they'd be more
likely to show up on the lo1 interface. But they don't. (I also
tried "tcpdump -ienc0" on the client with both tests but the packets
never show up anywhere.)
=== Test #2 ===
The client can't ping the server at all!
$ route -T1 exec ping6 2001:db8:2::1
PING 2001:db8:2::1 (2001:db8:2::1): 56 data bytes
ping6: sendmsg: Permission denied
ping: wrote mudge 64 chars, ret=-1
ping6: sendmsg: Permission denied
ping: wrote mudge 64 chars, ret=-1
Curiously, I get the same results if I try to have the client ping
itself at 2001:db8:2::485b:4ac7, but the client *can* ping itself
at both ::1 and fe80::1%lo1 from routing domain 1.
How in the world am I getting "Permission denied" when I ping
certain addresses? What does that even mean?
=== Test #3 ===
I know I'm getting ahead of myself, if I want this connection to be
usable for actual connections, the server is going to have to be able
to route traffic from *other* servers to the client. I happen to have
a second server handy, so I do the following:
On the original server:
$ tcpdump -l -ivio0 icmp6 | grep 4ac7
On the extra server:
$ ping6 2001:db8:2::485b:4ac7
What I see on tcpdump is that the original server is spamming its local
link with neighbor solicitation requests asking who has
2001:db8:2::485b:4ac7. That behavior strikes me as a little weird. When
the server originates a packet that is destined for the VPN IP address,
it correctly sends it over the IPsec tunnel (even though the reciever
of that packet appears to drop it on the floor). But when the server
receives a packet from someone else that is destined for that same VPN
IP address, the server forgets that this route exists and instead
thinks, "Hey, I recognize that subnet---I'll bet this address belongs
to someone else on the local link." Why the discrepancy? The server has
only one routing domain, so I would expect the same routing rules to
apply to both.
=== In summary ===
For all this mess of detail, I think I have only two problems:
1. Why does my client have only one security association and zero
flows? I suspect that this is the cause of all of my other issues
with the tunnel.
2. Why does the server route packets over the tunnel only when they
originate on the server itself? (The ip6.forwarding sysctl is
enabled---the problem is that it wants to route these packets
onto the local link, not that it refuses to route at all.)
Does anybody have any thoughts on what I'm doing wrong?
Thanks,
Anthony Coulter
