Hello,
I've observed some interesting behaviour when using static-port with
af-to rules.
I may be missing something obvious and have somehow misconfigured
something, but this smells fishy to me, so I thought I'd report it.
I've recently set up an IPv6-mostly network and wanted cone NAT
behaviour to allow things like STUN for FaceTime to work.
In the man pages it states that "static-port" is for use with nat-to
rules, but it turns out it is also accepted by af-to, and largely works.
The interesting behaviour I've noticed is that when static-port is used
in conjunction with af-to rules the initial packet of a connection
attempt to the NAT64 prefix is leaked out the egress interface. No such
behaviour is observed when the static-port option is not used.
Steps to reproduce:
1) Create af-to rule with static-port option
2) Notice how af-to rule misses the first packet / connection attempt
and connection times out
3) Notice how af-to works as expected for subsequent packets
Heres the PF rules I'm using to implement NAT64:
...
pass in on $lan_if inet6 from !(egress:network) to 64:ff9b::/96 af-to
inet from (egress:0) keep state
# Use static port for UDP NAT64 traffic so that STUN for video calling
and gaming etc works.
pass in on $lan_if inet6 proto udp from !(egress:network) to
64:ff9b::/96 af-to inet from (egress:0) static-port keep state
To try and illustrate the issue, here are some tcpdump outputs from
using dig to make a DNS query to 1.1.1.1 using IPv6 mapped format
64:ff9b::101:101.
thinkpad$ time dig @64:ff9b::101:101 +short google.ca
142.250.73.99
0m05.04s real 0m00.00s user 0m00.00s system
That took 5 seconds, that's a super long time!
Lets run tcpdump on the router and see whats going on:
Egress:
er8pro$ doas tcpdump -lni cnmac4 port domain
...
15:06:28.255337 2006:420:69:7:254e:c5cd:b280:7e0c.23782 >
64:ff9b::101:101.53: 63837+ [1au] A? google.ca.(38)
15:06:33.254922 55.44.33.161.23782 > 1.1.1.1.53: 63837+ [1au] A?
google.ca.(38) (DF)
15:06:33.267623 1.1.1.1.53 > 55.44.33.161.23782: 63837 1/0/1 A
142.250.73.99(54) (DF)
And here's how it looks from the LAN side:
er8pro$ doas tcpdump -lni cnmac5 port domain
...
15:13:20.331256 2006:420:69:7:254e:c5cd:b280:7e0c.48398 >
64:ff9b::101:101.53: 65486+ [1au] A? google.ca.(38)
15:13:25.327384 2006:420:69:7:254e:c5cd:b280:7e0c.48398 >
64:ff9b::101:101.53: 65486+ [1au] A? google.ca.(38)
15:13:25.341485 64:ff9b::101:101.53 >
2006:420:69:7:254e:c5cd:b280:7e0c.48398: 65486 1/0/1 A 142.250.73.67(54)
Here we can see the initial connection attempt to 64:ff9b::101:101 is
somehow "missed" by the af-to rule and leaks out onto egress and
inevitably times out. The subsequent connection attempt 5 seconds later
succeeds however.
To be clear, the behaviour only presents itself when using static-port
in conjunction with af-to rules. As soon as af-to rule has static-port
option removed, no such "leaks" occur. I know I could just make a pf
rule to explicitly block outgoing traffic to the NAT64 prefix, but it
doesn't change the fact that for some reason the af-to rule "misses" the
first connection attempt and forces a connection timeout if static-port
option is set.
I hope this makes sense. I know using af-to with static-port isn't
explicitly documented, but I think having support for "cone" NAT64 is a
pretty killer feature and is necessary to facilitate many if not most
peer to peer communications for IPv4. Living in rural Canada it's quite
inefficient to have my packets destined for the Yukon or Labrador get
there by way of Apple TURN servers in Cupertino, so for me at least, its
a hugely appreciated functionality.
I know that this email is getting long, but in case some more examples
are desired, here is the NAT64 prefix leaking on egress when attempting
a Facetime call and also mDNS queries leaking from my AndroidTV box that
seems to get confused and sends out mDNS queries from its CLAT.
When attempting to make a FaceTime call to a relative whose on an IPv4
only connection, the initial connection attempts are missed by af-to
rule, but on subsequent retries succeeds.
# Initial Connection Attempts leak
14:10:34.061299 2006:420:69:7:d934:b317:7d73:736b.16393 >
64:ff9b::11bc:b219.3490: udp 172 [flowlabel 0xf10d4]
14:10:43.275992 2006:420:69:7:d934:b317:7d73:736b.16393 >
64:ff9b::2a1f:1413.16393: udp 72 [flowlabel 0xd8709]
...
# After connection is reattempted af-to does the needful and starts
translating as expected.
14:11:16.309508 55.44.33.161.16393 > 42.31.20.19.16393: udp 303
14:11:16.313229 42.31.20.19.16393 > 55.44.33.161.16393: udp 1050
14:11:16.317260 42.31.20.19.16393 > 55.44.33.161.16393: udp 1047
14:11:16.321257 42.31.20.19.16393 > 55.44.33.161.16393: udp 1049
14:11:16.321259 42.31.20.19.16393 > 55.44.33.161.16393: udp 1049
# Android TV with CLAT leaking
14:12:12.422220 2006:420:69:7:34fb:e138:ba30:c741.5353 >
64:ff9b::e000:fb.5353: 0[|domain]
14:12:12.571486 2006:420:69:7:ef5d:8bb9:202e:fc60.5353 >
64:ff9b::e000:fb.5353: 0[|domain]
14:13:48.965738 2006:420:69:7:34fb:e138:ba30:c741.5353 >
64:ff9b::e000:fb.5353: 0 PTR? _googlecast._tcp.local.(40
Please let me know if there are any more details etc I can provide.
Regards,
Jordan
