Re: pf af-to silently dropping packets that are too big after translation
Hello, On Mon, Sep 02, 2024 at 11:44:24PM +, Jason Healy wrote: > > Just to double-check, is the MTU in this case (1300) for an IPv6 link? > If so, should the MTU be lowered when translated back to IPv4 (20 > bytes smaller) to account for the extra bloat of the headers? Or are > we already back to IPv4 by the time we hit RT? I'm having a little > trouble understanding the topology. pf(4) does exactly that. it adjusts MTU for difference of header sizes between IPv6 and IPv4. Snippet below comes from pf_translate_icmp_af(). 3061 break; 3062 case ICMP6_PACKET_TOO_BIG: 3063 type = ICMP_UNREACH; 3064 code = ICMP_UNREACH_NEEDFRAG; 3065 mtu -= 20; 3066 break; firewall shrinks reported MTU by 20 bytes/octets when it translates ICMPv6 too-big message to ICMPv4 need-frag. > > >I understand what are you trying to do. I'm afraid I will need > >output of ifconfig and pf.conf. Taking a brief look into your source > >code I can see icmp6_reflect() is aware of routing domains, so I would > >assume things should work too. > > I will send this under separate cover as a direct message to you. > I will try to look at it sometimes later. regards sashan
Re: pf af-to silently dropping packets that are too big after translation
Sashan, I really appreciate your deep dive into this! Your explanation makes sense; if PF is using the local state to translate the ICMP errors back to IPv4, I can see how it would have to use the addresses available to it from the translation, so the source of the packet is reasonable in this case. My only question about the second case is the MTU reported in the packet-too-big message: >If too-big error will be coming from host RT and error will >will match state created by af-to rule, then firewall will >use IPv4 address from state (a.k.a. destination IP from packet >sent by client). In that case client will see too-big error as >coming from destination host. looks odd but there is nothing >we can do about it. I understand and this makes sense; PF can't report a source IPv6 address that does not have an explicit mapping since that can't be represented in IPv4. Just to double-check, is the MTU in this case (1300) for an IPv6 link? If so, should the MTU be lowered when translated back to IPv4 (20 bytes smaller) to account for the extra bloat of the headers? Or are we already back to IPv4 by the time we hit RT? I'm having a little trouble understanding the topology. >I understand what are you trying to do. I'm afraid I will need >output of ifconfig and pf.conf. Taking a brief look into your source >code I can see icmp6_reflect() is aware of routing domains, so I would >assume things should work too. I will send this under separate cover as a direct message to you. Thanks again, Jason
Re: pf af-to silently dropping packets that are too big after translation
Hello,
On Fri, Aug 30, 2024 at 08:12:52PM +, Jason Healy wrote:
> Good afternoon,
>
> Thank you for your analysis, and my apologies for the slow reply. I have
no problem at all.
> Your first packet capture (icmp.pkt) contained packet-too-big messages like
> this:
>
> > 10.188.210.10 > 10.188.211.123: icmp: echo request (DF)
> > 10.188.211.123 > 10.188.210.10: icmp: \
> > 10.188.211.123 unreachable - need to frag (mtu 1480) (DF)
>
> The source of the packet-too-big is the target of the ping (10.188.211.123)
> and not the PF router (10.188.210.50; not explicitly stated but taken from
> your topology document). So while a packet-too-big was received, it was not
> generated by the router itself (is the link MTU between PF and the test box
> greater than 1500?)
>
The things are more subtle here. The short answer is the ICMP error comes
from
firewall (PF box). Let me explain what's happening. Referring to topology
found in OpenBSD regress [1] we need to look at PF box more closely.
in <-> out
10.188.210.50 10.188.211.50
fdd7:e83e:66bc:210:5054:ff:fe12:3450
fdd7:e83e:66bc:211:5054:ff:fe12:3450
the PF acts as a router for 10.188.211.0/24 subnet. There is also
rule does AF-translation for packets with 10.188.211.123 destination
to fdd7:e83e:66bc:211:5054:ff:fe12:3451 (IPv6 address of RT box)
ICMP echo-request to 10.188.211.123 enters PF at in interface.
It is inbound packet which matches af-to rule. PF creates state and
changes the destination IP address (and family) to
fdd7:e83e:66bc:211:5054:ff:fe12:3451.
The source address comes from rule. The rule uses IPv6 address of out
interface (fdd7:e83e:66bc:211:5054:ff:fe12:3450).
the firewall then injects the packet to IP stack to route it, line 8004
here:
7984 ip_forward(pd.m, ifp, NULL, flags);
7985 } else
7986 ip_output(pd.m, NULL, NULL, 0, NULL, NULL,
0);
7987 break;
7988 case AF_INET6:
7989 if (pd.dir == PF_IN) {
7990 int flags = IPV6_REDIRECT;
7991
7992 switch (atomic_load_int(&ip6_forwarding)) {
7993 case 2:
7994 SET(flags, IPV6_FORWARDING_IPSEC);
7995 /* FALLTHROUGH */
7996 case 1:
7997 SET(flags, IPV6_FORWARDING);
7998 break;
7999 default:
8000 ip6stat_inc(ip6s_cantforward);
8001 action = PF_DROP;
8002 goto out;
8003 }
8004 ip6_forward(pd.m, NULL, flags);
8005 } else
8006 ip6_output(pd.m, NULL, NULL, 0, NULL,
NULL);
8007 break;
note: call to ip6_forward() happens for AF-translated packets only.
pf_test() caller sees packet as discarded.
ip6_forward() finds route (xmite interface) for packet with destination
fdd7:e83e:66bc:211:5054:ff:fe12:3451. As it attempts to transmit the packet
it finds out the packet does not fit to wire.
It uses the icmp6_error()/icmp6_reflect() to generate ICMPv6 error.
The ICMPv6 packet looks like this:
fdd7:e83e:66bc:211:5054:ff:fe12:3451 ->
fdd7:e83e:66bc:211:5054:ff:fe12:3450
(IPv6 address of in @ RT) (IPv6 address of out @ PF)
The packet is injected to IP stack on PF box by task. Firewall intercepts
such packet as outbound. The first thing firewall does it tries to find
matching state. It finds the state created by ICMP request. Found state also
indicates packet needs to be translated back to IPv4.
The address 10.188.211.123 comes from state.
I agree the translation can be more smart here because the mbuf which
represents packet should still keep PF_TAG_GENERATED flag. Firewall
can use it to distinguish local ICMP errors from those which come
from remote boxes. If PF_TAG_GENERATED flag is present then firewall
should pick up source address from local interface instead of using
IPv4 address found in state. But this is a detail because for
remote hosts firewall would have to use IPv4 address which comes
state anyway.
[1]
https://github.com/openbsd/src/blob/master/regress/sys/net/pf_forward/Makefile
> Your second packet capture (icmp-eco.pkt) contained packet-too-big messages
> like this:
>
> > 10.188.210.10 > 10.188.212.52: icmp: echo request (DF)
> > 10.188.211.51 > 10.188.210.10: icmp: \
> >
Re: pf af-to silently dropping packets that are too big after translation
Good afternoon, Thank you for your analysis, and my apologies for the slow reply. I have some questions about your results, and then some additional information from my environment. Your first packet capture (icmp.pkt) contained packet-too-big messages like this: > 10.188.210.10 > 10.188.211.123: icmp: echo request (DF) > 10.188.211.123 > 10.188.210.10: icmp: \ > 10.188.211.123 unreachable - need to frag (mtu 1480) (DF) The source of the packet-too-big is the target of the ping (10.188.211.123) and not the PF router (10.188.210.50; not explicitly stated but taken from your topology document). So while a packet-too-big was received, it was not generated by the router itself (is the link MTU between PF and the test box greater than 1500?) Your second packet capture (icmp-eco.pkt) contained packet-too-big messages like this: > 10.188.210.10 > 10.188.212.52: icmp: echo request (DF) > 10.188.211.51 > 10.188.210.10: icmp: \ > 10.188.212.52 unreachable - need to frag (mtu 1300) Here again we have received a packet-too-big message, but it wasn't generated by the PF box but rather by RT (which has the 1300 MTU). I'm interested in the case where the PF box is where the MTU shift occurs (due to the larger headers of IPv6), and so it must generate the error rather than just forwarding one from upstream. Thus, I'm not sure if PF is actually generating packet-too-big messages, as your tests don't show it. This is consistent with my testing, as I am not seeing any messages from PF, and there are no other hosts that could generate them since the packets are too big to put on the wire. Independent of this, I wanted to provide some additional information about my environment, as it is not as simple as the test environment. Our setup makes use of rdomains, which I did not include in the original ticket, but realize now does make for a different setup. I'll try to define the topology: PF box has three physical interfaces in use: em0 (member of trunk0) em1 (member of trunk0) em2 (management interface) em0/em1 are bonded using trunk(4) into interface trunk0 trunk0 is connected to an upstream switch with tagged VLANs enabled, so we create vlan(4) interfaces on top of trunk0 For the purposes of this bug, we will deal with a single vlan, vlan42 vlan42 has both an IPv4 and IPv6 address assigned to it. Our intent is to use it as a kind of "hairpin CLAT"; IPv4 packets are received on vlan42's IPv4 interface, af-translated by PF, and emitted back on vlan42 as IPv6 packets. The default router on vlan42 (not managed by OpenBSD) will forward the packets to our NAT64 box and eventual delivery. To isolate VLANs from each other, each vlan interface is put into its own rtable (42, in this case). Under this setup, we do not see any ICMPv4 packet-too-big messages. We have attempted packet captures on both vlan42 and on em2 (management), but have not seen any ICMP codes. We even went so far as to add a IPv4 address and default route to em2 in case PF was sending them via the default rtable instead of the one assigned to the incoming interface, but nothing there either. We have a "pass out" default rule in our pf.conf, so I do not believe we are preventing any generated packets from leaving the box. I can provide full PF rules and network topology if necessary, but I think we should debug your test network case first to see if PF actually will generate a packet-too-big message before we move on to anything more complicated. Thanks, Jason -- . Jason Healy Director of Technology . Suffield Academy Opatrny Chair in Technology . . http://web.suffieldacademy.org/~jhealy
Re: pf af-to silently dropping packets that are too big after translation
Hello, I did check things with tpcdump. And it looks like pf(4) is doing the right thing. let me explain. To play with pf(4) I'm using same network set up as we use for regression testing [1] [1] https://github.com/openbsd/src/blob/master/regress/sys/net/pf_forward/Makefile > >How-To-Repeat: > Load a rule in PF that translates IPv4 to IPv6: > > pass in inet af-to inet6 from 2001:db8::1 to 64:ff9b::/96 > on host pf I'm running firewall with rule as follows: pass in on vio1 inet from 10.188.210.0/24 to 10.188.211.123 flags S/SA \ af-to inet6 from fdd7:e83e:66bc:211:5054:ff:fe12:3450 to \ fdd7:e83e:66bc:211:5054:ff:fe12:3451 > On an IPv4 client machine with the OpenBSD box as its gateway, > generate a packet that's too big to transmit once it's translated: > > ping -D -s 1472 8.8.8.8 I start test trying to reach host rt on its IPv4 address. I start with payload size 1472: (1) src$ ping -D -s 1472 10.188.211.51 PING 10.188.211.51 (10.188.211.51): 1472 data bytes 1480 bytes from 10.188.211.51: icmp_seq=0 ttl=254 time=3.234 ms ^C --- 10.188.211.51 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 3.234/3.234/3.234/0.000 ms now repeat with payload size 1473: (2) src$ ping -D -s 1473 10.188.211.51 PING 10.188.211.51 (10.188.211.51): 1473 data bytes ping: sendmsg: Message too long ping: wrote 10.188.211.51 1501 chars, ret=-1 ^C --- 10.188.211.51 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss ping command reports error. so far so good. Let's get the af-to rule into action now. To do it we need ping to reach address 10.188.211.123 which gets translated to IPv6 fdd7:e83e:66bc:211:5054:ff:fe12:3451. This time I start with payload size 1450: (3) src$ ping -D -s 1450 10.188.211.123 PING 10.188.211.123 (10.188.211.123): 1450 data bytes 1458 bytes from 10.188.211.123: icmp_seq=0 ttl=63 time=2.825 ms 1458 bytes from 10.188.211.123: icmp_seq=1 ttl=63 time=1.597 ms ^C --- 10.188.211.123 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 1.597/2.211/2.825/0.614 ms the next attempt goes with payload size 1472: (4) src$ ping -D -s 1472 10.188.211.123 PING 10.188.211.123 (10.188.211.123): 1472 data bytes ^C --- 10.188.211.123 ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet loss Indeed, unlike the case (2), there is no error reply. Let's see what happens on wire, while ping is running. src# tcpdump -i vio1 tcpdump: listening on vio1, link-type EN10MB 12:14:09.651371 10.188.210.10 > 10.188.211.123: icmp: echo request (DF) 12:14:09.652008 10.188.211.123 > 10.188.210.10: icmp: \ 10.188.211.123 unreachable - need to frag (mtu 1480) (DF) 12:14:10.654563 10.188.210.10 > 10.188.211.123: icmp: echo request (DF) 12:14:10.655747 10.188.211.123 > 10.188.210.10: icmp: \ 10.188.211.123 unreachable - need to frag (mtu 1480) (DF) 12:14:11.654456 10.188.210.10 > 10.188.211.123: icmp: echo request (DF) 12:14:11.655567 10.188.211.123 > 10.188.210.10: icmp: \ 10.188.211.123 unreachable - need to frag (mtu 1480) (DF) so there actually is ICMP error message for ICMP probe. Taking a closer look at those packets I could see the TTL in IP header is 0 which itself is odd and requires further investigation. Packet dump is attached in icmp.pkt file. there is a one more test to be done to see if ping(8) command is able to process TOOBIG icmp error it receives. Note that error in case (2) comes from local IP stack when ICMP probe with large payload is dispatched to wire. For this test I'm going to ping host eco. The MTU between host eco and rt is set to 1300. src$ ping -D -s 1293 10.188.212.52 PING 10.188.212.52 (10.188.212.52): 1293 data bytes ^C --- 10.188.212.52 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss the attached icmp-eco.pkt shows regular ICMP too big messages which come from host rt. Looks like ip stack at host src fails to indicate error to ping(8) command when those ICMP error are received. I have not looked at ping(8)/ip stack why this happens. thanks and regards sashan icmp.pkt Description: Binary data icmp-eco.pkt Description: Binary data
