I was testing PF rules that use divert-packet with nat-to/rdr-to, and
found that a set of PF rules that work in OpenBSD 4.9 no longer work in
-current.
I tested with OpenBSD 4.9/i386 and the September 22, 2011 i386
snapshot. Their kern.version values are:
OpenBSD 4.9 (GENERIC) #671: Wed Mar 2 07:09:00 MST 2011
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC
OpenBSD 5.0-current (GENERIC) #60: Thu Sep 22 11:33:48 MDT 2011
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC
This is a long bug report, so I will start with a summary of the test
results:
Test Scenario OpenBSD 4.9 Sep 22, 2011 snap
------------------------- ----------- -----------------
A: Without NAT (Outbound) Success Success
B: Without NAT (Inbound) Success Success
C: With NAT (Outbound; nat-to) Success Failure
D: With NAT (Inbound; rdr-to) Success Failure
The rest of the bug report will describe the test network, the four
test scenarios, the test program used to read/reinject packets to/from
the divert socket, and the test results with OpenBSD 4.9 followed by
the Sep 22, 2011 snapshot.
======[ TEST NETWORK ]======
The test network consists of three nodes -- Outside, Firewall, and
Inside. It is set up within a VMware ESXi 4.1.0 environment as
follows.
+----------+
| Outside | OpenBSD 4.9/i386
+----+-----+
| 10.0.0.2
|
|
em0 | 10.0.0.1
+----+-----+
| Firewall |
+----+-----+
em1 | 192.168.1.1
|
|
| 192.168.1.2
+----+-----+
| Inside | OpenBSD 4.9/i386
+----------+
The device being tested is Firewall. The Outside and Inside nodes are
OpenBSD 4.9/i386 VMs that are used to send/receive traffic for the
tests.
======[ TEST SCENARIOS ]======
The tests were done by setting up Firewall with divert-packet PF rules
in four scenarios. The four scenarios were first tested with OpenBSD 4.9 on
the Firewall followed by the snapshot. The scenarios are:
Scenario A: Without NAT - Outbound
Scenario B: Without NAT - Inbound
Scenario C: With NAT - Outbound
Scenario D: With NAT - Inbound
In the Without NAT scenarios (A & B), the following PF rules were applied:
set skip on lo
pass # to establish keep-state
pass out on em0 divert-packet port 7000
pass in on em0 divert-packet port 7000
In the With NAT scenarios (C & D), the following PF rules were applied:
set skip on lo
pass # to establish keep-state
pass out on em0 divert-packet port 7000 nat-to (em0:0)
pass in on em0 inet proto tcp to (em0:0) port 13 divert-packet port 7000
rdr-to 192.168.1.2 port 13
In the Outbound scenarios, traffic was initiated from Inside to Outside by
running "ftp http://10.0.0.2/index.html"; on the Inside box to fetch a
file from the HTTP server running on Outside.
In the Inbound test, traffic was initiated from Outside to Inside.
This test consists of connecting to the daytime port (TCP port 13) on
the Inside box as follows:
(a) In the Without NAT scenarios, the connection was done directly from
the Outside box ("telnet 192.168.1.2 13");
(b) In the With NAT scenarios, the connection was made to the public IP
address of the Firewall ("telnet 10.0.0.1 13"), and the PF rdr-to rule
would forward that packet to the Inside box.
======[ TEST PROGRAM ]======
I wrote a very basic test program called div.c to read the packets from
the divert socket and reinject them back into the kernel. It does not
attempt to process the packets in any way.
--BEGIN--
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <netinet/ip_var.h>
#include <netinet/tcp.h>
#include <netinet/tcpip.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int
main(int argc, char *argv[])
{
int fd, s;
struct sockaddr_in sin;
socklen_t sin_len;
time_t now;
struct tm tres;
char buf[256] = "";
memset(&tres, 0, sizeof(tres));
time(&now);
localtime_r(&now, &tres);
memset(buf, 0, sizeof(buf));
fd = socket(AF_INET, SOCK_RAW, IPPROTO_DIVERT);
if (fd == -1) {
fprintf(stderr, "could not open divert socket\n");
exit(1);
}
memset(&sin, 0, sizeof(sin));
sin.sin_family = AF_INET;
sin.sin_port = htons(7000);
sin.sin_addr.s_addr = 0;
sin_len = sizeof(struct sockaddr_in);
s = bind(fd, (struct sockaddr *) &sin, sin_len);
if (s == -1) {
fprintf(stderr, "bind failed\n");
exit(1);
}
for (;;) {
ssize_t n;
char packet[131072];
struct ip *ip_hdr;
struct tcpiphdr *tcpip_hdr;
char src_ip[256], dst_ip[256];
memset(packet, 0, sizeof(packet));
n = recvfrom(fd, packet, sizeof(packet), 0, (struct sockaddr *)
&sin, &sin_len);
memset(&tres, 0, sizeof(tres));
time(&now);
localtime_r(&now, &tres);
memset(buf, 0, sizeof(buf));
strftime(buf, sizeof(buf), "%Y-%m-%d %H:%M:%S", &tres);
tcpip_hdr = (struct tcpiphdr *) packet;
ip_hdr = (struct ip *) packet;
memset(src_ip, 0, sizeof(src_ip));
memset(dst_ip, 0, sizeof(dst_ip));
if (inet_ntop(AF_INET, &ip_hdr->ip_src, src_ip, sizeof(src_ip))
== NULL) {
fprintf(stderr, "Invalid IPv4 source packet\n");
goto reinject;
}
if (inet_ntop(AF_INET, &ip_hdr->ip_dst, dst_ip, sizeof(dst_ip))
== NULL) {
fprintf(stderr, "Invalid IPv4 destination packet\n");
goto reinject;
}
fprintf(stderr, "[%s] IPv%u: %s:%u -> %s:%u\n",
buf,
ip_hdr->ip_v,
src_ip,
ntohs(tcpip_hdr->ti_sport),
dst_ip,
ntohs(tcpip_hdr->ti_dport)
);
reinject:
n = sendto(fd, packet, n, 0, (struct sockaddr *) &sin, sin_len);
}
return 0;
}
--END--
======[ TEST RESULTS WITH OPENBSD 4.9 ]======
The results using OpenBSD 4.9 on the Firewall are as follows:
---[ SCENARIO A: Without NAT - Outbound (OpenBSD 4.9) ]---
Result: Success
The ftp command successfully retrieved index.html from 10.0.0.2.
The output of div (the test program) is:
[2011-10-03 05:36:01] IPv4: 192.168.1.2:5714 -> 10.0.0.2:80
[2011-10-03 05:36:01] IPv4: 10.0.0.2:80 -> 192.168.1.2:5714
[2011-10-03 05:36:01] IPv4: 192.168.1.2:5714 -> 10.0.0.2:80
[2011-10-03 05:36:01] IPv4: 192.168.1.2:5714 -> 10.0.0.2:80
[2011-10-03 05:36:01] IPv4: 10.0.0.2:80 -> 192.168.1.2:5714
[2011-10-03 05:36:01] IPv4: 10.0.0.2:80 -> 192.168.1.2:5714
[2011-10-03 05:36:01] IPv4: 10.0.0.2:80 -> 192.168.1.2:5714
[2011-10-03 05:36:01] IPv4: 192.168.1.2:5714 -> 10.0.0.2:80
[2011-10-03 05:36:01] IPv4: 192.168.1.2:5714 -> 10.0.0.2:80
[2011-10-03 05:36:01] IPv4: 192.168.1.2:5714 -> 10.0.0.2:80
[2011-10-03 05:36:01] IPv4: 10.0.0.2:80 -> 192.168.1.2:5714
---[ SCENARIO B: Without NAT - Inbound (OpenBSD 4.9) ]---
Result: Success
The telnet command to the daytime port succeeded. div's output is:
[2011-10-03 05:35:31] IPv4: 10.0.0.2:33118 -> 192.168.1.2:13
[2011-10-03 05:35:31] IPv4: 192.168.1.2:13 -> 10.0.0.2:33118
[2011-10-03 05:35:31] IPv4: 10.0.0.2:33118 -> 192.168.1.2:13
[2011-10-03 05:35:31] IPv4: 192.168.1.2:13 -> 10.0.0.2:33118
[2011-10-03 05:35:31] IPv4: 192.168.1.2:13 -> 10.0.0.2:33118
[2011-10-03 05:35:31] IPv4: 10.0.0.2:33118 -> 192.168.1.2:13
[2011-10-03 05:35:31] IPv4: 10.0.0.2:33118 -> 192.168.1.2:13
[2011-10-03 05:35:31] IPv4: 192.168.1.2:13 -> 10.0.0.2:33118
---[ SCENARIO C: With NAT - Outbound (OpenBSD 4.9) ]---
Result: Success
The ftp command successfully retrieved index.html from 10.0.0.2.
div's output shows un-NAT'ed IPs for inbound packets but it works
nonetheless.
[2011-10-03 05:29:10] IPv4: 10.0.0.1:50413 -> 10.0.0.2:80
[2011-10-03 05:29:10] IPv4: 10.0.0.2:80 -> 192.168.1.2:24413
[2011-10-03 05:29:10] IPv4: 10.0.0.1:50413 -> 10.0.0.2:80
[2011-10-03 05:29:10] IPv4: 10.0.0.1:50413 -> 10.0.0.2:80
[2011-10-03 05:29:10] IPv4: 10.0.0.2:80 -> 192.168.1.2:24413
[2011-10-03 05:29:10] IPv4: 10.0.0.2:80 -> 192.168.1.2:24413
[2011-10-03 05:29:10] IPv4: 10.0.0.2:80 -> 192.168.1.2:24413
[2011-10-03 05:29:10] IPv4: 10.0.0.1:50413 -> 10.0.0.2:80
[2011-10-03 05:29:10] IPv4: 10.0.0.1:50413 -> 10.0.0.2:80
[2011-10-03 05:29:10] IPv4: 10.0.0.1:50413 -> 10.0.0.2:80
[2011-10-03 05:29:10] IPv4: 10.0.0.2:80 -> 192.168.1.2:24413
---[ SCENARIO D: With NAT - Inbound (OpenBSD 4.9) ]---
Result: Success
The daytime connection worked successfully. Again, div's output shows
un-NAT'ed IPs for inbound packets but it works.
[2011-10-03 05:26:23] IPv4: 10.0.0.2:13574 -> 192.168.1.2:13
[2011-10-03 05:26:23] IPv4: 10.0.0.1:13 -> 10.0.0.2:13574
[2011-10-03 05:26:23] IPv4: 10.0.0.2:13574 -> 192.168.1.2:13
[2011-10-03 05:26:23] IPv4: 10.0.0.1:13 -> 10.0.0.2:13574
[2011-10-03 05:26:23] IPv4: 10.0.0.1:13 -> 10.0.0.2:13574
[2011-10-03 05:26:23] IPv4: 10.0.0.2:13574 -> 192.168.1.2:13
[2011-10-03 05:26:23] IPv4: 10.0.0.2:13574 -> 192.168.1.2:13
[2011-10-03 05:26:23] IPv4: 10.0.0.1:13 -> 10.0.0.2:13574
======[ TEST RESULTS WITH OPENBSD -CURRENT ]======
Here are the results using OpenBSD -current (Sep 22, 2011 i386
snapshot) on the Firewall.
---[ SCENARIO A: Without NAT - Outbound (OpenBSD -current) ]---
Result: Success
The ftp command successfully retrieved index.html from 10.0.0.2.
[2011-10-03 14:48:27] IPv4: 192.168.1.2:46617 -> 10.0.0.2:80
[2011-10-03 14:48:27] IPv4: 10.0.0.2:80 -> 192.168.1.2:46617
[2011-10-03 14:48:27] IPv4: 192.168.1.2:46617 -> 10.0.0.2:80
[2011-10-03 14:48:27] IPv4: 192.168.1.2:46617 -> 10.0.0.2:80
[2011-10-03 14:48:27] IPv4: 10.0.0.2:80 -> 192.168.1.2:46617
[2011-10-03 14:48:27] IPv4: 10.0.0.2:80 -> 192.168.1.2:46617
[2011-10-03 14:48:27] IPv4: 10.0.0.2:80 -> 192.168.1.2:46617
[2011-10-03 14:48:27] IPv4: 192.168.1.2:46617 -> 10.0.0.2:80
[2011-10-03 14:48:27] IPv4: 192.168.1.2:46617 -> 10.0.0.2:80
[2011-10-03 14:48:27] IPv4: 192.168.1.2:46617 -> 10.0.0.2:80
[2011-10-03 14:48:27] IPv4: 10.0.0.2:80 -> 192.168.1.2:46617
---[ SCENARIO B: Without NAT - Inbound (OpenBSD -current) ]---
Result: Success
daytime connection succeeded.
[2011-10-03 14:48:52] IPv4: 10.0.0.2:24373 -> 192.168.1.2:13
[2011-10-03 14:48:52] IPv4: 192.168.1.2:13 -> 10.0.0.2:24373
[2011-10-03 14:48:52] IPv4: 10.0.0.2:24373 -> 192.168.1.2:13
[2011-10-03 14:48:52] IPv4: 192.168.1.2:13 -> 10.0.0.2:24373
[2011-10-03 14:48:52] IPv4: 192.168.1.2:13 -> 10.0.0.2:24373
[2011-10-03 14:48:52] IPv4: 10.0.0.2:24373 -> 192.168.1.2:13
[2011-10-03 14:48:52] IPv4: 10.0.0.2:24373 -> 192.168.1.2:13
[2011-10-03 14:48:52] IPv4: 192.168.1.2:13 -> 10.0.0.2:24373
---[ SCENARIO C: With NAT - Outbound (OpenBSD -current) ]---
Result: Failure
The ftp command "hangs" when trying to retrieve index.html from
10.0.0.2. Please note the timestamps in the div output below.
[2011-10-03 15:05:26] IPv4: 10.0.0.1:52881 -> 10.0.0.2:80
[2011-10-03 15:05:26] IPv4: 10.0.0.2:80 -> 192.168.1.2:25077
[2011-10-03 15:05:29] IPv4: 10.0.0.2:80 -> 192.168.1.2:25077
[2011-10-03 15:05:30] IPv4: 10.0.0.2:80 -> 192.168.1.2:29843
[2011-10-03 15:05:32] IPv4: 10.0.0.1:52881 -> 10.0.0.2:80
[2011-10-03 15:05:32] IPv4: 10.0.0.2:80 -> 192.168.1.2:25077
[2011-10-03 15:05:35] IPv4: 10.0.0.2:80 -> 192.168.1.2:25077
[2011-10-03 15:05:44] IPv4: 10.0.0.1:52881 -> 10.0.0.2:80
[2011-10-03 15:05:44] IPv4: 10.0.0.2:80 -> 192.168.1.2:25077
[2011-10-03 15:05:47] IPv4: 10.0.0.2:80 -> 192.168.1.2:25077
Here is the tcpdump trace on em0 of the Firewall (the NIC directly
connected to Outside).
# tcpdump -ni em0 -s 1500
tcpdump: listening on em0, link-type EN10MB
15:05:26.345068 10.0.0.1.52881 > 10.0.0.2.80: S 4258721679:4258721679(0) win
16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 3921547586 0> (DF)
15:05:26.345363 10.0.0.2.80 > 10.0.0.1.52881: S 1164814693:1164814693(0) ack
4258721680 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp
4136635551 3921547586> (DF)
15:05:29.339992 10.0.0.2.80 > 10.0.0.1.52881: S 1164814693:1164814693(0) ack
4258721680 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp
4136635557 3921547586> (DF)
15:05:30.599582 10.0.0.2.80 > 10.0.0.1.56182: S 2918134765:2918134765(0) ack
2460333704 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp
1780258867 417263802> (DF)
15:05:32.338647 10.0.0.1.52881 > 10.0.0.2.80: S 4258721679:4258721679(0) win
16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 3921547598 0> (DF)
15:05:32.338978 10.0.0.2.80 > 10.0.0.1.52881: S 1164814693:1164814693(0) ack
4258721680 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp
4136635563 3921547598> (DF)
15:05:35.338100 10.0.0.2.80 > 10.0.0.1.52881: S 1164814693:1164814693(0) ack
4258721680 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp
4136635569 3921547598> (DF)
15:05:44.335083 10.0.0.1.52881 > 10.0.0.2.80: S 4258721679:4258721679(0) win
16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 3921547622 0> (DF)
15:05:44.335399 10.0.0.2.80 > 10.0.0.1.52881: S 1164814693:1164814693(0) ack
4258721680 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp
4136635587 3921547622> (DF)
15:05:47.334391 10.0.0.2.80 > 10.0.0.1.52881: S 1164814693:1164814693(0) ack
4258721680 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp
4136635593 3921547622> (DF)
When I used tcpdump to capture packets on the em1 interface, nothing
appeared. It looks like the packets did not cross over to the em1
interface.
This is how it looks like on the Inside box:
# ftp http://10.0.0.2/index.html
Trying 10.0.0.2...
---[ SCENARIO D: With NAT - Inbound (OpenBSD -current) ]---
Result: Failure
The telnet command to the daytime port failed and eventually timed out.
This is the output of div (again, please note timestamps):
[2011-10-03 15:07:00] IPv4: 10.0.0.2:8625 -> 192.168.1.2:13
[2011-10-03 15:07:06] IPv4: 10.0.0.2:8625 -> 192.168.1.2:13
[2011-10-03 15:07:18] IPv4: 10.0.0.2:8625 -> 192.168.1.2:13
[2011-10-03 15:07:42] IPv4: 10.0.0.2:8625 -> 192.168.1.2:13
tcpdump output:
# tcpdump -ni em0 -s 1500
tcpdump: listening on em0, link-type EN10MB
15:07:00.690907 10.0.0.2.8625 > 10.0.0.1.13: S 309367114:309367114(0) win 16384
<mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 3263115685 0> (DF) [tos
0x10]
15:07:06.679936 10.0.0.2.8625 > 10.0.0.1.13: S 309367114:309367114(0) win 16384
<mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 3263115697 0> (DF) [tos
0x10]
15:07:18.676184 10.0.0.2.8625 > 10.0.0.1.13: S 309367114:309367114(0) win 16384
<mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 3263115721 0> (DF) [tos
0x10]
15:07:42.668793 10.0.0.2.8625 > 10.0.0.1.13: S 309367114:309367114(0) win 16384
<mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 3263115769 0> (DF) [tos
0x10]
How it looks like on the Outside box:
# telnet 10.0.0.1 13
Trying 10.0.0.1...
telnet: connect to address 10.0.0.1: Connection timed out
======[ SUMMARY OF TEST RESULTS ]======
Test Scenario OpenBSD 4.9 Sep 22, 2011 snap
------------------------- ----------- -----------------
A: Without NAT (Outbound) Success Success
B: Without NAT (Inbound) Success Success
C: With NAT (Outbound; nat-to) Success Failure
D: With NAT (Inbound; rdr-to) Success Failure
I will continue investigating. Meanwhile, if you would like further
information or need me to do any further tests or try diffs, please let
me know and I will be glad to do so.
Thank you,
Lawrence
