Hi Misc,
I'm making some trials and benchmarks about pf's divert. My test environment is
like this;
I have 2 Linux devices and I have an OpenBSD device which are directly
connected to an OpenBSD Device. This OpenBSD device acts likes a router.
Network settings in OpenBSD:
---------
vertigo# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:30:18:0a:a6:2e
index 2 priority 0 llprio 3
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
status: active
inet 172.20.35.1 netmask 0xffffff00 broadcast 172.20.35.255
vertigo# ifconfig em2
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:30:18:0a:a6:2f
index 3 priority 0 llprio 3
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
status: active
inet 172.20.36.1 netmask 0xffffff00 broadcast 172.20.36.255
vertigo#
-------
-------
vertigo# sysctl net.inet.ip.forwarding
net.inet.ip.forwarding=1
-------
my pf.conf is like this;
-----------
vertigo# cat /etc/pf.conf
# $OpenBSD: pf.conf,v 1.55 2017/12/03 20:40:04 sthen Exp $
#
# See pf.conf(5) and /etc/examples/pf.conf
set skip on lo
# em1
pass in log quick on em1 from em1:network to172.20.36.0/24
pass out log quick on em1 from172.20.36.0/24to em1:network
# em2
pass in log quick on em2 from em2:network to172.20.35.0/24
pass out log quick on em2 from172.20.35.0/24to em2:network
# block all
block log quick all
block return # block stateless traffic
pass # establish keep-state
# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010
# Port build user does not need network
block return out log proto {tcp udp} user _pbuild
--------
On Linux-1, I'm making benchmark tests to Linux-2 with qperf,
--------
vertigo@linux1:~$ qperf 172.20.35.2 tcp_bw tcp_lat -t 10
tcp_bw:
bw = 115 MB/sec
tcp_lat:
latency = 157 us
vertigo@linux1:~$ qperf 172.20.35.2 udp_bw udp_lat -t 10
udp_bw:
send_bw = 120 MB/sec
recv_bw = 120 MB/sec
udp_lat:
latency = 158 us
vertigo@linux1:~$
---------
After that, I updated the pf.conf and diverted packages to port 700
------
vertigo# cat /etc/pf.conf
# $OpenBSD: pf.conf,v 1.55 2017/12/03 20:40:04 sthen Exp $
#
# See pf.conf(5) and /etc/examples/pf.conf
set skip on lo
# em1
pass in log quick on em1 from em1:network to172.20.36.0/24
pass out log quick on em1 from172.20.36.0/24to em1:network divert-packet port
700
# em2
pass in log quick on em2 from em2:network to172.20.35.0/24
pass out log quick on em2 from172.20.35.0/24to em2:network divert-packet port
700
# block all
block log quick all
block return # block stateless traffic
pass # establish keep-state
# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010
# Port build user does not need network
block return out log proto {tcp udp} user _pbuild
------
On port 700, I run the example program on OpenBSD divert manpage.
https://man.openbsd.org/divert.4
-----
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/ip.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <string.h>
#include <err.h>
#define DIVERT_PORT 700
int
main(int argc, char *argv[])
{
int fd, s;
struct sockaddr_in sin;
socklen_t sin_len;
fd = socket(AF_INET, SOCK_RAW, IPPROTO_DIVERT);
if (fd == -1)
err(1, "socket");
memset(&sin, 0, sizeof(sin));
sin.sin_family = AF_INET;
sin.sin_port = htons(DIVERT_PORT);
sin.sin_addr.s_addr = 0;
sin_len = sizeof(struct sockaddr_in);
s = bind(fd, (struct sockaddr *) &sin, sin_len);
if (s == -1)
err(1, "bind");
for (;;) {
ssize_t n;
char packet[IP_MAXPACKET];
struct ip *ip;
struct tcphdr *th;
int hlen;
char src[48], dst[48];
memset(packet, 0, sizeof(packet));
n = recvfrom(fd, packet, sizeof(packet), 0,
(struct sockaddr *) &sin, &sin_len);
if (n == -1) {
warn("recvfrom");
continue;
}
if (n < sizeof(struct ip)) {
warnx("packet is too short");
continue;
}
ip = (struct ip *) packet;
hlen = ip->ip_hl << 2;
if (hlen < sizeof(struct ip) || ntohs(ip->ip_len) < hlen ||
n < ntohs(ip->ip_len)) {
warnx("invalid IPv4 packet");
continue;
}
th = (struct tcphdr *) (packet + hlen);
if (inet_ntop(AF_INET, &ip->ip_src, src,
sizeof(src)) == NULL)
(void)strlcpy(src, "?", sizeof(src));
if (inet_ntop(AF_INET, &ip->ip_dst, dst,
sizeof(dst)) == NULL)
(void)strlcpy(dst, "?", sizeof(dst));
printf("%s:%u -> %s:%u\n",
src,
ntohs(th->th_sport),
dst,
ntohs(th->th_dport)
);
n = sendto(fd, packet, n, 0, (struct sockaddr *) &sin,
sin_len);
if (n == -1)
warn("sendto");
}
return 0;
}
And I made the same test again;
There is a big difference in TCP benchmark tests,
I repeated the tests, but the results did not change.
---------
vertigo@linux1:~$ qperf 172.20.35.2 tcp_bw tcp_lat -t 10
tcp_bw:
bw = 32.6 KB/sec
tcp_lat:
latency = 214 us
vertigo@linux1:~$ qperf 172.20.35.2 udp_bw udp_lat -t 10
udp_bw:
send_bw = 120 MB/sec
recv_bw = 111 MB/sec
udp_lat:
latency = 214 us
vertigo@linux1:~$
----------
I could not understand why this big difference occurs. Is there any way to
improve "divert" s performance?
Sent with [ProtonMail](https://protonmail.com) Secure Email.
