Re: unbound network optimizations
Dear Jordan,
thanks for your email.
On 12.12.19 20:54, Jordan Geoghegan wrote:
>> OpenBSD is a guest VM on a debian buster host using virtual e1000
>> network card ("Intel 82540EM" driver in openbsd). No firewall
>> in between. The VM is a tor-exit node.
>
> I've heard others recommend using the vio driver over the em driver numerous
> times on here if running a virtualized instance. You may have a better time
> than you are now by using the VirtIO drivers.
> The intel nic emulation can sometimes have issues. Better to use an interface
> designed for virtualized environments.
I'm testing both. I thought I might have some throughout issues with the virtio
driver
and switched to the e1000 for testing. Unfortunately the e1000 driver leads to
openbsd
kernel panics > 200 Mbit/s so I switched back to the virtio driver which is
more stable.
The "error: recvfrom xxx failed: Host is down" problem is unaffected by
whichever NIC
driver I use.
Cheers,
Winter
Re: unbound network optimizations
On 12 Dec at 20:54, Jordan Geoghegan wrote:
>
>
> On 2019-12-12 06:21, Winter Paulson wrote:
> > Hello,
> >
> > I'm also experiencing the "Host is down" problem:
> >
> > unbound: [85343:0] error: recvfrom 361 failed: Host is down
> >
> > Running openbsd 6.6 (GENERIC.MP), current syspatch,
> > native unbound as a full resolver, pf disabled.
> >
> > OpenBSD is a guest VM on a debian buster host using virtual e1000
> > network card ("Intel 82540EM" driver in openbsd). No firewall
> > in between. The VM is a tor-exit node.
>
> I've heard others recommend using the vio driver over the em driver numerous
> times on here if running a virtualized instance. You may have a better time
> than you are now by using the VirtIO drivers. The intel nic emulation can
> sometimes have issues. Better to use an interface designed for virtualized
> environments.
I am seeing the same on OpenBSD on vmd/vmm. Similar setup.
Mischa
Re: unbound network optimizations
On 2019-12-12 06:21, Winter Paulson wrote:
Hello,
I'm also experiencing the "Host is down" problem:
unbound: [85343:0] error: recvfrom 361 failed: Host is down
Running openbsd 6.6 (GENERIC.MP), current syspatch,
native unbound as a full resolver, pf disabled.
OpenBSD is a guest VM on a debian buster host using virtual e1000
network card ("Intel 82540EM" driver in openbsd). No firewall
in between. The VM is a tor-exit node.
I've heard others recommend using the vio driver over the em driver
numerous times on here if running a virtualized instance. You may have a
better time than you are now by using the VirtIO drivers. The intel nic
emulation can sometimes have issues. Better to use an interface designed
for virtualized environments.
Cheers,
Jordan
Re: unbound network optimizations
Hello,
I'm also experiencing the "Host is down" problem:
unbound: [85343:0] error: recvfrom 361 failed: Host is down
Running openbsd 6.6 (GENERIC.MP), current syspatch,
native unbound as a full resolver, pf disabled.
OpenBSD is a guest VM on a debian buster host using virtual e1000
network card ("Intel 82540EM" driver in openbsd). No firewall
in between. The VM is a tor-exit node.
$ netstat -s -p udp
udp:
9379065 datagrams received
0 with incomplete header
0 with bad data length field
16 with bad checksum
32675 with no checksum
9346390 input packets software-checksummed
6059478 output packets software-checksummed
201392 dropped due to no socket
0 broadcast/multicast datagrams dropped due to no socket
0 dropped due to missing IPsec protection
0 dropped due to full socket buffers
9177657 delivered
18686884 datagrams output
9378087 missed PCB cache
/etc/sysctl.conf
kern.maxfiles=3
net.inet.udp.sendspace=262144
net.inet.udp.recvspace=262144
kern.somaxconn=2500
kern.maxproc=3000
kern.bufcachepercent=50
net.inet.tcp.rfc3390=1
/etc/login.conf
unbound:\
:openfiles=13500:\
:tc=daemon:
/var/unbound/etc/unbound.conf
num-threads: 2
msg-cache-slabs: 4
rrset-cache-slabs: 4
infra-cache-slabs: 4
key-cache-slabs: 4
rrset-cache-size: 100m
msg-cache-size: 50m
outgoing-range: 450
outgoing-num-tcp: 25
incoming-num-tcp: 25
msg-buffer-size: 65552
Each unbound thread is doing around 1.2 million queries a day.
I'd also be interested in knowing if increasing udp.sendspace and
udp.recvspace has negative implications or what else might be done.
Thank you!
w.
Re: unbound network optimizations
Replying to my own thread as it was pointed out that I neglected to add some information. OpenBSD 6.5 (GENERIC.MP) #7: Wed Nov 20 23:21:48 MST 2019 Native unbound (latest syspatch) Bge interfaces running on an LACP trunk with IPv4 and IPv6 addresses. NameMtu Network Address Ipkts IfailOpkts Ofail Colls bge0150044:a8:42:37:bb:b8 114258345 0 84203414 0 0 bge1150044:a8:42:37:bb:b8 57304058 0 84467834 0 0 bge2* 150044:a8:42:37:bb:ba0 00 0 0 bge3* 150044:a8:42:37:bb:bb0 00 0 0 enc0* 00 00 0 0 trunk0 150044:a8:42:37:bb:b8 171549799 0 16865964412 0 trunk0 1500 fe80::%trunk0/64 fe80::601b:75c2:6b28:7276%trunk0 171549799 0 16865964412 0 -Steve S. -Original Message- From: Steven Surdock Sent: Monday, December 2, 2019 1:34 PM To: misc@openbsd.org Subject: unbound network optimizations I'm running a pair of unbound resolvers and am attempting to optimize performance on them. This stemmed from noticing a couple of issues in the logs. Dec 2 11:26:52 ns1 unbound: [54230:5] error: recvfrom 26 failed: Host is down Dec 2 11:27:11 ns1 unbound: [54230:5] notice: sendto failed: Resource temporarily unavailable Dec 2 11:27:11 ns1 unbound: [54230:5] notice: remote address is 192.168.2.42 port 5088 I believed the first message is related to a dropped UDP request or subsequent response. 'netstat -p -u udp' shows "dropped due to full socket buffers". This was significantly reduced by increasing, net.inet.udp.recvspace=262144 net.inet.udp.sendspace=262144 Unfortunately, I'm still seeing a few UDP drops. Is there a danger in setting this is high? ns1$ netstat -s -p udp udp: 698584369 datagrams received 0 with incomplete header 0 with bad data length field 2508 with bad checksum 676259 with no checksum 86709458 input packets software-checksummed 706308843 output packets software-checksummed 641800 dropped due to no socket 0 broadcast/multicast datagrams dropped due to no socket 0 dropped due to missing IPsec protection 77324 dropped due to full socket buffers 697862737 delivered 706308952 datagrams output 698578008 missed PCB cache The second log message seems to stem from a dropped TCP request. There seems to be a significant number of these and I'm assuming they stem from "452447 SYN packets dropped due to queue or memory full" as the number of log message is in the same range as the number of dropped SYN packets. ns1$ netstat -s -p tcp tcp: 1856161 packets sent 359575 data packets (73608768 bytes) 27022 data packets (5076843 bytes) retransmitted 0 fast retransmitted packets 928517 ack-only packets (414664 delayed) 0 URG only packets 67 window probe packets 2217 window update packets 538808 control packets 271352 packets software-checksummed 2391157 packets received 739060 acks (for 71221089 bytes) 225691 duplicate acks 506 acks for unsent data 0 acks for old data 473441 packets (101441404 bytes) received in-sequence 111074 completely duplicate packets (75769595 bytes) 21701 old duplicate packets 3 packets with some duplicate data (112 bytes duplicated) 231945 out-of-order packets (88494422 bytes) 21 packets (0 bytes) of data after window 0 window probes 34417 window update packets 6771 packets received after close 52 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded for missing IPsec protection 0 discarded due to memory shortage 231084 packets software-checksummed 0 bad/missing md5 checksums 0 good md5 checksums 213191 connection requests 156110 connection accepts 340472 connections established (including accepts) 369167 connections closed (including 14600 drops) 0 connections drained 14167 embryonic connections dropped 860911 segments updated rtt (of 838375 attempts) 40788 retransmit timeouts 3005 connections dropped by rexmit timeout 69 persist timeouts 6563 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 12445 correct ACK header predictions 222843 correct data packet header predictions
unbound network optimizations
I'm running a pair of unbound resolvers and am attempting to optimize performance on them. This stemmed from noticing a couple of issues in the logs. Dec 2 11:26:52 ns1 unbound: [54230:5] error: recvfrom 26 failed: Host is down Dec 2 11:27:11 ns1 unbound: [54230:5] notice: sendto failed: Resource temporarily unavailable Dec 2 11:27:11 ns1 unbound: [54230:5] notice: remote address is 192.168.2.42 port 5088 I believed the first message is related to a dropped UDP request or subsequent response. 'netstat -p -u udp' shows "dropped due to full socket buffers". This was significantly reduced by increasing, net.inet.udp.recvspace=262144 net.inet.udp.sendspace=262144 Unfortunately, I'm still seeing a few UDP drops. Is there a danger in setting this is high? ns1$ netstat -s -p udp udp: 698584369 datagrams received 0 with incomplete header 0 with bad data length field 2508 with bad checksum 676259 with no checksum 86709458 input packets software-checksummed 706308843 output packets software-checksummed 641800 dropped due to no socket 0 broadcast/multicast datagrams dropped due to no socket 0 dropped due to missing IPsec protection 77324 dropped due to full socket buffers 697862737 delivered 706308952 datagrams output 698578008 missed PCB cache The second log message seems to stem from a dropped TCP request. There seems to be a significant number of these and I'm assuming they stem from "452447 SYN packets dropped due to queue or memory full" as the number of log message is in the same range as the number of dropped SYN packets. ns1$ netstat -s -p tcp tcp: 1856161 packets sent 359575 data packets (73608768 bytes) 27022 data packets (5076843 bytes) retransmitted 0 fast retransmitted packets 928517 ack-only packets (414664 delayed) 0 URG only packets 67 window probe packets 2217 window update packets 538808 control packets 271352 packets software-checksummed 2391157 packets received 739060 acks (for 71221089 bytes) 225691 duplicate acks 506 acks for unsent data 0 acks for old data 473441 packets (101441404 bytes) received in-sequence 111074 completely duplicate packets (75769595 bytes) 21701 old duplicate packets 3 packets with some duplicate data (112 bytes duplicated) 231945 out-of-order packets (88494422 bytes) 21 packets (0 bytes) of data after window 0 window probes 34417 window update packets 6771 packets received after close 52 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded for missing IPsec protection 0 discarded due to memory shortage 231084 packets software-checksummed 0 bad/missing md5 checksums 0 good md5 checksums 213191 connection requests 156110 connection accepts 340472 connections established (including accepts) 369167 connections closed (including 14600 drops) 0 connections drained 14167 embryonic connections dropped 860911 segments updated rtt (of 838375 attempts) 40788 retransmit timeouts 3005 connections dropped by rexmit timeout 69 persist timeouts 6563 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 12445 correct ACK header predictions 222843 correct data packet header predictions 828362 PCB cache misses 40214 dropped due to no socket 0 ECN connections accepted 0 ECE packets received 0 CWR packets received 9148 CE packets received 0 ECT packets sent 0 ECE packets sent 0 CWR packets sent cwr by fastrecovery: 385 cwr by timeout: 40788 cwr by ecn: 0 3161 bad connection attempts 452447 SYN packets dropped due to queue or memory full 161093 SYN cache entries added 0 hash collisions 156110 completed 0 aborted (no space to build PCB) 252 timed out 0 dropped due to overflow 0 dropped due to bucket overflow 4731 dropped due to RST 0 dropped due to ICMP unreachable 2809 SYN,ACKs retransmitted 913 duplicate SYNs received for entries already in the cache 0 SYNs dropped (no route or
