On Wed, Oct 7, 2020 at 10:37 PM Aaron Mason <simplersolut...@gmail.com> wrote: > > On Mon, Oct 5, 2020 at 12:22 PM Scott Seekamp <sseek...@risei.net> wrote: > > > > I had a similar speed drop on an Edge Router 4. I don’t know if it’s the > > same situation on the Lite, but I believe it’s expected due to hardware > > acceleration support (or lack of) and single core performance on the pf > > side. > > I read somewhere that this drop can happen even with the factory OS - > the routing is handled by an ASIC (which is how they push near-gigabit > forwarding speeds) but if you do any sort of filtering, it falls back > to software routing. Since the ASIC is black box voodoo, OpenBSD will > always use the CPU for routing.
The more I read on Ubiquiti ERL, I realize this may indeed be the case -- "hardware offloading" is what they call it. See https://help.ui.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-Offloading Thanks. -Amarendra > > Scott > > > > > On Oct 4, 2020, at 17:24, Amarendra Godbole <amarendra.godb...@gmail.com> > > > wrote: > > > > > > Sorry I forgot including "ifconfig" output: > > > > > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768 > > > index 5 priority 0 llprio 3 > > > groups: lo > > > inet6 ::1 prefixlen 128 > > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 > > > inet 127.0.0.1 netmask 0xff000000 > > > > > > cnmac0: flags=808843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF4> > > > mtu 1500 > > > lladdr a8:28:dc:cc:2e:6f > > > index 1 priority 0 llprio 3 > > > groups: egress > > > media: Ethernet autoselect (1000baseT full-duplex,master) > > > status: active > > > inet 73.xx.xx.xx netmask 0xfffffe00 broadcast 73.xx.xx.255 > > > > > > cnmac1: > > > flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > > > mtu 1500 > > > lladdr 78:8a:20:46:a8:c1 > > > index 2 priority 0 llprio 3 > > > media: Ethernet autoselect (1000baseT full-duplex) > > > status: active > > > > > > cnmac2: > > > flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > > > mtu 1500 > > > lladdr 78:8a:20:46:a8:c2 > > > index 3 priority 0 llprio 3 > > > media: Ethernet autoselect (none) > > > status: no carrier > > > enc0: flags=0<> > > > index 4 priority 0 llprio 3 > > > groups: enc > > > status: active > > > > > > bridge0: flags=41<UP,RUNNING> > > > index 6 llprio 3 > > > groups: bridge > > > priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp > > > cnmac2 flags=7<LEARNING,DISCOVER,BLOCKNONIP> > > > port 3 ifpriority 0 ifcost 0 > > > cnmac1 flags=7<LEARNING,DISCOVER,BLOCKNONIP> > > > port 2 ifpriority 0 ifcost 0 > > > vether0 flags=7<LEARNING,DISCOVER,BLOCKNONIP> > > > port 7 ifpriority 0 ifcost 0 > > > > > > vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu > > > 1500 > > > lladdr fe:e1:ba:d0:c8:a9 > > > index 7 priority 0 llprio 3 > > > groups: vether > > > media: Ethernet autoselect > > > status: active > > > inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255 > > > > > > pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33136 > > > index 8 priority 0 llprio 3 > > > groups: pflog > > > > > >> On Sun, Oct 4, 2020 at 2:22 PM Amarendra Godbole > > >> <amarendra.godb...@gmail.com> wrote: > > >> > > >> Hi misc@ > > >> > > >> I recently introduced an OpenBSD firewall inline and noticed a > > >> reduction in overall download speeds. I am trying to understand why > > >> this may be so. The firewall is Ubiquiti ERL running 6.7 release. > > >> Internet connection is Comcast xfinity via cable modem, plan 200 > > >> Mbits/s down and 10 Mbits/s up. Details follow: > > >> > > >> 1. config #1: MacBook - Linksys WRT1200AC - xfinity cable modem > > >> (speed: ~210 Mbits/s down, 6 Mbits/s up) > > >> 2. config #2: MacBook - Linksys WRT1200AC - Ubiquiti ERL - xfinity > > >> cable modem (speed: ~90 MBits down, 6 Mbits/s up) > > >> 3. config #3 (Line speed): MacBook wired to cable modem (~230 Mbits/s > > >> down, ~8 Mbits/s up). > > >> > > >> Linksys is running latest OpenWrt, and speed tests were run on MacBook > > >> connected wired to Linksys. It was difficult to try tcpbench since the > > >> setup was cumbersome, and iperf3 public servers end up being busy more > > >> often than not (and threads on misc@ indicated iperf3 wasn't as > > >> reliable either). Test numbers come from speedtest.net and > > >> speed.cloudflare.com. While I realize this speed test is hardly > > >> accurate, I have tried to maintain the same configuration (no ERL and > > >> inline ERL) to obtain relative numbers. > > >> > > >> I am trying to understand the reduction from 210 Mbits/s down to 90 > > >> Mbits/s down between config #1 and config #2 above. The slowdown is > > >> not noticeable to family, so this is more of my intellectual curiosity > > >> than screams over a buffering video! :-) > > >> > > >> Relevant system information (dmesg, etc.) below. All sysctl values > > >> attached as sysctl.txt I gathered it by reading similar threads on > > >> misc@. If I missed anything, please let me know. Thanks in advance. > > >> > > >> -Amarendra > > >> > > >> > > >> dmesg: > > >> > > >> Copyright (c) 1982, 1986, 1989, 1991, 1993 > > >> The Regents of the University of California. All rights reserved. > > >> Copyright (c) 1995-2020 OpenBSD. All rights reserved. > > >> https://www.OpenBSD.org > > >> OpenBSD 6.7 (GENERIC.MP) #134: Thu May 7 16:05:06 MDT 2020 > > >> dera...@octeon.openbsd.org:/usr/src/sys/arch/octeon/compile/GENERIC.MP > > >> real mem = 536870912 (512MB) > > >> avail mem = 506740736 (483MB) > > >> mainbus0 at root: board 20002 rev 2.18 > > >> cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation > > >> cpu0: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way > > >> cpu1 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation > > >> cpu1: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way > > >> clock0 at mainbus0: int 5 > > >> octcrypto0 at mainbus0 > > >> iobus0 at mainbus0 > > >> simplebus0 at iobus0: "soc" > > >> octciu0 at simplebus0 > > >> octsmi0 at simplebus0 > > >> octpip0 at simplebus0 > > >> octgmx0 at octpip0 interface 0 > > >> cnmac0 at octgmx0: RGMII, address 78:8a:20:46:a8:c0 > > >> atphy0 at cnmac0 phy 7: AR8035 10/100/1000 PHY, rev. 2 > > >> cnmac1 at octgmx0: RGMII, address 78:8a:20:46:a8:c1 > > >> atphy1 at cnmac1 phy 6: AR8035 10/100/1000 PHY, rev. 2 > > >> cnmac2 at octgmx0: RGMII, address 78:8a:20:46:a8:c2 > > >> atphy2 at cnmac2 phy 5: AR8035 10/100/1000 PHY, rev. 2 > > >> com0 at simplebus0: ns16550a, 64 byte fifo > > >> com0: console > > >> dwctwo0 at iobus0 base 0x1180068000000 irq 56 > > >> usb0 at dwctwo0: USB revision 2.0 > > >> uhub0 at usb0 configuration 1 interface 0 "Octeon DWC2 root hub" rev > > >> 2.00/1.00 addr 1 > > >> octrng0 at iobus0 base 0x1400000000000 irq 0 > > >> /dev/ksyms: Symbol table not valid. > > >> umass0 at uhub0 port 1 configuration 1 interface 0 "Lexar USB Flash > > >> Drive" rev 2.10/11.00 addr 2 > > >> umass0: using SCSI over Bulk-Only > > >> scsibus0 at umass0: 2 targets, initiator 0 > > >> sd0 at scsibus0 targ 1 lun 0: <Lexar, USB Flash Drive, 1100> removable > > >> serial.21c40cd1719080003000 > > >> sd0: 30526MB, 512 bytes/sector, 62517248 sectors > > >> vscsi0 at root > > >> scsibus1 at vscsi0: 256 targets > > >> softraid0 at root > > >> scsibus2 at softraid0: 256 targets > > >> boot device: sd0 > > >> root on sd0a (2124441bc835a462.a) swap on sd0b dump on sd0b > > >> WARNING: No TOD clock, believing file system. > > >> WARNING: CHECK AND RESET THE DATE! > > >> > > >> > > >> pftcl -s: > > >> > > >> match in all scrub (no-df random-id max-mss 1440) > > >> block drop in quick on ! cnmac0 inet from xx.xx.xx.xx/23 to any > > >> block drop in quick inet from xx.xx.xx.xx to any > > >> block drop all > > >> pass out quick on egress inet from (vether0:network) to any flags S/SA > > >> nat-to (egress) round-robin > > >> pass out quick inet all flags S/SA > > >> pass in on vether0 inet all flags S/SA > > >> pass in on cnmac1 inet all flags S/SA > > >> pass in on cnmac2 inet all flags S/SA > > >> > > >> pfctl -si: > > >> > > >> Status: Enabled for 3 days 18:14:11 Debug: err > > >> > > >> Interface Stats for egress IPv4 IPv6 > > >> > > >> Bytes In 64537867779 0 > > >> Bytes Out 7005140381 0 > > >> Packets In > > >> Passed 56024370 0 > > >> Blocked 43050 0 > > >> Packets Out > > >> Passed 22271061 0 > > >> Blocked 0 0 > > >> > > >> State Table Total Rate > > >> current entries 847 > > >> half-open tcp 23 > > >> searches 158989755 489.4/s > > >> inserts 1095307 3.4/s > > >> removals 1094460 3.4/s > > >> Counters > > >> match 1343178 4.1/s > > >> bad-offset 0 0.0/s > > >> fragment 0 0.0/s > > >> short 1 0.0/s > > >> normalize 0 0.0/s > > >> memory 0 0.0/s > > >> bad-timestamp 0 0.0/s > > >> congestion 41 0.0/s > > >> ip-option 26164 0.1/s > > >> proto-cksum 0 0.0/s > > >> state-mismatch 10758 0.0/s > > >> state-insert 0 0.0/s > > >> state-limit 0 0.0/s > > >> src-limit 0 0.0/s > > >> synproxy 0 0.0/s > > >> translate 0 0.0/s > > >> no-route 0 0.0/s > > >> > > >> pfctl -s memory: > > >> > > >> states hard limit 100000 > > >> src-nodes hard limit 10000 > > >> frags hard limit 16384 > > >> tables hard limit 1000 > > >> table-entries hard limit 200000 > > >> pktdelay-pkts hard limit 10000 > > >> > > >> The netlivlocks value keeps on increasing regularly: > > >> kern.netlivelocks=57911 > > >> > > >> netstat -m: > > >> > > >> 1009 mbufs in use: > > >> 917 mbufs allocated to data > > >> 5 mbufs allocated to packet headers > > >> 87 mbufs allocated to socket names and addresses > > >> 801/7256 mbuf 2048 byte clusters in use (current/peak) > > >> 0/15 mbuf 2112 byte clusters in use (current/peak) > > >> 0/24 mbuf 4096 byte clusters in use (current/peak) > > >> 0/8 mbuf 8192 byte clusters in use (current/peak) > > >> 0/0 mbuf 9216 byte clusters in use (current/peak) > > >> 0/0 mbuf 12288 byte clusters in use (current/peak) > > >> 0/0 mbuf 16384 byte clusters in use (current/peak) > > >> 0/8 mbuf 65536 byte clusters in use (current/peak) > > >> 6512/17088/131072 Kbytes allocated to network (current/peak/max) > > >> 0 requests for memory denied > > >> 0 requests for memory delayed > > >> 0 calls to protocol drain routines > > >> > > >> netstat -i: > > >> > > >> Name Mtu Network Address Ipkts Ifail Opkts > > >> Ofail Colls > > >> lo0 32768 <Link> 198 0 198 > > >> 0 0 > > >> lo0 32768 localhost/1 localhost 198 0 198 > > >> 0 0 > > >> lo0 32768 fe80::%lo0/ fe80::1%lo0 198 0 198 > > >> 0 0 > > >> lo0 32768 127/8 localhost 198 0 198 > > >> 0 0 > > >> cnmac0 1600 <Link> a8:28:dc:cc:2e:6f 56088774 0 22283491 > > >> 2688 0 > > >> cnmac0 1600 73.231.60/2 c-73-231-60-128.h 56088774 0 22283491 > > >> 2688 0 > > >> cnmac1 1600 <Link> 78:8a:20:46:a8:c1 23646497 4 56569853 > > >> 48 0 > > >> cnmac2 1600 <Link> 78:8a:20:46:a8:c2 14823 0 226198 > > >> 226198 0 > > >> enc0* 0 <Link> 0 0 0 > > >> 0 0 > > >> bridge0 1500 <Link> 23187238 0 57022219 > > >> 0 0 > > >> vether0 32768 <Link> fe:e1:ba:d0:c8:a9 23056709 0 56795991 > > >> 0 0 > > >> vether0 32768 192.168.10/ 192.168.10.1 23056709 0 56795991 > > >> 0 0 > > >> pflog0 33136 <Link> 0 0 26171 > > >> 0 0 > > > > > > > > -- > Aaron Mason - Programmer, open source addict > I've taken my software vows - for beta or for worse >
