Re: OpenBSD as a router on Oracle T5120
On Tue, Jan 21, 2014 at 5:32 AM, Patrick Lamaiziere wrote: > Le Mon, 20 Jan 2014 18:59:02 -0200, > Eduardo Meyer a écrit : > > > hello, > > > > I am doing some basic testings on the above mentioned scenario and I > > am stuck on some limits which I consider to be very low: I cannot get > > more than 27Kpps and 200Mbit/s routing performance without starting > > to loose packets. > > ... > > All 6 network cards are Intel 82571EB which support MSI-X and should, > > in theory support IRQ balance. > > MSI are disabled on this chipset since OpenBSD 5.2... > You can try to renabled MSI in em(4), here this helps a lot (on amd64). > > Check the thread "(5.3) load problem on em(4) MSI / interrupt ?" on > misc@ > > https://www.mail-archive.com/misc@openbsd.org/msg123743.html > > Regards, > > patch on 5.3: > --- /usr/src/sys/dev/pci/if_em.c.orig Tue Oct 1 14:45:36 2013 > +++ /usr/src/sys/dev/pci/if_em.cTue Oct 1 14:48:52 2013 > @@ -337,7 +337,7 @@ > * Only use MSI on the newer PCIe parts, with the exception > * of 82571/82572 due to "Byte Enables 2 and 3 Are Not Set" > errata */ > - if (sc->hw.mac_type <= em_82572) > + if (sc->hw.mac_type < em_82571) > sc->osdep.em_pa.pa_flags &= ~PCI_FLAGS_MSI_ENABLED; > > /* Parameters (to be read from user) */ > Thank you everyone, here we go with the results. Disabling PF caused a 15% performance improvement. OpenBSD 5.5 made the system more responsive under this load but made no real difference on pps or bps results. MSIX did not cause any trouble and it helped to raise pps up to 30K and bps up to 240M which is better but still very low :-( Thank you all :-)
Re: OpenBSD as a router on Oracle T5120
Le Mon, 20 Jan 2014 18:59:02 -0200, Eduardo Meyer a écrit : > hello, > > I am doing some basic testings on the above mentioned scenario and I > am stuck on some limits which I consider to be very low: I cannot get > more than 27Kpps and 200Mbit/s routing performance without starting > to loose packets. > ... > All 6 network cards are Intel 82571EB which support MSI-X and should, > in theory support IRQ balance. MSI are disabled on this chipset since OpenBSD 5.2... You can try to renabled MSI in em(4), here this helps a lot (on amd64). Check the thread "(5.3) load problem on em(4) MSI / interrupt ?" on misc@ https://www.mail-archive.com/misc@openbsd.org/msg123743.html Regards, patch on 5.3: --- /usr/src/sys/dev/pci/if_em.c.orig Tue Oct 1 14:45:36 2013 +++ /usr/src/sys/dev/pci/if_em.cTue Oct 1 14:48:52 2013 @@ -337,7 +337,7 @@ * Only use MSI on the newer PCIe parts, with the exception * of 82571/82572 due to "Byte Enables 2 and 3 Are Not Set" errata */ - if (sc->hw.mac_type <= em_82572) + if (sc->hw.mac_type < em_82571) sc->osdep.em_pa.pa_flags &= ~PCI_FLAGS_MSI_ENABLED; /* Parameters (to be read from user) */
Re: OpenBSD as a router on Oracle T5120
On Mon, Jan 20, 2014 at 04:00:05PM -0800, Chris Cappuccio wrote: > Eduardo Meyer [dudu.me...@gmail.com] wrote: > > > > I will try > > ftp://ftp.openbsd.org//pub/OpenBSD/snapshots/sparc64/install55.iso right > > now. Other than simply running it is there anything else I should look at, > > or any new command line tool to play around? > > > > Nope. All improvements here are going to be in the kernel. There is > work going on in the myx driver, for instance this change by dlg: > > "introduce fine grained locking around the lists of packet handlers > myx maintains. this moves it away from relying on splnet to protect > them." > > http://www.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_myx.c.diff?r1=1.47;r2=1.48 > > Nobody has done anything like this in if_em yet, but perhaps someone > will step up. For that matter, myx interrupts aren't yet distributed > across cores, but appear to be getting closer to that possibility. Don't expect too much from this yet. This is more of a test balloon to see what is neccessary also I guess at first the performance will degrade. -- :wq Claudio
Re: OpenBSD as a router on Oracle T5120
Eduardo Meyer writes: [...] > I will try > ftp://ftp.openbsd.org//pub/OpenBSD/snapshots/sparc64/install55.iso right > now. Other than simply running it is there anything else I should look at, > or any new command line tool to play around? Disabling pf and kern.pool_debug would save some CPU cycles. -- jca | PGP: 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE (previous: 0x06A11494 / 61DB D9A0 00A4 67CF 2A90 8961 6191 8FBF 06A1 1494)
Re: OpenBSD as a router on Oracle T5120
Eduardo Meyer [dudu.me...@gmail.com] wrote: > > I will try > ftp://ftp.openbsd.org//pub/OpenBSD/snapshots/sparc64/install55.iso right > now. Other than simply running it is there anything else I should look at, > or any new command line tool to play around? > Nope. All improvements here are going to be in the kernel. There is work going on in the myx driver, for instance this change by dlg: "introduce fine grained locking around the lists of packet handlers myx maintains. this moves it away from relying on splnet to protect them." http://www.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_myx.c.diff?r1=1.47;r2=1.48 Nobody has done anything like this in if_em yet, but perhaps someone will step up. For that matter, myx interrupts aren't yet distributed across cores, but appear to be getting closer to that possibility.
Re: OpenBSD as a router on Oracle T5120
On Mon, Jan 20, 2014 at 8:34 PM, Chris Cappuccio wrote: > Eduardo Meyer [dudu.me...@gmail.com] wrote: > > hello, > > > > I am doing some basic testings on the above mentioned scenario and I am > > stuck on some limits which I consider to be very low: I cannot get more > > than 27Kpps and 200Mbit/s routing performance without starting to loose > > packets. > > > > System is: > > > > # uname -srm > > > > OpenBSD 5.4 sparc64 > ... > > CPU00: 0.0% user, 0.0% nice, 0.0% system, 98.2% interrupt, 1.8% idle > ... > > CPU30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > > > > All my NICs are getting interrupted on CPU0. > > > > OpenBSD doesn't yet support any other mode of operation, although you > may be seeing improvements here shortly. > > > Thank you for any hint ]:) > > > > You may wish to try a 5.5-beta snapshot which will improve the single-core > performance slightly, although the significant improvement of distributing > across all cores it not yet available. > Dear Chris, thank you for your kind reply, I will try ftp://ftp.openbsd.org//pub/OpenBSD/snapshots/sparc64/install55.iso right now. Other than simply running it is there anything else I should look at, or any new command line tool to play around? Thank you. -- === Eduardo Meyer pessoal: dudu.me...@gmail.com profissional: ddm.farmac...@saude.gov.br
Re: OpenBSD as a router on Oracle T5120
Eduardo Meyer [dudu.me...@gmail.com] wrote: > hello, > > I am doing some basic testings on the above mentioned scenario and I am > stuck on some limits which I consider to be very low: I cannot get more > than 27Kpps and 200Mbit/s routing performance without starting to loose > packets. > > System is: > > # uname -srm > > OpenBSD 5.4 sparc64 ... > CPU00: 0.0% user, 0.0% nice, 0.0% system, 98.2% interrupt, 1.8% idle ... > CPU30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > > All my NICs are getting interrupted on CPU0. > OpenBSD doesn't yet support any other mode of operation, although you may be seeing improvements here shortly. > Thank you for any hint ]:) > You may wish to try a 5.5-beta snapshot which will improve the single-core performance slightly, although the significant improvement of distributing across all cores it not yet available.
OpenBSD as a router on Oracle T5120
hello, I am doing some basic testings on the above mentioned scenario and I am stuck on some limits which I consider to be very low: I cannot get more than 27Kpps and 200Mbit/s routing performance without starting to loose packets. System is: # uname -srm OpenBSD 5.4 sparc64 # sysctl hw hw.machine=sparc64 hw.model=SUNW,UltraSPARC-T2 (rev 0.0) @ 1415.103 MHz hw.ncpu=32 hw.byteorder=4321 hw.pagesize=8192 hw.disknames=sd0:dc8022901cadee32,sd1:,cd0: hw.diskcount=3 hw.cpuspeed=1415 hw.vendor=Sun hw.product=SUNW,SPARC-Enterprise-T5120 hw.physmem=8455716864 hw.usermem=8455700480 hw.ncpufound=32 hw.allowpowerdown=1 No tuning, and no firewall to (pfctl -d). I am routing from em0 to em1 but also tried from em0 to em5 and em4 with em5 mixing onboard and PCI ports and results are the very same. Output from top points the bottleneck: load averages: 0.17, 0.21, 0.12 bgp.newtelecom.net.br18:06:20 9 processes: 8 idle, 1 on processor CPU00: 0.0% user, 0.0% nice, 0.0% system, 98.2% interrupt, 1.8% idle CPU01: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU02: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU03: 0.2% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.8% idle CPU04: 0.2% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.6% idle CPU05: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU06: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU07: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU08: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU09: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU10: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU12: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU14: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU16: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU18: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU20: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU21: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU22: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU24: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU25: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU26: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU27: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU28: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU29: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle All my NICs are getting interrupted on CPU0. All 6 network cards are Intel 82571EB which support MSI-X and should, in theory support IRQ balance. So my question is, is there anything I can do to allow OpenBSD use more than one CPU or at least choose which CPU will be used for each NIC? What other tunings and settings and tweaks should I look for? Is this performance expected to be so low on this machine? I got much better numbers w/ OpenBSD on i386 servers. Thank you for any hint ]:) -- === Eduardo Meyer
