Re: IPSEC VPN performance
On Mon, Oct 1, 2012 at 5:55 PM, Russell Garrison russell.garri...@gmail.com wrote: Is iPerf running threaded? What about dd to null and a loopback listener? Beware: only -current (since Tue Sep 25) net/iperf port has threading enabled. ciao, David
Re: IPSEC VPN performance
On 2 October 2012 08:57, David Coppa dco...@gmail.com wrote: On Mon, Oct 1, 2012 at 5:55 PM, Russell Garrison russell.garri...@gmail.com wrote: Is iPerf running threaded? What about dd to null and a loopback listener? Beware: only -current (since Tue Sep 25) net/iperf port has threading enabled. ciao, David Why not using tcpbench where you can actually specify the parameters and know what is going on :). Play with buffer sizes and you'll see a big difference, using -u will give you the actual PPS.
Re: IPSEC VPN performance
On Tue, Oct 02, 2012 at 09:59:05AM +0200, Christiano F. Haesbaert wrote: Why not using tcpbench where you can actually specify the parameters and know what is going on :). Play with buffer sizes and you'll see a big difference, using -u will give you the actual PPS. I agree with this. Also, if you want to compare with other people's you should use the same tools and specific settings such as buffer sizes. Otherwise, no point in comparing and just decide on your own if 600Mbps with netcat is good enough for you. As I mentiend in http://marc.info/?l=openbsd-miscm=134033767126930, I tested with tcpbench -B 262144 -S 262144 -n 10
Re: IPSEC VPN performance
On Tue, Oct 2, 2012 at 9:59 AM, Christiano F. Haesbaert haesba...@haesbaert.org wrote: Why not using tcpbench where you can actually specify the parameters and know what is going on :). Play with buffer sizes and you'll see a big difference, using -u will give you the actual PPS. I agree, I stopped using Iperf after tcpbench was in our tree and ready (I think it was at n2k8). Nice tool. While Iperf and tcpbench are good for testing the single- or even multi-TCP stream performance of the local test systems A and B, I wouldn't really count on them to test the real routing performance of a Device-Under-Test C in the middle. It is really hard to get meaningful max. PPS numbers, especially when you want to max out Gigabit or start playing with 10G. There will always be the limitations of the software and network stack of the test systems that will have difficulties to generate enough PPS to threaten a modern OpenBSD router (OK, IPsec is a different story...). A normal OpenBSD router does not involve any networking in userland which makes it MUCH faster than anything you can test with these tools. Of course, you can use many hosts on the A side or some fancy kernel-based packet generators, but this still doesn't give you any numbers because you will have to receive the packets and analyze the results somewhere on the B side... (and you simply cannot rely on systat if running on the OpenBSD router for that - another very basic but non-satisfying workaround would be to look at the performance counters of a managed switch in the middle). Most network and security vendors and larger data centers use these insanely expensive appliances for network performance testing that use FPGAs and customs chips to handle the load and give you accurate numbers. Many other vendors just depend on software testing, lie, round up or just make up numbers. These appliances can even test IPsec performance with thousands of simulated tunnels and/or millions of PPS and max. Mbps. We used to have an Ixia in my former company and it really helped to find and eliminate some bottlenecks in OpenBSD. We also tested IPsec performance on amd64, but this was before AES-NI and iked and I don't remember the numbers. Pure routing performance could go up to around 9Gbps on fast servers, but only with larger packets (1k-1.5k, not counting jumbos) because the max. PPS in OpenBSD was magically limited at this point (again, this is almost two years ago and many improvements happened afterwards). I would be very interested in getting updated numbers but I don't have access to such an appliance anymore. In summary, it is fine to run Iperf/tcpbench for getting an idea about your router performance up to a few hundred Mbps, but these numbers are not perfect and can go totally wrong when you reach Gigabit or 10G. Reyk
Re: IPSEC VPN performance
I just reran the test again. I still receive about 600Mbps using iPerf however using client # dd if=/dev/zero bs=1000 count=100 | nc -v 172.16.2.2 12345 server # nc -v -l 12345 /dev/null I get numbers around 350Mbps. I tend to think iPerf is more reliable in this situation. Any ideas why the tests vary so much? -Jim On 9/28/12 9:18 PM, Ryan McBride wrote: 600Mbps seems about right, I tested a pair of E5649-based boxes to 550Mbps last year (with aes-128-gcm): http://marc.info/?l=openbsd-miscm=134033767126930 You'll probably get slightly more than 600 with with multiple TCP streams. Assuming PF was enabled for your test (the default configuration), the performance should be about the same with a proper ruleset. Traffic for existing states won't hit the ruleset at all. On Fri, Sep 28, 2012 at 06:39:14PM -0400, Jim Miller wrote: Yes. Let me double check everything again on Monday. Keep in mind that all devices had 1Gb ethernet interfaces and everything was directly cabled. No pf rules either. w/o ipsec I could get 900mbps through the openbsd boxes. Now you've got me thinking I need to recheck everything. -Jim On 9/28/12 5:19 PM, Hrvoje Popovski wrote: Hi, On 28.9.2012 22:09, Jim Miller wrote: So using another Mac w/ 1Gb ethernet adapter to a Linux box w/ 1Gb eth I was able to achieve approx. 600Mbps performance through the test setup (via iperf and my dd method). 600Mbps via ipsec between two Intel E31220 ?
Re: IPSEC VPN performance
On Mon, Oct 01, 2012 at 11:20:06AM -0400, Jim Miller wrote: I just reran the test again. I still receive about 600Mbps using iPerf however using client # dd if=/dev/zero bs=1000 count=100 | nc -v 172.16.2.2 12345 server # nc -v -l 12345 /dev/null I get numbers around 350Mbps. I tend to think iPerf is more reliable in this situation. Any ideas why the tests vary so much? I suspect nc does less efficient buffering. -Otto -Jim On 9/28/12 9:18 PM, Ryan McBride wrote: 600Mbps seems about right, I tested a pair of E5649-based boxes to 550Mbps last year (with aes-128-gcm): http://marc.info/?l=openbsd-miscm=134033767126930 You'll probably get slightly more than 600 with with multiple TCP streams. Assuming PF was enabled for your test (the default configuration), the performance should be about the same with a proper ruleset. Traffic for existing states won't hit the ruleset at all. On Fri, Sep 28, 2012 at 06:39:14PM -0400, Jim Miller wrote: Yes. Let me double check everything again on Monday. Keep in mind that all devices had 1Gb ethernet interfaces and everything was directly cabled. No pf rules either. w/o ipsec I could get 900mbps through the openbsd boxes. Now you've got me thinking I need to recheck everything. -Jim On 9/28/12 5:19 PM, Hrvoje Popovski wrote: Hi, On 28.9.2012 22:09, Jim Miller wrote: So using another Mac w/ 1Gb ethernet adapter to a Linux box w/ 1Gb eth I was able to achieve approx. 600Mbps performance through the test setup (via iperf and my dd method). 600Mbps via ipsec between two Intel E31220 ?
Re: IPSEC VPN performance
Perhaps the pipe size causes degradations, I seem to recall getting better results on benchmarks without pipes. Den 1 okt 2012 18:07 skrev Otto Moerbeek o...@drijf.net: On Mon, Oct 01, 2012 at 11:20:06AM -0400, Jim Miller wrote: I just reran the test again. I still receive about 600Mbps using iPerf however using client # dd if=/dev/zero bs=1000 count=100 | nc -v 172.16.2.2 12345 server # nc -v -l 12345 /dev/null I get numbers around 350Mbps. I tend to think iPerf is more reliable in this situation. Any ideas why the tests vary so much? I suspect nc does less efficient buffering. -Otto -Jim On 9/28/12 9:18 PM, Ryan McBride wrote: 600Mbps seems about right, I tested a pair of E5649-based boxes to 550Mbps last year (with aes-128-gcm): http://marc.info/?l=openbsd-miscm=134033767126930 You'll probably get slightly more than 600 with with multiple TCP streams. Assuming PF was enabled for your test (the default configuration), the performance should be about the same with a proper ruleset. Traffic for existing states won't hit the ruleset at all. On Fri, Sep 28, 2012 at 06:39:14PM -0400, Jim Miller wrote: Yes. Let me double check everything again on Monday. Keep in mind that all devices had 1Gb ethernet interfaces and everything was directly cabled. No pf rules either. w/o ipsec I could get 900mbps through the openbsd boxes. Now you've got me thinking I need to recheck everything. -Jim On 9/28/12 5:19 PM, Hrvoje Popovski wrote: Hi, On 28.9.2012 22:09, Jim Miller wrote: So using another Mac w/ 1Gb ethernet adapter to a Linux box w/ 1Gb eth I was able to achieve approx. 600Mbps performance through the test setup (via iperf and my dd method). 600Mbps via ipsec between two Intel E31220 ?
Re: IPSEC VPN performance
Thus said Jim Miller on Mon, 01 Oct 2012 11:20:06 EDT: # dd if=/dev/zero bs=1000 count=100 | nc -v 172.16.2.2 12345 What if you try a different bs? $ dd if=/dev/zero bs=1000 count=100 /dev/null 100+0 records in 100+0 records out 10 bytes transferred in 1.102 secs (907004798 bytes/sec) vs $ dd if=/dev/zero bs=1 count=10 /dev/null 10+0 records in 10+0 records out 10 bytes transferred in 0.163 secs (6112058480 bytes/sec) That looks like an order of magnitude to me... not sure what you'll get with client/server over the network, but can't hurt to try. Andy
Re: IPSEC VPN performance
On Thu, Sep 27, 2012 at 05:30:38PM -0400, Jim Miller wrote: Hi, I'm trying to determine if the performance I'm seeing between two OpenBSD 5.1 IPSEC VPN endpoints is typical (or expected). I recognize there are quite a few variables to consider and I'm sure I've not toggled each one but I could use a sanity check regardless. Question: With the configuration below when I disable ipsec I can route traffic between the two hosts (hosts A and B) at about 900mbps. When I add the VPN I am getting speeds of approx. 40mbps. The CPU load on the OpenBSD boxes spikes to about 80% on one of the cores but the other 3 are essentially unaffected. Enabling/Disabling AES-NI in the bios doesn't seem to actually do anything as the cpu message in dmesg still shows the AES flag. The test I'm using is this Host A: # nc -v -l 12345 | /dev/null Host B: # dd if=/dev/zero bs=1000 count=1 | nc -v host a 12345 The reason these performance numbers are concerning to me is that I wanted a solution that would allow me to get decent (a.k.a. 100mbps +/- 10%) without having to buy expensive cisco/juniper devices. I would start playing with different modes, to see if that makes a difference. It could very well be that AES-NI is only used in certain modes. Start with the iked defaults for a start. Am I dreaming or have others had better performance? Also, any recent data on AES-NI optimizations would be helpful. Thanks Jim Hardware Configuration: - (2) identical SuperMicro systems with quad core E31220 w/ AES-NI enabled amd64 or i386? Why strip info from dmesg? It *might* mkae a difference. -Otto cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF cpu1: .. cpu2: ... cpu3: ... - 2GB ram - AES-NI enabled in bios - (4) Intel PRO/1000 MT (82574L) Software Configuration: VPN A /etc/iked.conf ikev2 active esp \ from 172.16.1.0/24 to 172.16.2.0/24 \ local 10.0.0.1 peer 10.0.0.2 \ ikesa enc aes-256 auth hmac-sha2-512 group modp4096 \ childsa enc aes-256-gmac \ psk helpmeplease VPN B (reverse of A config) Host A - 172.16.1.2 (behind VPN A) Host B- 172.16.2.2 (behind VPN B) VPN A (10.0.0.1) talks to B (10.0.0.2) via a crossover cable. No switches/routers/hubs/etc in this test system. All hosts running linux with 1000mb phys.
Re: IPSEC VPN performance
On Thu, Sep 27, 2012 at 11:30 PM, Jim Miller jmil...@sri-inc.com wrote: Hi, I'm trying to determine if the performance I'm seeing between two OpenBSD 5.1 IPSEC VPN endpoints is typical (or expected). I recognize there are quite a few variables to consider and I'm sure I've not toggled each one but I could use a sanity check regardless. Question: With the configuration below when I disable ipsec I can route traffic between the two hosts (hosts A and B) at about 900mbps. When I add the VPN I am getting speeds of approx. 40mbps. The CPU load on the OpenBSD boxes spikes to about 80% on one of the cores but the other 3 are essentially unaffected. Enabling/Disabling AES-NI in the bios doesn't seem to actually do anything as the cpu message in dmesg still shows the AES flag. The test I'm using is this Host A: # nc -v -l 12345 | /dev/null Host B: # dd if=/dev/zero bs=1000 count=1 | nc -v host a 12345 The reason these performance numbers are concerning to me is that I wanted a solution that would allow me to get decent (a.k.a. 100mbps +/- 10%) without having to buy expensive cisco/juniper devices. Am I dreaming or have others had better performance? Also, any recent data on AES-NI optimizations would be helpful. Thanks Jim Hardware Configuration: - (2) identical SuperMicro systems with quad core E31220 w/ AES-NI enabled cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF cpu1: .. cpu2: ... cpu3: ... - 2GB ram - AES-NI enabled in bios - (4) Intel PRO/1000 MT (82574L) Software Configuration: VPN A /etc/iked.conf ikev2 active esp \ from 172.16.1.0/24 to 172.16.2.0/24 \ local 10.0.0.1 peer 10.0.0.2 \ ikesa enc aes-256 auth hmac-sha2-512 group modp4096 \ childsa enc aes-256-gmac \ psk helpmeplease VPN B (reverse of A config) Host A - 172.16.1.2 (behind VPN A) Host B- 172.16.2.2 (behind VPN B) VPN A (10.0.0.1) talks to B (10.0.0.2) via a crossover cable. No switches/routers/hubs/etc in this test system. All hosts running linux with 1000mb phys. Hi, I have two suggestions: 1) try -current as forwarding performance was improved; 2) try aes-128-gcm for child sa (traffic encryption). aes-256-gmac-gmac means don't encrypt, just authenticate. I must say I'm curious about Xeon E3 AES-NI performance myself as we have tested only core i5, i7 and previous generation xeons, but the cpu you've picked should be the right choice. Cheers, Mike
Re: IPSEC VPN performance
On Fri, Sep 28, 2012 at 11:45 AM, Otto Moerbeek o...@drijf.net wrote: On Thu, Sep 27, 2012 at 05:30:38PM -0400, Jim Miller wrote: Hi, I'm trying to determine if the performance I'm seeing between two OpenBSD 5.1 IPSEC VPN endpoints is typical (or expected). I recognize there are quite a few variables to consider and I'm sure I've not toggled each one but I could use a sanity check regardless. Question: With the configuration below when I disable ipsec I can route traffic between the two hosts (hosts A and B) at about 900mbps. When I add the VPN I am getting speeds of approx. 40mbps. The CPU load on the OpenBSD boxes spikes to about 80% on one of the cores but the other 3 are essentially unaffected. Enabling/Disabling AES-NI in the bios doesn't seem to actually do anything as the cpu message in dmesg still shows the AES flag. The test I'm using is this Host A: # nc -v -l 12345 | /dev/null Host B: # dd if=/dev/zero bs=1000 count=1 | nc -v host a 12345 The reason these performance numbers are concerning to me is that I wanted a solution that would allow me to get decent (a.k.a. 100mbps +/- 10%) without having to buy expensive cisco/juniper devices. I would start playing with different modes, to see if that makes a difference. It could very well be that AES-NI is only used in certain modes. Start with the iked defaults for a start. aes-ni is used for all aes-related modes (aes-cbc, aes-ctr, aes-gcm and aes-gmac)... on amd64. Am I dreaming or have others had better performance? Also, any recent data on AES-NI optimizations would be helpful. Thanks Jim Hardware Configuration: - (2) identical SuperMicro systems with quad core E31220 w/ AES-NI enabled amd64 or i386? Why strip info from dmesg? It *might* mkae a difference. wow. it definitely makes a difference: aes-ni is not supported on i386. -Otto
Re: IPSEC VPN performance
On 2012 Sep 27 (Thu) at 17:30:38 -0400 (-0400), Jim Miller wrote: :Hardware Configuration: :- (2) identical SuperMicro systems with quad core E31220 w/ AES-NI enabled : :cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) :3.10 GHz :cpu0: :FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF :cpu1: .. :cpu2: ... :cpu3: ... :- 2GB ram :- AES-NI enabled in bios :- (4) Intel PRO/1000 MT (82574L) : Please, for the love of everythign that is holy and non, do NOT strip any info from dmesg. We want all of it, as some parts that you think don't matter DO. In this case: the arch will make a big difference. -- Happiness isn't something you experience; it's something you remember. -- Oscar Levant
Re: IPSEC VPN performance
Sorry I was stingy on the dmesg output. Here's the full dump. I will test with other AES modes now. -Jim OpenBSD 5.1 (GENERIC.MP) #188: Sun Feb 12 09:55:11 MST 2012 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC.MP cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF real mem = 2119032832 (2020MB) avail mem = 2074247168 (1978MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 12/22/11, SMBIOS rev. 2.7 @ 0xeb4c0 (54 entries) bios0: vendor American Megatrends Inc. version 2.00 date 05/08/2012 bios0: Supermicro X9SCI/X9SCA acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC FPDT MCFG PRAD HPET SSDT SPMI SSDT SSDT SPCR EINJ ERST HEST BERT acpi0: wakeup devices PS2K(S4) PS2M(S4) UAR1(S4) UAR2(S4) P0P1(S4) USB1(S4) USB2(S4) USB3(S4) USB4(S4) USB5(S4) USB6(S4) USB7(S4) PXSX(S4) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) RP07(S4) PXSX(S4) RP08(S4) PEGP(S4) PEG0(S4) PEG1(S4) PEG2(S4) PEG3(S4) GLAN(S4) EHC1(S4) EHC2(S4) HDEF(S4) PWRB(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 99MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu2: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu3: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins acpimcfg0 at acpi0 addr 0xf800, bus 0-63 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 6 (P0P1) acpiprt2 at acpi0: bus 1 (RP01) acpiprt3 at acpi0: bus -1 (RP02) acpiprt4 at acpi0: bus -1 (RP03) acpiprt5 at acpi0: bus -1 (RP04) acpiprt6 at acpi0: bus 2 (RP05) acpiprt7 at acpi0: bus 3 (RP06) acpiprt8 at acpi0: bus 4 (RP07) acpiprt9 at acpi0: bus 5 (RP08) acpiprt10 at acpi0: bus -1 (PEG0) acpiprt11 at acpi0: bus -1 (PEG1) acpiprt12 at acpi0: bus -1 (PEG2) acpiprt13 at acpi0: bus -1 (PEG3) acpiec0 at acpi0: Failed to read resource settings acpicpu0 at acpi0: C3, C1, PSS acpicpu1 at acpi0: C3, C1, PSS acpicpu2 at acpi0: C3, C1, PSS acpicpu3 at acpi0: C3, C1, PSS acpipwrres0 at acpi0: FN00 acpipwrres1 at acpi0: FN01 acpipwrres2 at acpi0: FN02 acpipwrres3 at acpi0: FN03 acpipwrres4 at acpi0: FN04 acpitz0 at acpi0: critical temperature is 95 degC acpitz1 at acpi0: critical temperature is 95 degC acpibat0 at acpi0: BAT0 not present acpibat1 at acpi0: BAT1 not present acpibat2 at acpi0: BAT2 not present acpibtn0 at acpi0: PWRB acpibtn1 at acpi0: LID0 acpivideo0 at acpi0: GFX0 acpivout0 at acpivideo0: DD02 bios0: ROM list: 0xc/0x8000 0xc8000/0x1000 0xc9000/0x1000 0xca000/0x1000 0xcb000/0x1000 ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 3093 MHz: speeds: 3101, 3100, 3000, 2900, 2800, 2700, 2600, 2500, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 Intel Xeon E3-1200 Host rev 0x09 Intel 6 Series MEI rev 0x04 at pci0 dev 22 function 0 not configured vendor Intel, unknown product 0x1c3b (class communications subclass miscellaneous, rev 0x04) at pci0 dev 22 function 1 not configured ehci0 at pci0 dev 26 function 0 Intel 6 Series USB rev 0x05: apic 2 int 16 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 ppb0 at pci0 dev 28 function 0 Intel 6 Series PCIE rev 0xb5: apic 2 int 16 pci1 at ppb0 bus 1 ppb1 at pci0 dev 28 function 4 Intel 6 Series PCIE rev 0xb5: apic 2 int 16 pci2 at ppb1 bus 2 em0 at pci2 dev 0 function 0 Intel PRO/1000 MT (82574L) rev 0x00: msi, address 00:25:90:75:91:c0 ppb2 at pci0 dev 28 function 5 Intel 6 Series PCIE rev 0xb5: apic 2 int 17 pci3 at ppb2 bus 3 em1 at pci3 dev 0 function 0 Intel
Re: IPSEC VPN performance
On Fri, Sep 28, 2012 at 08:38:37AM -0400, Jim Miller wrote: Sorry I was stingy on the dmesg output. Here's the full dump. I will test with other AES modes now. And then install amd64 ;-) -Otto -Jim OpenBSD 5.1 (GENERIC.MP) #188: Sun Feb 12 09:55:11 MST 2012 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC.MP cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF real mem = 2119032832 (2020MB) avail mem = 2074247168 (1978MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 12/22/11, SMBIOS rev. 2.7 @ 0xeb4c0 (54 entries) bios0: vendor American Megatrends Inc. version 2.00 date 05/08/2012 bios0: Supermicro X9SCI/X9SCA acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC FPDT MCFG PRAD HPET SSDT SPMI SSDT SSDT SPCR EINJ ERST HEST BERT acpi0: wakeup devices PS2K(S4) PS2M(S4) UAR1(S4) UAR2(S4) P0P1(S4) USB1(S4) USB2(S4) USB3(S4) USB4(S4) USB5(S4) USB6(S4) USB7(S4) PXSX(S4) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) RP07(S4) PXSX(S4) RP08(S4) PEGP(S4) PEG0(S4) PEG1(S4) PEG2(S4) PEG3(S4) GLAN(S4) EHC1(S4) EHC2(S4) HDEF(S4) PWRB(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 99MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu2: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz (GenuineIntel 686-class) 3.10 GHz cpu3: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,LAHF ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins acpimcfg0 at acpi0 addr 0xf800, bus 0-63 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 6 (P0P1) acpiprt2 at acpi0: bus 1 (RP01) acpiprt3 at acpi0: bus -1 (RP02) acpiprt4 at acpi0: bus -1 (RP03) acpiprt5 at acpi0: bus -1 (RP04) acpiprt6 at acpi0: bus 2 (RP05) acpiprt7 at acpi0: bus 3 (RP06) acpiprt8 at acpi0: bus 4 (RP07) acpiprt9 at acpi0: bus 5 (RP08) acpiprt10 at acpi0: bus -1 (PEG0) acpiprt11 at acpi0: bus -1 (PEG1) acpiprt12 at acpi0: bus -1 (PEG2) acpiprt13 at acpi0: bus -1 (PEG3) acpiec0 at acpi0: Failed to read resource settings acpicpu0 at acpi0: C3, C1, PSS acpicpu1 at acpi0: C3, C1, PSS acpicpu2 at acpi0: C3, C1, PSS acpicpu3 at acpi0: C3, C1, PSS acpipwrres0 at acpi0: FN00 acpipwrres1 at acpi0: FN01 acpipwrres2 at acpi0: FN02 acpipwrres3 at acpi0: FN03 acpipwrres4 at acpi0: FN04 acpitz0 at acpi0: critical temperature is 95 degC acpitz1 at acpi0: critical temperature is 95 degC acpibat0 at acpi0: BAT0 not present acpibat1 at acpi0: BAT1 not present acpibat2 at acpi0: BAT2 not present acpibtn0 at acpi0: PWRB acpibtn1 at acpi0: LID0 acpivideo0 at acpi0: GFX0 acpivout0 at acpivideo0: DD02 bios0: ROM list: 0xc/0x8000 0xc8000/0x1000 0xc9000/0x1000 0xca000/0x1000 0xcb000/0x1000 ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 3093 MHz: speeds: 3101, 3100, 3000, 2900, 2800, 2700, 2600, 2500, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 Intel Xeon E3-1200 Host rev 0x09 Intel 6 Series MEI rev 0x04 at pci0 dev 22 function 0 not configured vendor Intel, unknown product 0x1c3b (class communications subclass miscellaneous, rev 0x04) at pci0 dev 22 function 1 not configured ehci0 at pci0 dev 26 function 0 Intel 6 Series USB rev 0x05: apic 2 int 16 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 ppb0 at pci0 dev 28 function 0 Intel 6 Series PCIE rev 0xb5: apic 2 int 16 pci1 at ppb0 bus 1 ppb1 at pci0 dev 28 function 4 Intel 6 Series PCIE rev 0xb5: apic 2 int 16 pci2 at ppb1 bus 2 em0 at pci2
Re: IPSEC VPN performance
Good catch. I've since upgraded to the amd64 kernel. See the below dmesg. The performance jumped from 40mbps to approx. 70mbps. This is obviously a significant jump. I've tried switching the childsa from aes-256-gmac, aes-256-gcm, aes-128 and the times are fairly constant. I assume the AES-NI instructions are being used by the processor but I don't know for sure. Ideally I'd like to see if I could get performance up on par with a Cisco ASA 5505. I've had those devices with the same test hit 90mbps. Any ideas? Thanks everyone Jim OpenBSD 5.1 (GENERIC.MP) #207: Sun Feb 12 09:42:14 MST 2012 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2118471680 (2020MB) avail mem = 2047971328 (1953MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xeb4c0 (54 entries) bios0: vendor American Megatrends Inc. version 2.00 date 05/08/2012 bios0: Supermicro X9SCI/X9SCA acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC FPDT MCFG PRAD HPET SSDT SPMI SSDT SSDT SPCR EINJ ERST HEST BERT acpi0: wakeup devices PS2K(S4) PS2M(S4) UAR1(S4) UAR2(S4) P0P1(S4) USB1(S4) USB2(S4) USB3(S4) USB4(S4) USB5(S4) USB6(S4) USB7(S4) PXSX(S4) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) RP07(S4) PXSX(S4) RP08(S4) PEGP(S4) PEG0(S4) PEG1(S4) PEG2(S4) PEG3(S4) GLAN(S4) EHC1(S4) EHC2(S4) HDEF(S4) PWRB(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3093.40 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,NXE,LONG,LAHF cpu0: 256KB 64b/line 8-way L2 cache cpu0: apic clock running at 99MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,NXE,LONG,LAHF cpu1: 256KB 64b/line 8-way L2 cache cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,NXE,LONG,LAHF cpu2: 256KB 64b/line 8-way L2 cache cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,XSAVE,AVX,NXE,LONG,LAHF cpu3: 256KB 64b/line 8-way L2 cache ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins acpimcfg0 at acpi0 addr 0xf800, bus 0-63 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 6 (P0P1) acpiprt2 at acpi0: bus 1 (RP01) acpiprt3 at acpi0: bus -1 (RP02) acpiprt4 at acpi0: bus -1 (RP03) acpiprt5 at acpi0: bus -1 (RP04) acpiprt6 at acpi0: bus 2 (RP05) acpiprt7 at acpi0: bus 3 (RP06) acpiprt8 at acpi0: bus 4 (RP07) acpiprt9 at acpi0: bus 5 (RP08) acpiprt10 at acpi0: bus -1 (PEG0) acpiprt11 at acpi0: bus -1 (PEG1) acpiprt12 at acpi0: bus -1 (PEG2) acpiprt13 at acpi0: bus -1 (PEG3) acpiec0 at acpi0: Failed to read resource settings acpicpu0 at acpi0: C3, C1, PSS acpicpu1 at acpi0: C3, C1, PSS acpicpu2 at acpi0: C3, C1, PSS acpicpu3 at acpi0: C3, C1, PSS acpipwrres0 at acpi0: FN00 acpipwrres1 at acpi0: FN01 acpipwrres2 at acpi0: FN02 acpipwrres3 at acpi0: FN03 acpipwrres4 at acpi0: FN04 acpitz0 at acpi0: critical temperature is 95 degC acpitz1 at acpi0: critical temperature is 95 degC acpibat0 at acpi0: BAT0 not present acpibat1 at acpi0: BAT1 not present acpibat2 at acpi0: BAT2 not present acpibtn0 at acpi0: PWRB acpibtn1 at acpi0: LID0 acpivideo0 at acpi0: GFX0 acpivout0 at acpivideo0: DD02 ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 3092 MHz: speeds: 3101, 3100, 3000, 2900, 2800, 2700, 2600, 2500, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 Intel Xeon E3-1200 Host rev 0x09 Intel 6 Series MEI rev 0x04 at pci0 dev 22 function 0 not configured vendor Intel, unknown product 0x1c3b (class communications subclass miscellaneous, rev 0x04) at pci0 dev 22 function 1 not configured ehci0 at pci0 dev 26 function 0 Intel 6 Series USB rev 0x05: apic 2 int 16 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 Intel EHCI root hub rev 2.00/1.00 addr 1 ppb0 at pci0 dev 28 function 0 Intel 6 Series PCIE rev 0xb5: msi
Re: IPSEC VPN performance
Jim Miller jmil...@sri-inc.com wrote: The test I'm using is this Host A: # nc -v -l 12345 | /dev/null Host B: # dd if=/dev/zero bs=1000 count=1 | nc -v host a 12345 I increased the count a bit: 10 bytes transferred in 53.265 secs (18773882 bytes/sec) That's with AES-256-GCM between two Sandy Bridge Xeons (Intel Xeon CPU E5-2637 @ 3.00GHz), i.e., with AES-NI, running OpenBSD-current/amd64. -- Christian naddy Weisgerber na...@mips.inka.de
Re: IPSEC VPN performance
So I just realized another serious flaw in my testing. I was using a Mac Air w/ USB 100Mb ethernet adapter for one of the hosts behind the OpenBSD VPN devices. And it must have been limiting the speed more than I thought. So using another Mac w/ 1Gb ethernet adapter to a Linux box w/ 1Gb eth I was able to achieve approx. 600Mbps performance through the test setup (via iperf and my dd method). Still it baffles me as to why the ASA 5505 performed better with the Mac Air's USB 100mbps connection than the OpenVPN boxes. The ASA was able to do approx 88mbps while I never got above 72mbps on the OpenBSD test. Either way, case closed. I'd say that's fast enough. Lessons' learned: - Use the amd64 kernel not i386 - w/ AES-NI enabled AES-256-GMAC, AES-256-GCM, AES-128 all performed about the same - For some reason on my supermicro board disabling AES-NI doesn't have an effect as OpenBSD still seems to find the instructions - Don't use USB for testing performance. ;) Thanks to all that helped. -Jim On 9/28/12 3:10 PM, Christian Weisgerber wrote: Jim Miller jmil...@sri-inc.com wrote: The test I'm using is this Host A: # nc -v -l 12345 | /dev/null Host B: # dd if=/dev/zero bs=1000 count=1 | nc -v host a 12345 I increased the count a bit: 10 bytes transferred in 53.265 secs (18773882 bytes/sec) That's with AES-256-GCM between two Sandy Bridge Xeons (Intel Xeon CPU E5-2637 @ 3.00GHz), i.e., with AES-NI, running OpenBSD-current/amd64.
Re: IPSEC VPN performance
Hi, On 28.9.2012 22:09, Jim Miller wrote: So using another Mac w/ 1Gb ethernet adapter to a Linux box w/ 1Gb eth I was able to achieve approx. 600Mbps performance through the test setup (via iperf and my dd method). 600Mbps via ipsec between two Intel E31220 ?
Re: IPSEC VPN performance
Yes. Let me double check everything again on Monday. Keep in mind that all devices had 1Gb ethernet interfaces and everything was directly cabled. No pf rules either. w/o ipsec I could get 900mbps through the openbsd boxes. Now you've got me thinking I need to recheck everything. -Jim On 9/28/12 5:19 PM, Hrvoje Popovski wrote: Hi, On 28.9.2012 22:09, Jim Miller wrote: So using another Mac w/ 1Gb ethernet adapter to a Linux box w/ 1Gb eth I was able to achieve approx. 600Mbps performance through the test setup (via iperf and my dd method). 600Mbps via ipsec between two Intel E31220 ?
Re: IPSEC VPN performance
600Mbps seems about right, I tested a pair of E5649-based boxes to 550Mbps last year (with aes-128-gcm): http://marc.info/?l=openbsd-miscm=134033767126930 You'll probably get slightly more than 600 with with multiple TCP streams. Assuming PF was enabled for your test (the default configuration), the performance should be about the same with a proper ruleset. Traffic for existing states won't hit the ruleset at all. On Fri, Sep 28, 2012 at 06:39:14PM -0400, Jim Miller wrote: Yes. Let me double check everything again on Monday. Keep in mind that all devices had 1Gb ethernet interfaces and everything was directly cabled. No pf rules either. w/o ipsec I could get 900mbps through the openbsd boxes. Now you've got me thinking I need to recheck everything. -Jim On 9/28/12 5:19 PM, Hrvoje Popovski wrote: Hi, On 28.9.2012 22:09, Jim Miller wrote: So using another Mac w/ 1Gb ethernet adapter to a Linux box w/ 1Gb eth I was able to achieve approx. 600Mbps performance through the test setup (via iperf and my dd method). 600Mbps via ipsec between two Intel E31220 ?
