Hi, we recently found that the switch to constant-time AES has quite a heavy impact on IPsec performance. But since according to CVS that was part of OpenBSD 6.2 already, it's probably something else.
https://github.com/openbsd/src/commit/d223d7cb85c1f2f705da547a0134b949655abe6a Patrick On Wed, Jul 17, 2019 at 12:26:31PM +0000, [email protected] wrote: > Hello, > > I'm currently doing some IPsec performance testing between OpenBSD 6.3 and > 6.5. > Dmesg and ipsec.conf is below for information. > > Testing with iperf3 and 1500B packets, throughput drops around 1/3, from 919 > Mbps to 623 Mbps. > I also tried 6.4, which has similar perfomance to 6.5. > I went through plus64.html without finding a change that could explain this. > > > Could someone explain me what caused such a performance drop ? > Is there any solutions or plans to get the original performance back ? > > Thank you > > > root@bsdWAN ~ # cat /etc/ipsec.conf > # Conf transport > ike esp transport proto gre \ > from 192.168.3.254 to 192.168.3.1 peer 192.168.3.1 \ > main auth hmac-sha2-256 enc aes-256 group modp1024 lifetime 86400 \ > quick auth hmac-sha2-256 enc aes-256 group modp1024 lifetime 28800 \ > psk "mekmitasdigoat" > > root@bsdWAN ~ # dmesg > OpenBSD 6.5 (GENERIC.MP) #2: Tue May 14 10:19:35 UTC 2019 > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 8395776000 (8006MB) > avail mem = 8131694592 (7754MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x8ef68000 (45 entries) > bios0: vendor Dell Inc. version "1.4.5" date 08/09/2016 > bios0: Dell Inc. PowerEdge R330 > acpi0 at bios0: rev 2 > acpi0: sleep states S0 S5 > acpi0: tables DSDT FACP BOOT SSDT SLIC HPET LPIT APIC MCFG WDAT SSDT DBGP > DBG2 SSDT SSDT SSDT SSDT SSDT SSDT PRAD HEST BERT ERST EINJ DMAR FPDT > acpi0: wakeup devices PEGP(S0) PEG0(S0) PEGP(S0) PEG1(S0) PEGP(S0) PEG2(S0) > XHC_(S0) XDCI(S0) PXSX(S0) RP01(S0) PXSX(S0) RP02(S0) PXSX(S0) RP03(S0) > PXSX(S0) RP04(S0) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpihpet0 at acpi0: 23999999 Hz > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3293.54 MHz, 06-5e-03 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges > cpu0: apic clock running at 24MHz > cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE > cpu1 at mainbus0: apid 2 (application processor) > cpu1: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 4 (application processor) > cpu2: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 > cpu2: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu2: 256KB 64b/line 8-way L2 cache > cpu2: smt 0, core 2, package 0 > cpu3 at mainbus0: apid 6 (application processor) > cpu3: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 > cpu3: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu3: 256KB 64b/line 8-way L2 cache > cpu3: smt 0, core 3, package 0 > ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins > acpimcfg0 at acpi0 > acpimcfg0: addr 0xe0000000, bus 0-255 > acpiprt0 at acpi0: bus 0 (PCI0) > acpiprt0: no apic found for irq 32 > acpiprt0: no apic found for irq 33 > acpiprt0: no apic found for irq 34 > acpiprt1 at acpi0: bus 1 (PEG0) > acpiprt2 at acpi0: bus 2 (PEG1) > acpiprt3 at acpi0: bus 3 (PEG2) > acpiprt4 at acpi0: bus -1 (RP01) > acpiprt5 at acpi0: bus -1 (RP02) > acpiprt6 at acpi0: bus -1 (RP03) > acpiprt7 at acpi0: bus -1 (RP04) > acpiprt8 at acpi0: bus -1 (RP05) > acpiprt9 at acpi0: bus -1 (RP06) > acpiprt10 at acpi0: bus -1 (RP07) > acpiprt11 at acpi0: bus -1 (RP08) > acpiprt12 at acpi0: bus 4 (RP09) > acpiprt13 at acpi0: bus -1 (RP10) > acpiprt14 at acpi0: bus 5 (RP11) > acpiprt15 at acpi0: bus -1 (RP12) > acpiprt16 at acpi0: bus -1 (RP13) > acpiprt17 at acpi0: bus -1 (RP14) > acpiprt18 at acpi0: bus -1 (RP15) > acpiprt19 at acpi0: bus -1 (RP16) > acpiprt20 at acpi0: bus -1 (RP17) > acpiprt21 at acpi0: bus -1 (RP18) > acpiprt22 at acpi0: bus -1 (RP19) > acpiprt23 at acpi0: bus -1 (RP20) > acpicpu0 at acpi0: C1(@1 halt!) > acpicpu1 at acpi0: C1(@1 halt!) > acpicpu2 at acpi0: C1(@1 halt!) > acpicpu3 at acpi0: C1(@1 halt!) > acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001 > acpicmos0 at acpi0 > acpibtn0 at acpi0: SLPB > "PNP0C14" at acpi0 not configured > "PNP0C33" at acpi0 not configured > acpivideo0 at acpi0: GFX0 > acpivout0 at acpivideo0: DD1F > ipmi at mainbus0 not configured > memory map conflict 0xe00fd000/0x1000 > memory map conflict 0xfe000000/0x11000 > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 v5 Host" rev 0x07 > ppb0 at pci0 dev 1 function 0 "Intel Core 6G PCIE" rev 0x07: msi > pci1 at ppb0 bus 1 > 1:0:0: mem address conflict 0xfff80000/0x80000 > 1:0:1: mem address conflict 0xfff80000/0x80000 > ix0 at pci1 dev 0 function 0 "Intel X540T" rev 0x01: msi, address > a0:36:9f:cb:00:fc > ix1 at pci1 dev 0 function 1 "Intel X540T" rev 0x01: msi, address > a0:36:9f:cb:00:fe > ppb1 at pci0 dev 1 function 1 "Intel Core 6G PCIE" rev 0x07: msi > pci2 at ppb1 bus 2 > 2:0:0: mem address conflict 0xfffc0000/0x40000 > 2:0:1: mem address conflict 0xfffc0000/0x40000 > 2:0:2: mem address conflict 0xfffc0000/0x40000 > 2:0:3: mem address conflict 0xfffc0000/0x40000 > bge0 at pci2 dev 0 function 0 "Broadcom BCM5719" rev 0x01, unknown BCM5719 > (0x5719001), APE firmware NCSI 1.3.16.0: msi, address 00:0a:f7:9b:90:04 > brgphy0 at bge0 phy 1: BCM5719C 10/100/1000baseT PHY, rev. 0 > bge1 at pci2 dev 0 function 1 "Broadcom BCM5719" rev 0x01, unknown BCM5719 > (0x5719001), APE firmware NCSI 1.3.16.0: msi, address 00:0a:f7:9b:90:05 > brgphy1 at bge1 phy 2: BCM5719C 10/100/1000baseT PHY, rev. 0 > bge2 at pci2 dev 0 function 2 "Broadcom BCM5719" rev 0x01, unknown BCM5719 > (0x5719001), APE firmware NCSI 1.3.16.0: msi, address 00:0a:f7:9b:90:06 > brgphy2 at bge2 phy 3: BCM5719C 10/100/1000baseT PHY, rev. 0 > bge3 at pci2 dev 0 function 3 "Broadcom BCM5719" rev 0x01, unknown BCM5719 > (0x5719001), APE firmware NCSI 1.3.16.0: msi, address 00:0a:f7:9b:90:07 > brgphy3 at bge3 phy 4: BCM5719C 10/100/1000baseT PHY, rev. 0 > ppb2 at pci0 dev 1 function 2 "Intel Core 6G PCIE" rev 0x07: msi > pci3 at ppb2 bus 3 > mfii0 at pci3 dev 0 function 0 "Symbios Logic MegaRAID SAS3008" rev 0x02: msi > mfii0: "PERC H330 Adapter", firmware 25.5.0.0019 > scsibus1 at mfii0: 32 targets > sd0 at scsibus1 targ 0 lun 0: <DELL, PERC H330 Adp, 4.27> SCSI3 0/direct > fixed naa.6847beb0d82d32001feaa67a21d08db4 > sd0: 190208MB, 512 bytes/sector, 389545984 sectors > scsibus2 at mfii0: 256 targets > xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x31: msi, xHCI > 1.0 > usb0 at xhci0: USB revision 3.0 > uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 > addr 1 > pchtemp0 at pci0 dev 20 function 2 "Intel 100 Series Thermal" rev 0x31 > "Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 0 not configured > "Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 1 not configured > ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x31: msi, AHCI > 1.3.1 > ahci0: port 4: 1.5Gb/s > scsibus3 at ahci0: 32 targets > cd0 at scsibus3 targ 4 lun 0: <HL-DT-ST, DVD+-RW GU90N, A3C0> ATAPI 5/cdrom > removable > ppb3 at pci0 dev 29 function 0 "Intel 100 Series PCIE" rev 0xf1: msi > pci4 at ppb3 bus 4 > 4:0:0: mem address conflict 0xfffc0000/0x40000 > 4:0:1: mem address conflict 0xfffc0000/0x40000 > bge4 at pci4 dev 0 function 0 "Broadcom BCM5720" rev 0x00, BCM5720 A0 > (0x5720000), APE firmware NCSI 1.3.16.0: msi, address 10:98:36:a9:dc:99 > brgphy4 at bge4 phy 1: BCM5720C 10/100/1000baseT PHY, rev. 0 > bge5 at pci4 dev 0 function 1 "Broadcom BCM5720" rev 0x00, BCM5720 A0 > (0x5720000), APE firmware NCSI 1.3.16.0: msi, address 10:98:36:a9:dc:9a > brgphy5 at bge5 phy 2: BCM5720C 10/100/1000baseT PHY, rev. 0 > ppb4 at pci0 dev 29 function 2 "Intel 100 Series PCIE" rev 0xf1: msi > pci5 at ppb4 bus 5 > ppb5 at pci5 dev 0 function 0 "Renesas SH7758 PCIE Switch" rev 0x00 > pci6 at ppb5 bus 6 > ppb6 at pci6 dev 0 function 0 "Renesas SH7758 PCIE Switch" rev 0x00 > pci7 at ppb6 bus 7 > ppb7 at pci7 dev 0 function 0 "Renesas SH7758 PCIE-PCI" rev 0x00 > pci8 at ppb7 bus 8 > vga1 at pci8 dev 0 function 0 "Matrox MGA G200eR" rev 0x01 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > pcib0 at pci0 dev 31 function 0 "Intel C236 LPC" rev 0x31 > "Intel 100 Series PMC" rev 0x31 at pci0 dev 31 function 2 not configured > ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x31: apic 2 > int 16 > iic0 at ichiic0 > iic0: addr 0x19 00=00 01=00 02=00 03=00 04=00 05=c1 06=1c 07=22 08=00 words > 00=00ef 01=0000 02=0000 03=0000 04=0000 05=c154 06=1c85 07=2221 > isa0 at pcib0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo > pckbc0 at isa0 port 0x60/5 irq 1 irq 12 > pcppi0 at isa0 port 0x61 > spkr0 at pcppi0 > vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation) > uhub1 at uhub0 port 3 configuration 1 interface 0 "no manufacturer Gadget USB > HUB" rev 2.00/0.00 addr 2 > uhidev0 at uhub0 port 4 configuration 1 interface 0 "Avocent Dell 03R874" rev > 1.10/1.00 addr 3 > uhidev0: iclass 3/1 > ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33 > wskbd0 at ukbd0: console keyboard, using wsdisplay0 > uhidev1 at uhub0 port 4 configuration 1 interface 1 "Avocent Dell 03R874" rev > 1.10/1.00 addr 3 > uhidev1: iclass 3/1, 3 report ids > ums0 at uhidev1 reportid 1: 5 buttons, Z dir > wsmouse0 at ums0 mux 0 > uhid0 at uhidev1 reportid 2: input=2, output=0, feature=0 > uhid1 at uhidev1 reportid 3: input=1, output=0, feature=0 > vscsi0 at root > scsibus4 at vscsi0: 256 targets > softraid0 at root > scsibus5 at softraid0: 256 targets > root on sd0a (2aaef25992267da5.a) swap on sd0b dump on sd0b > > -- > Cordialement, > Pierre BARDOU > > > > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. >
