Re: IPsec performance regression between 6.3 and 6.4
Hi, we recently found that the switch to constant-time AES has quite a heavy impact on IPsec performance. But since according to CVS that was part of OpenBSD 6.2 already, it's probably something else. https://github.com/openbsd/src/commit/d223d7cb85c1f2f705da547a0134b949655abe6a Patrick On Wed, Jul 17, 2019 at 12:26:31PM +, pierre1.bar...@orange.com wrote: > Hello, > > I'm currently doing some IPsec performance testing between OpenBSD 6.3 and > 6.5. > Dmesg and ipsec.conf is below for information. > > Testing with iperf3 and 1500B packets, throughput drops around 1/3, from 919 > Mbps to 623 Mbps. > I also tried 6.4, which has similar perfomance to 6.5. > I went through plus64.html without finding a change that could explain this. > > > Could someone explain me what caused such a performance drop ? > Is there any solutions or plans to get the original performance back ? > > Thank you > > > root@bsdWAN ~ # cat /etc/ipsec.conf > # Conf transport > ike esp transport proto gre \ > from 192.168.3.254 to 192.168.3.1 peer 192.168.3.1 \ > main auth hmac-sha2-256 enc aes-256 group modp1024 lifetime 86400 \ > quick auth hmac-sha2-256 enc aes-256 group modp1024 lifetime 28800 \ > psk "mekmitasdigoat" > > root@bsdWAN ~ # dmesg > OpenBSD 6.5 (GENERIC.MP) #2: Tue May 14 10:19:35 UTC 2019 > root@openbsd65.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 8395776000 (8006MB) > avail mem = 8131694592 (7754MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x8ef68000 (45 entries) > bios0: vendor Dell Inc. version "1.4.5" date 08/09/2016 > bios0: Dell Inc. PowerEdge R330 > acpi0 at bios0: rev 2 > acpi0: sleep states S0 S5 > acpi0: tables DSDT FACP BOOT SSDT SLIC HPET LPIT APIC MCFG WDAT SSDT DBGP > DBG2 SSDT SSDT SSDT SSDT SSDT SSDT PRAD HEST BERT ERST EINJ DMAR FPDT > acpi0: wakeup devices PEGP(S0) PEG0(S0) PEGP(S0) PEG1(S0) PEGP(S0) PEG2(S0) > XHC_(S0) XDCI(S0) PXSX(S0) RP01(S0) PXSX(S0) RP02(S0) PXSX(S0) RP03(S0) > PXSX(S0) RP04(S0) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpihpet0 at acpi0: 2399 Hz > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3293.54 MHz, 06-5e-03 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges > cpu0: apic clock running at 24MHz > cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE > cpu1 at mainbus0: apid 2 (application processor) > cpu1: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 4 (application processor) > cpu2: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 > cpu2: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu2: 256KB 64b/line 8-way L2 cache > cpu2: smt 0, core 2, package 0 > cpu3 at mainbus0: apid 6 (application processor) > cpu3: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 > cpu3: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu3: 256KB 64b/line 8-way L2 cache > cpu3: smt 0,
IPsec performance regression between 6.3 and 6.4
Hello, I'm currently doing some IPsec performance testing between OpenBSD 6.3 and 6.5. Dmesg and ipsec.conf is below for information. Testing with iperf3 and 1500B packets, throughput drops around 1/3, from 919 Mbps to 623 Mbps. I also tried 6.4, which has similar perfomance to 6.5. I went through plus64.html without finding a change that could explain this. Could someone explain me what caused such a performance drop ? Is there any solutions or plans to get the original performance back ? Thank you root@bsdWAN ~ # cat /etc/ipsec.conf # Conf transport ike esp transport proto gre \ from 192.168.3.254 to 192.168.3.1 peer 192.168.3.1 \ main auth hmac-sha2-256 enc aes-256 group modp1024 lifetime 86400 \ quick auth hmac-sha2-256 enc aes-256 group modp1024 lifetime 28800 \ psk "mekmitasdigoat" root@bsdWAN ~ # dmesg OpenBSD 6.5 (GENERIC.MP) #2: Tue May 14 10:19:35 UTC 2019 root@openbsd65.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8395776000 (8006MB) avail mem = 8131694592 (7754MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x8ef68000 (45 entries) bios0: vendor Dell Inc. version "1.4.5" date 08/09/2016 bios0: Dell Inc. PowerEdge R330 acpi0 at bios0: rev 2 acpi0: sleep states S0 S5 acpi0: tables DSDT FACP BOOT SSDT SLIC HPET LPIT APIC MCFG WDAT SSDT DBGP DBG2 SSDT SSDT SSDT SSDT SSDT SSDT PRAD HEST BERT ERST EINJ DMAR FPDT acpi0: wakeup devices PEGP(S0) PEG0(S0) PEGP(S0) PEG1(S0) PEGP(S0) PEG2(S0) XHC_(S0) XDCI(S0) PXSX(S0) RP01(S0) PXSX(S0) RP02(S0) PXSX(S0) RP03(S0) PXSX(S0) RP04(S0) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 2399 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3293.54 MHz, 06-5e-03 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, 3292.34 MHz, 06-5e-03 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 3, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xe000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt0: no apic found for irq 32 acpiprt0: no apic found for irq 33 acpiprt0: no apic found for irq 34 acpiprt1 at acpi0: bus 1 (PEG0) acpiprt2 at acpi0: bus 2 (PEG1) acpiprt3 at acpi0: bus 3 (PEG2) acpiprt4 at acpi0: bus -1 (RP01) acpiprt5 at acpi0: bus -1 (RP02) acpiprt6 at acpi0: bus -1 (RP03) acpiprt7 at acpi0: bus -1 (RP04) acpiprt8 at
