Re: Speed problems with both system openssl and security/openssl-devel
Lev Serebryakov wrote this message on Mon, Sep 17, 2018 at 23:52 +0300: > Thursday, September 13, 2018, 2:46:46 AM, you wrote: > > > Linux have openssl 1.1.0f, and I've tried both system /usr/bin/openssl > > (1.0.2p) > > and /usr/local/bin/openssl from security/openssl-devel port (1.1.0i), > > results are > > virtually the same. I have "ASM" and "SSE2" options enabled in port. > > > What happens here? Why does FreeBSD's build of openssl use AES-NI so > > inefficient? > More datapoints. > > (1) aes-256-cbc behaves really wired. Time output is > completely bogus without "-elapsed" and speed is unbelievably low with > "-elapsed". aes-256-gcm doesn't have this anomaly This is because you're likely using /dev/crypto for the operations instead of software... $openssl engine (cryptodev) BSD cryptodev engine (dynamic) Dynamic engine loading support The times below are mesured on how much cpu time openssl spent, while all the work was done in the kernel... if you disable cryptodev usage, you should see better performance... > without "-elapsed" (please note "in 0.xxs" here): > > Doing aes-256-cbc for 3s on 16 size blocks: 503555 aes-256-cbc's in 0.60s > Doing aes-256-cbc for 3s on 64 size blocks: 520386 aes-256-cbc's in 0.54s > Doing aes-256-cbc for 3s on 256 size blocks: 435106 aes-256-cbc's in 0.44s > Doing aes-256-cbc for 3s on 1024 size blocks: 242832 aes-256-cbc's in 0.38s > Doing aes-256-cbc for 3s on 8192 size blocks: 49087 aes-256-cbc's in 0.09s > ... > aes-256-cbc 13393.26k61782.64k 254599.17k 663093.25k 4289287.51k > > Doing aes-256-gcm for 3s on 16 size blocks: 12051311 aes-256-gcm's in 3.03s > Doing aes-256-gcm for 3s on 64 size blocks: 6428598 aes-256-gcm's in 3.04s > Doing aes-256-gcm for 3s on 256 size blocks: 2122316 aes-256-gcm's in 3.00s > Doing aes-256-gcm for 3s on 1024 size blocks: 610443 aes-256-gcm's in 3.13s > Doing aes-256-gcm for 3s on 8192 size blocks: 75836 aes-256-gcm's in 3.03s > ... > aes-256-gcm 63611.04k 135380.66k 181104.30k 199531.13k 204947.96k > > with "-elapsed": > > Doing aes-256-cbc for 3s on 16 size blocks: 493829 aes-256-cbc's in 3.01s > Doing aes-256-cbc for 3s on 64 size blocks: 530550 aes-256-cbc's in 3.06s > Doing aes-256-cbc for 3s on 256 size blocks: 426699 aes-256-cbc's in 3.01s > Doing aes-256-cbc for 3s on 1024 size blocks: 243305 aes-256-cbc's in 3.03s > Doing aes-256-cbc for 3s on 8192 size blocks: 48069 aes-256-cbc's in 3.01s > ... > aes-256-cbc 2626.91k11087.41k36317.07k82191.94k 130919.48k > > Doing aes-256-gcm for 3s on 16 size blocks: 12041385 aes-256-gcm's in 3.08s > Doing aes-256-gcm for 3s on 64 size blocks: 6445757 aes-256-gcm's in 3.05s > Doing aes-256-gcm for 3s on 256 size blocks: 2129499 aes-256-gcm's in 3.01s > Doing aes-256-gcm for 3s on 1024 size blocks: 587396 aes-256-gcm's in 3.01s > Doing aes-256-gcm for 3s on 8192 size blocks: 75806 aes-256-gcm's in 3.03s > ... > aes-256-gcm 62590.75k 135047.68k 181245.26k 199977.06k 204866.89k -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
Hello Lev, Thursday, September 13, 2018, 2:46:46 AM, you wrote: > Linux have openssl 1.1.0f, and I've tried both system /usr/bin/openssl > (1.0.2p) > and /usr/local/bin/openssl from security/openssl-devel port (1.1.0i), results > are > virtually the same. I have "ASM" and "SSE2" options enabled in port. > What happens here? Why does FreeBSD's build of openssl use AES-NI so > inefficient? More datapoints. (1) aes-256-cbc behaves really wired. Time output is completely bogus without "-elapsed" and speed is unbelievably low with "-elapsed". aes-256-gcm doesn't have this anomaly without "-elapsed" (please note "in 0.xxs" here): Doing aes-256-cbc for 3s on 16 size blocks: 503555 aes-256-cbc's in 0.60s Doing aes-256-cbc for 3s on 64 size blocks: 520386 aes-256-cbc's in 0.54s Doing aes-256-cbc for 3s on 256 size blocks: 435106 aes-256-cbc's in 0.44s Doing aes-256-cbc for 3s on 1024 size blocks: 242832 aes-256-cbc's in 0.38s Doing aes-256-cbc for 3s on 8192 size blocks: 49087 aes-256-cbc's in 0.09s ... aes-256-cbc 13393.26k61782.64k 254599.17k 663093.25k 4289287.51k Doing aes-256-gcm for 3s on 16 size blocks: 12051311 aes-256-gcm's in 3.03s Doing aes-256-gcm for 3s on 64 size blocks: 6428598 aes-256-gcm's in 3.04s Doing aes-256-gcm for 3s on 256 size blocks: 2122316 aes-256-gcm's in 3.00s Doing aes-256-gcm for 3s on 1024 size blocks: 610443 aes-256-gcm's in 3.13s Doing aes-256-gcm for 3s on 8192 size blocks: 75836 aes-256-gcm's in 3.03s ... aes-256-gcm 63611.04k 135380.66k 181104.30k 199531.13k 204947.96k with "-elapsed": Doing aes-256-cbc for 3s on 16 size blocks: 493829 aes-256-cbc's in 3.01s Doing aes-256-cbc for 3s on 64 size blocks: 530550 aes-256-cbc's in 3.06s Doing aes-256-cbc for 3s on 256 size blocks: 426699 aes-256-cbc's in 3.01s Doing aes-256-cbc for 3s on 1024 size blocks: 243305 aes-256-cbc's in 3.03s Doing aes-256-cbc for 3s on 8192 size blocks: 48069 aes-256-cbc's in 3.01s ... aes-256-cbc 2626.91k11087.41k36317.07k82191.94k 130919.48k Doing aes-256-gcm for 3s on 16 size blocks: 12041385 aes-256-gcm's in 3.08s Doing aes-256-gcm for 3s on 64 size blocks: 6445757 aes-256-gcm's in 3.05s Doing aes-256-gcm for 3s on 256 size blocks: 2129499 aes-256-gcm's in 3.01s Doing aes-256-gcm for 3s on 1024 size blocks: 587396 aes-256-gcm's in 3.01s Doing aes-256-gcm for 3s on 8192 size blocks: 75806 aes-256-gcm's in 3.03s ... aes-256-gcm 62590.75k 135047.68k 181245.26k 199977.06k 204866.89k (2) I've added debug output to 'crypto/evp/e_aes.c' and it shows, that AES-NIU should be used. (3) openssl111 from ports doesn't show these problems. -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
On 17.09.2018 10:40, Daniel Nebdal wrote: > Could it be relevant that the Debian binary was probably compiled with > gcc, and the FreeBSD binary with clang? Maybe. Now I'm trying to trace codepath of "openssl speed -evp aes-256-cbc" on FreeBSD to understand where and why it refuses to use AES. No much luck, though, openssl sources are very convoluted :-( > This seems like the sort of code that plausibly could bring out some > compiler corner cases. (It's weird that 1.1.1 is fine, though.) -- // Lev Serebryakov ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
On Fri, 14 Sep 2018 at 02:12, Lev Serebryakov wrote: > > Hello John, > > Friday, September 14, 2018, 1:44:13 AM, you wrote: > > >> % grep aesni ~/nanobsd/gatevay.v3/J3160 > >> device aesni > > > From my understanding of the OpenSSL code, it doesn't use the kernel driver > > at all (the kernel driver is only needed for in-kernel crypto such as IPSec > > or GELI). > It is my understanding too. > > > AESNI are just instructions that can be used in userland, and > > OpenSSL's AESNI acceleration is purely different routines in userland. > > I would verify if AESNI shows up in the CPU features in dmesg first (if it > > doesn't I'd check for a BIOS option disabling it). > It is enabled. It is used for sure by openssl 1.1.0 on Linux and bu openssl > 1.1.1 > on FreeBSD, but not by openssl 1.0.2 and 1.1.0 on FreeBSD. Problem is, > openssl 1.1.1 is not used by anything on FreeBSD (yet) and almost > everything uses system (1.0.2) and only some other ports could use 1.1.0 > from ports. > > -- > Best regards, > Levmailto:l...@freebsd.org > Could it be relevant that the Debian binary was probably compiled with gcc, and the FreeBSD binary with clang? This seems like the sort of code that plausibly could bring out some compiler corner cases. (It's weird that 1.1.1 is fine, though.) -- Daniel Nebdal ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
Hello John, Friday, September 14, 2018, 1:44:13 AM, you wrote: >> % grep aesni ~/nanobsd/gatevay.v3/J3160 >> device aesni > From my understanding of the OpenSSL code, it doesn't use the kernel driver > at all (the kernel driver is only needed for in-kernel crypto such as IPSec > or GELI). It is my understanding too. > AESNI are just instructions that can be used in userland, and > OpenSSL's AESNI acceleration is purely different routines in userland. > I would verify if AESNI shows up in the CPU features in dmesg first (if it > doesn't I'd check for a BIOS option disabling it). It is enabled. It is used for sure by openssl 1.1.0 on Linux and bu openssl 1.1.1 on FreeBSD, but not by openssl 1.0.2 and 1.1.0 on FreeBSD. Problem is, openssl 1.1.1 is not used by anything on FreeBSD (yet) and almost everything uses system (1.0.2) and only some other ports could use 1.1.0 from ports. -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
On 9/13/18 1:27 AM, Lev Serebryakov wrote: > Hello Kevin, > > Thursday, September 13, 2018, 6:32:30 AM, you wrote: > > >> This is probably not the issue, but aesni is not in the GENERIC kernel. Are >> you sure aesni.ko is loaded? >> % kldstat | grep aesni > I'm not using modules, as it is NanoBSD image build for minimal size ant > maximal efficiency. But I have aesni in my kernel config for sure: > > % grep aesni ~/nanobsd/gatevay.v3/J3160 > device aesni From my understanding of the OpenSSL code, it doesn't use the kernel driver at all (the kernel driver is only needed for in-kernel crypto such as IPSec or GELI). AESNI are just instructions that can be used in userland, and OpenSSL's AESNI acceleration is purely different routines in userland. I would verify if AESNI shows up in the CPU features in dmesg first (if it doesn't I'd check for a BIOS option disabling it). -- John Baldwin ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
Hello Dimitry, Thursday, September 13, 2018, 11:52:08 AM, you wrote: > I can't reproduce your findings, at least not on a Core i7-4790K: > type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes > FreeBSD 93454 89077 117328 281016 285456 > Ubuntu 93405 88892 114192 122346 120266 > FreeBSD-evp633283688010 700775 701168 700669 > Ubuntu-evp 623889681075 697211 700505 698460 > That was with base openssl 1.0.2p on FreeBSD 12-ALPHA5, and 1.1.0g on > Ubuntu 18.04. Here are additional numbers on J3160, with security/openssl111 port added: Lin 1.1.0f 18942 20637 21300 57967 58769 58769 Free 1.0.2p 18938 20627 21243 57940 58785 Free 1.1.0i 18929 20593 21285 58287 58766 58751 Free 1.1.1 18936 20588 21266 57923 58690 58731 Lin-evp 1.1.0f 97049 151466 183905 194385 197514 197727 Free-evp 1.0.2p 2828 10665 35364 81836 130502 Free-evp 1.1.0i 2869 10813 34932 81292 131308 137376 Free-evp 1.1.1 96959 151359 183390 194431 197076 197686 I've checked poudriere logs, all ports are built with -DAES_ASM. -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
Hello Dimitry, Thursday, September 13, 2018, 11:52:08 AM, you wrote: > I can't reproduce your findings, at least not on a Core i7-4790K: I can not reproduce it on E3-1220v3 + 11-STABLE either. But security/openssl111 works as expected on J3160 and it bothers me. Something is wrong not with hardware, but with software here. -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
On 13 Sep 2018, at 01:46, Lev Serebryakov wrote: > > I'm benchmarking new hardware (rather limited one, but still) which > supports AES-NI (Celeron J3160). > > I'm comparing simple "openssl speed aes-256-cbc" and "openssl speed -evp > aes-256-cbc" on FreeBSD 12-ALPHA4 (built by myself with all debug options > turned off) and Debian Linux 9.5.0 booted from install DVD (without > installation). > > Simple "openssl speed aes-256-cbc" shows same numbers both in > single-threaded and multi-threaded mode (for all 4 cores). Linux is > marginally faster, > but it is in the margin of measurement error. > > But "openssl speed -evp aes-256-cbc" gives me very disappointing results. > FreeBSD's openssl is WAY slower than Linux one. It is even slower than > non-evp mode for small blocks. > > Here are results (As reported by openssl, with fractions dropped): > > Lin 1894220637 21300 57967 58769 58769 > Free 1893120591 21282 58342 58731 58779 > Lin-evp 97049 151466 183905 194385 197514 197727 > Free-evp 283810845 35362 81892 131264 137579 > > Linux have openssl 1.1.0f, and I've tried both system /usr/bin/openssl > (1.0.2p) > and /usr/local/bin/openssl from security/openssl-devel port (1.1.0i), results > are > virtually the same. I have "ASM" and "SSE2" options enabled in port. > > What happens here? Why does FreeBSD's build of openssl use AES-NI so > inefficient? I can't reproduce your findings, at least not on a Core i7-4790K: type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes FreeBSD 93454 89077 117328 281016 285456 Ubuntu 93405 88892 114192 122346 120266 FreeBSD-evp633283688010 700775 701168 700669 Ubuntu-evp 623889681075 697211 700505 698460 That was with base openssl 1.0.2p on FreeBSD 12-ALPHA5, and 1.1.0g on Ubuntu 18.04. -Dimitry signature.asc Description: Message signed with OpenPGP
Re: Speed problems with both system openssl and security/openssl-devel
Hello Kevin, Thursday, September 13, 2018, 6:32:30 AM, you wrote: > This is probably not the issue, but aesni is not in the GENERIC kernel. Are > you sure aesni.ko is loaded? > % kldstat | grep aesni I'm not using modules, as it is NanoBSD image build for minimal size ant maximal efficiency. But I have aesni in my kernel config for sure: % grep aesni ~/nanobsd/gatevay.v3/J3160 device aesni % -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Speed problems with both system openssl and security/openssl-devel
On Wed, Sep 12, 2018 at 4:48 PM Lev Serebryakov wrote: > Hello Brnrd, > >I'm benchmarking new hardware (rather limited one, but still) which > supports AES-NI (Celeron J3160). > >I'm comparing simple "openssl speed aes-256-cbc" and "openssl speed -evp > aes-256-cbc" on FreeBSD 12-ALPHA4 (built by myself with all debug options > turned off) and Debian Linux 9.5.0 booted from install DVD (without > installation). > > Simple "openssl speed aes-256-cbc" shows same numbers both in > single-threaded and multi-threaded mode (for all 4 cores). Linux is > marginally faster, > but it is in the margin of measurement error. > > But "openssl speed -evp aes-256-cbc" gives me very disappointing results. > FreeBSD's openssl is WAY slower than Linux one. It is even slower than > non-evp mode for small blocks. > > Here are results (As reported by openssl, with fractions dropped): > > Lin 1894220637 21300 57967 58769 58769 > Free 1893120591 21282 58342 58731 58779 > Lin-evp 97049 151466 183905 194385 197514 197727 > Free-evp 283810845 35362 81892 131264 137579 > > Linux have openssl 1.1.0f, and I've tried both system /usr/bin/openssl > (1.0.2p) > and /usr/local/bin/openssl from security/openssl-devel port (1.1.0i), > results are > virtually the same. I have "ASM" and "SSE2" options enabled in port. > > What happens here? Why does FreeBSD's build of openssl use AES-NI so > inefficient? > > -- > Best regards, > Lev mailto:l...@freebsd.org This is probably not the issue, but aesni is not in the GENERIC kernel. Are you sure aesni.ko is loaded? % kldstat | grep aesni If not found, "kldload aesni" and add aesni_load="YES" to /boot/loader.conf -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkober...@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Speed problems with both system openssl and security/openssl-devel
Hello Brnrd, I'm benchmarking new hardware (rather limited one, but still) which supports AES-NI (Celeron J3160). I'm comparing simple "openssl speed aes-256-cbc" and "openssl speed -evp aes-256-cbc" on FreeBSD 12-ALPHA4 (built by myself with all debug options turned off) and Debian Linux 9.5.0 booted from install DVD (without installation). Simple "openssl speed aes-256-cbc" shows same numbers both in single-threaded and multi-threaded mode (for all 4 cores). Linux is marginally faster, but it is in the margin of measurement error. But "openssl speed -evp aes-256-cbc" gives me very disappointing results. FreeBSD's openssl is WAY slower than Linux one. It is even slower than non-evp mode for small blocks. Here are results (As reported by openssl, with fractions dropped): Lin 1894220637 21300 57967 58769 58769 Free 1893120591 21282 58342 58731 58779 Lin-evp 97049 151466 183905 194385 197514 197727 Free-evp 283810845 35362 81892 131264 137579 Linux have openssl 1.1.0f, and I've tried both system /usr/bin/openssl (1.0.2p) and /usr/local/bin/openssl from security/openssl-devel port (1.1.0i), results are virtually the same. I have "ASM" and "SSE2" options enabled in port. What happens here? Why does FreeBSD's build of openssl use AES-NI so inefficient? -- Best regards, Lev mailto:l...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
