Re: [openssl-dev] [openssl.org #3843] OpenSSL 1.0.1* and below: incorrect use of _lrotl()
Hi Andy, Thank you for your reply! I am CC'ing Lei on mine. On Wed, May 20, 2015 at 12:55:10PM +0200, Andy Polyakov via RT wrote: For reference. icc was not cared for for quite some time. Initially it was possible for me, by then university employee, to use it, but then they changes terms and it became impossible for me to maintain it. But I've just noticed they provide some starter version of something, I'll see... Yes, this might be usable for you: https://software.intel.com/en-us/qualify-for-free-software/opensourcecontributor Intel provides select Intel Software Development Products at no cost to qualified open source contributors who are working on open source projects compliant with the Open Source Initiative (OSI). But linux-x86_64-icc is not present in and was never supported in pre-1.0.2. Oh, I didn't realize that. Like I mentioned, we're actually building with icc for MIC. When we build with icc for x86_64 host, we typically simply link against the distro's gcc-built OpenSSL, so didn't run into this issue ourselves until we started building for MIC and thus had to make our own OpenSSL build with icc. (Indeed, I've been building OpenSSL from source on many other occasions, and as part of a distro too, but that's not with icc and unrelated to JtR project.) So you ought to provide custom line. This remark doesn't mean that fix can't be backported, but out of curiosity, what's your config line? Currently, Lei put this into JtR -jumbo README-MIC: Build LibreSSL (version 2.1.6): $ cd libressl-2.1.6 $ ./configure CC=icc -mmic --host=k1om-linux --prefix=$MIC $ make make install The previous instructions were: Build OpenSSL (version 1.0.0q): $ cd openssl-1.0.0q $ patch Configure $JOHN/src/unused/openssl.patch $ ./Configure linux-mic shared --prefix=$MIC $ make make install I'm not sure what was in $JOHN/src/unused/openssl.patch - I guess it had to add linux-mic support. Lei, please reply to all. Is assembly engaged? If so, how fast is it? Or is it so that you count on compiler to produce vector code that would process multiple inputs in parallel with SIMD? We're using OpenSSL (or LibreSSL) as an easy but slower option, replacing it with our own SIMD code right in JtR tree whenever we can and where this makes sense. So we're not trying to optimize OpenSSL's code. It remains scalar and unmodified, and our use of it is just to have things working where we do not have optimized code yet or where we prefer simpler rather than faster code (such as for some lightweight precomputation in some rare cases where this makes sense). This varies by crypto primitive, but overall we currently have SIMD intrinsics code for MMX, SSE2+/AVX, XOP, AVX2, MIC/AVX-512, and for bitslice DES also for AltiVec and NEON. One thing for which we still use OpenSSL's code in performance-critical manner is SSH key passphrase cracking (which involves RSA). There are probably many more examples like this, but this is a prominent one that comes to mind. There must be a lot of room for optimization here. As to compiler auto-vectorization - no, we are not relying on it. On related note. What's Xeon Phi in this context? I mean are we talking about Knights Corner Unfortunately, yes. BTW, you're welcome to play with it if you like: http://openwall.info/wiki/HPC/Village (that features own compatible-with-nothing SIMD instruction set) Yes, but at source code level many intrinsics match AVX-512. So we use it as a way to prepare for AVX-512. In many cases, it's just a recompile away. There are some notable exceptions to this, though - in fact, you happened to list some below. or Knights Landing (that features AVX512)? If latter, it might be interesting to extend multi-block SHA support(*), which should allow to achieve pretty cool results (with vector rotate and ternary logic instructions, not to mention 16 lanes:-). [As for interesting. It's possible but not really interesting in Knights Corner case, because effort is too specific, just a single obscure and hardly available CPU, while AVX512 is planned even for other processors so that code will be reusable.] This will take some #ifdef's to provide vector rotates as a macro when building for MIC and to use the ternary logic intrinsics only when building for true AVX-512 - nasty, but I think reasonable. For now, we're simply using the common subset between MIC and AVX-512: https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/src/pseudo_intrinsics.h https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/src/sse-intrinsics.c (*) BTW, did you try existing one? No, totally missed it! Found it now, good work! $ find -name 'sha*-mb*' ./crypto/sha/asm/sha256-mb-x86_64.pl ./crypto/sha/asm/sha1-mb-x86_64.pl How is an application using OpenSSL supposed to access this functionality? Is there documentation? So far, I only found uses in OpenSSL's own e_aes_cbc_hmac_sha*.c and no export of these symbols. You could
[openssl-dev] [openssl.org #3843] OpenSSL 1.0.1* and below: incorrect use of _lrotl()
Hi, Lei Zhang (re)discovered that OpenSSL 1.0.1* and below gets miscompiled, resulting in incorrect computation of at least SHA-1 hashes (and probably SHA-0, MD4, MD5) when it's compiled with icc for 64-bit Linux (x86_64 or mic), but not for Windows. The problem is already fixed in 1.0.2 and in LibreSSL. The problem is that OpenSSL uses the _lrotl() intrinsic to rotate 32-bit integers, whereas it is defined to operate on unsigned long, which obviously is 64-bit on many platforms. Lei's report: http://www.openwall.com/lists/john-dev/2015/03/26/1 A previous report (from 2011): https://software.intel.com/en-us/articles/openssl-generates-incorrect-shamd5-value-if-built-with-icc-compiler I suggest that this be fixed for all currently supported branches of OpenSSL. For now, Lei switched to using LibreSSL in our John the Ripper -jumbo builds for Xeon Phi, but we'd like to (re-)include instructions for building with OpenSSL as well. Thanks, Alexander ___ openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Re: [openssl.org #29] -Wl,-Bsymbolic in 0.9.6d broke shared builds
On Fri, Nov 15, 2002 at 09:05:13AM +0100, Richard Levitte - VMS Whacker via RT wrote: rt # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools rt # all: clean-shared Makefile.ssl sub_all rt make Makefile.ssl rt make sub_all DIRS=crypto ssl rsaref rt LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools You do know, don't you, that you really have no need for rsaref any more? Well, I left it in because the original Makefile would build it too. Is that just to support Configure rsaref? Anyhow, I will now consider this ticket resolved. OK. Thank you for getting back to me on this. I think I'll simplify our package a bit now, dropping the -Bsymbolic patch and building rsaref. -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: [openssl.org #29] -Wl,-Bsymbolic in 0.9.6d broke shared builds
On Thu, Nov 14, 2002 at 03:00:37PM +0100, Richard Levitte via RT wrote: [[EMAIL PROTECTED] - Sun May 12 22:48:56 2002]: JFYI, when updating our package from 0.9.6c to 0.9.6d I've noticed that the new shared libcrypto library doesn't work anymore. The openssl(1) binary wouldn't recognize any of the block ciphers. I tracked this down to the addition of -Wl,-Bsymbolic. Removing that option solved the problem for us. Hmm, if you, instead of removing that option, build and check the resulting library with 'nm', what exactly do you get? It's a mystery to me why things should go wrong for you. I've now tried removing the patch from our 0.9.6g package and what I get is: 1. Both versions appear to produce a working library now, however: 2. The sizes and symbol offsets in them differ: With -Wl,-Bsymbolic (original): -rwxr-xr-x root root 827429 Nov 15 09:28 /usr/lib/libcrypto.so.0.9.6 Without -Wl,-Bsymbolic (patched): -rwxr-xr-x root root 858309 Nov 15 09:40 /usr/lib/libcrypto.so.0.9.6 3. The sets of symbols (as reported by nm -a) are exactly the same. About the -Wl,-Bsymbolic problem, have you tried downloading a recent 0.9.6 snapshot and checked that the problem is still present? I haven't tried removing the patch from our package in further updates (to 0.9.6e and 0.9.6g), but have tried so now (see above). It does appear that the problem is gone, but I'd feel more comfortable knowing what made it go. One change to our package which could be relevant is this: * Wed Sep 25 2002 Solar Designer [EMAIL PROTECTED] - Don't do an explicit make build-shared, it's not needed and could only cause harm (link libssl against libcrypto statically), but luckily didn't; pointed out by Dmitry V. Levin of ALT Linux. Basically, with 0.9.6d we used to do: # Check these against the DIRS= line and all target in top-level Makefile # when updating to a new version of OpenSSL; with 0.9.6d the Makefile says: # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools # all: clean-shared Makefile.ssl sub_all make Makefile.ssl make sub_all DIRS=crypto ssl rsaref make build-shared LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools Now this has changed to: # Check these against the DIRS= line and all target in top-level Makefile # when updating to a new version of OpenSSL; with 0.9.6g the Makefile says: # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools # all: clean-shared Makefile.ssl sub_all make Makefile.ssl make sub_all DIRS=crypto ssl rsaref LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
[openssl.org #29] -Wl,-Bsymbolic in 0.9.6d broke shared builds
JFYI, when updating our package from 0.9.6c to 0.9.6d I've noticed that the new shared libcrypto library doesn't work anymore. The openssl(1) binary wouldn't recognize any of the block ciphers. I tracked this down to the addition of -Wl,-Bsymbolic. Removing that option solved the problem for us. GNU ld version 2.10.91 (with BFD 2.10.1.0.4) Supported emulations: elf_i386 i386linux Also 0.9.6d would tend to recompile some previously built files during the make install stage, resulting in static linking of openssl(1) (which is undesired in our case). I didn't investigate this too deeply but rather applied the following workaround for now: touch now find -type f -print0 | xargs -0 touch -r now -- right after unpacking the original tarball (before any patches are applied or anything is built). That helped. -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
