Re: [openssl-dev] [openssl.org #3843] OpenSSL 1.0.1* and below: incorrect use of _lrotl()
Hi Andy, Thank you for your reply! I am CC'ing Lei on mine. On Wed, May 20, 2015 at 12:55:10PM +0200, Andy Polyakov via RT wrote: For reference. icc was not cared for for quite some time. Initially it was possible for me, by then university employee, to use it, but then they changes terms and it became impossible for me to maintain it. But I've just noticed they provide some starter version of something, I'll see... Yes, this might be usable for you: https://software.intel.com/en-us/qualify-for-free-software/opensourcecontributor Intel provides select Intel Software Development Products at no cost to qualified open source contributors who are working on open source projects compliant with the Open Source Initiative (OSI). But linux-x86_64-icc is not present in and was never supported in pre-1.0.2. Oh, I didn't realize that. Like I mentioned, we're actually building with icc for MIC. When we build with icc for x86_64 host, we typically simply link against the distro's gcc-built OpenSSL, so didn't run into this issue ourselves until we started building for MIC and thus had to make our own OpenSSL build with icc. (Indeed, I've been building OpenSSL from source on many other occasions, and as part of a distro too, but that's not with icc and unrelated to JtR project.) So you ought to provide custom line. This remark doesn't mean that fix can't be backported, but out of curiosity, what's your config line? Currently, Lei put this into JtR -jumbo README-MIC: Build LibreSSL (version 2.1.6): $ cd libressl-2.1.6 $ ./configure CC=icc -mmic --host=k1om-linux --prefix=$MIC $ make make install The previous instructions were: Build OpenSSL (version 1.0.0q): $ cd openssl-1.0.0q $ patch Configure $JOHN/src/unused/openssl.patch $ ./Configure linux-mic shared --prefix=$MIC $ make make install I'm not sure what was in $JOHN/src/unused/openssl.patch - I guess it had to add linux-mic support. Lei, please reply to all. Is assembly engaged? If so, how fast is it? Or is it so that you count on compiler to produce vector code that would process multiple inputs in parallel with SIMD? We're using OpenSSL (or LibreSSL) as an easy but slower option, replacing it with our own SIMD code right in JtR tree whenever we can and where this makes sense. So we're not trying to optimize OpenSSL's code. It remains scalar and unmodified, and our use of it is just to have things working where we do not have optimized code yet or where we prefer simpler rather than faster code (such as for some lightweight precomputation in some rare cases where this makes sense). This varies by crypto primitive, but overall we currently have SIMD intrinsics code for MMX, SSE2+/AVX, XOP, AVX2, MIC/AVX-512, and for bitslice DES also for AltiVec and NEON. One thing for which we still use OpenSSL's code in performance-critical manner is SSH key passphrase cracking (which involves RSA). There are probably many more examples like this, but this is a prominent one that comes to mind. There must be a lot of room for optimization here. As to compiler auto-vectorization - no, we are not relying on it. On related note. What's Xeon Phi in this context? I mean are we talking about Knights Corner Unfortunately, yes. BTW, you're welcome to play with it if you like: http://openwall.info/wiki/HPC/Village (that features own compatible-with-nothing SIMD instruction set) Yes, but at source code level many intrinsics match AVX-512. So we use it as a way to prepare for AVX-512. In many cases, it's just a recompile away. There are some notable exceptions to this, though - in fact, you happened to list some below. or Knights Landing (that features AVX512)? If latter, it might be interesting to extend multi-block SHA support(*), which should allow to achieve pretty cool results (with vector rotate and ternary logic instructions, not to mention 16 lanes:-). [As for interesting. It's possible but not really interesting in Knights Corner case, because effort is too specific, just a single obscure and hardly available CPU, while AVX512 is planned even for other processors so that code will be reusable.] This will take some #ifdef's to provide vector rotates as a macro when building for MIC and to use the ternary logic intrinsics only when building for true AVX-512 - nasty, but I think reasonable. For now, we're simply using the common subset between MIC and AVX-512: https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/src/pseudo_intrinsics.h https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/src/sse-intrinsics.c (*) BTW, did you try existing one? No, totally missed it! Found it now, good work! $ find -name 'sha*-mb*' ./crypto/sha/asm/sha256-mb-x86_64.pl ./crypto/sha/asm/sha1-mb-x86_64.pl How is an application using OpenSSL supposed to access this functionality? Is there documentation? So far, I only found uses in OpenSSL's own e_aes_cbc_hmac_sha*.c and no export of these symbols. You could
[openssl-dev] [openssl.org #3843] OpenSSL 1.0.1* and below: incorrect use of _lrotl()
Hi, Lei Zhang (re)discovered that OpenSSL 1.0.1* and below gets miscompiled, resulting in incorrect computation of at least SHA-1 hashes (and probably SHA-0, MD4, MD5) when it's compiled with icc for 64-bit Linux (x86_64 or mic), but not for Windows. The problem is already fixed in 1.0.2 and in LibreSSL. The problem is that OpenSSL uses the _lrotl() intrinsic to rotate 32-bit integers, whereas it is defined to operate on unsigned long, which obviously is 64-bit on many platforms. Lei's report: http://www.openwall.com/lists/john-dev/2015/03/26/1 A previous report (from 2011): https://software.intel.com/en-us/articles/openssl-generates-incorrect-shamd5-value-if-built-with-icc-compiler I suggest that this be fixed for all currently supported branches of OpenSSL. For now, Lei switched to using LibreSSL in our John the Ripper -jumbo builds for Xeon Phi, but we'd like to (re-)include instructions for building with OpenSSL as well. Thanks, Alexander ___ openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
openssl 0.9.6h - i
Hi, I've noticed this while reviewing diffs between 0.9.6h and 0.9.6i: --- openssl-0.9.6h/crypto/x509/x509_vfy.c Thu Nov 28 11:06:30 2002 +++ openssl-0.9.6i/crypto/x509/x509_vfy.c Tue Dec 10 11:28:16 2002 @@ -897,7 +897,7 @@ ctx-chain=NULL; } CRYPTO_free_ex_data(x509_store_ctx_method,ctx,(ctx-ex_data)); - OPENSSL_cleanse(ctx-ex_data,sizeof(CRYPTO_EX_DATA)); + memset(ctx-ex_data,0,sizeof(CRYPTO_EX_DATA)); } Is this change intentional, it appears to undo the change introduced between 0.9.6g and 0.9.6h? -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: openssl 0.9.6h - i
Sorry for following up on my own mail, but I just wanted to say that reverting the change now seems right to me (it's setting ex_data-sk to NULL, not zeroing out sensitive data). On Thu, Feb 20, 2003 at 10:06:55AM +0300, Solar Designer wrote: Hi, I've noticed this while reviewing diffs between 0.9.6h and 0.9.6i: --- openssl-0.9.6h/crypto/x509/x509_vfy.c Thu Nov 28 11:06:30 2002 +++ openssl-0.9.6i/crypto/x509/x509_vfy.c Tue Dec 10 11:28:16 2002 @@ -897,7 +897,7 @@ ctx-chain=NULL; } CRYPTO_free_ex_data(x509_store_ctx_method,ctx,(ctx-ex_data)); - OPENSSL_cleanse(ctx-ex_data,sizeof(CRYPTO_EX_DATA)); + memset(ctx-ex_data,0,sizeof(CRYPTO_EX_DATA)); } Is this change intentional, it appears to undo the change introduced between 0.9.6g and 0.9.6h? -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: [openssl.org #29] -Wl,-Bsymbolic in 0.9.6d broke shared builds
On Thu, Nov 14, 2002 at 03:00:37PM +0100, Richard Levitte via RT wrote: [[EMAIL PROTECTED] - Sun May 12 22:48:56 2002]: JFYI, when updating our package from 0.9.6c to 0.9.6d I've noticed that the new shared libcrypto library doesn't work anymore. The openssl(1) binary wouldn't recognize any of the block ciphers. I tracked this down to the addition of -Wl,-Bsymbolic. Removing that option solved the problem for us. Hmm, if you, instead of removing that option, build and check the resulting library with 'nm', what exactly do you get? It's a mystery to me why things should go wrong for you. I've now tried removing the patch from our 0.9.6g package and what I get is: 1. Both versions appear to produce a working library now, however: 2. The sizes and symbol offsets in them differ: With -Wl,-Bsymbolic (original): -rwxr-xr-x root root 827429 Nov 15 09:28 /usr/lib/libcrypto.so.0.9.6 Without -Wl,-Bsymbolic (patched): -rwxr-xr-x root root 858309 Nov 15 09:40 /usr/lib/libcrypto.so.0.9.6 3. The sets of symbols (as reported by nm -a) are exactly the same. About the -Wl,-Bsymbolic problem, have you tried downloading a recent 0.9.6 snapshot and checked that the problem is still present? I haven't tried removing the patch from our package in further updates (to 0.9.6e and 0.9.6g), but have tried so now (see above). It does appear that the problem is gone, but I'd feel more comfortable knowing what made it go. One change to our package which could be relevant is this: * Wed Sep 25 2002 Solar Designer [EMAIL PROTECTED] - Don't do an explicit make build-shared, it's not needed and could only cause harm (link libssl against libcrypto statically), but luckily didn't; pointed out by Dmitry V. Levin of ALT Linux. Basically, with 0.9.6d we used to do: # Check these against the DIRS= line and all target in top-level Makefile # when updating to a new version of OpenSSL; with 0.9.6d the Makefile says: # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools # all: clean-shared Makefile.ssl sub_all make Makefile.ssl make sub_all DIRS=crypto ssl rsaref make build-shared LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools Now this has changed to: # Check these against the DIRS= line and all target in top-level Makefile # when updating to a new version of OpenSSL; with 0.9.6g the Makefile says: # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools # all: clean-shared Makefile.ssl sub_all make Makefile.ssl make sub_all DIRS=crypto ssl rsaref LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: [openssl.org #29] -Wl,-Bsymbolic in 0.9.6d broke shared builds
On Fri, Nov 15, 2002 at 09:05:13AM +0100, Richard Levitte - VMS Whacker via RT wrote: rt # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools rt # all: clean-shared Makefile.ssl sub_all rt make Makefile.ssl rt make sub_all DIRS=crypto ssl rsaref rt LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools You do know, don't you, that you really have no need for rsaref any more? Well, I left it in because the original Makefile would build it too. Is that just to support Configure rsaref? Anyhow, I will now consider this ticket resolved. OK. Thank you for getting back to me on this. I think I'll simplify our package a bit now, dropping the -Bsymbolic patch and building rsaref. -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: [openssl.org #29] -Wl,-Bsymbolic in 0.9.6d broke shared builds
On Fri, Nov 15, 2002 at 09:05:13AM +0100, Richard Levitte - VMS Whacker via RT wrote: rt # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools rt # all: clean-shared Makefile.ssl sub_all rt make Makefile.ssl rt make sub_all DIRS=crypto ssl rsaref rt LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools You do know, don't you, that you really have no need for rsaref any more? Well, I left it in because the original Makefile would build it too. Is that just to support Configure rsaref? Anyhow, I will now consider this ticket resolved. OK. Thank you for getting back to me on this. I think I'll simplify our package a bit now, dropping the -Bsymbolic patch and building rsaref. -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: [openssl.org #29] -Wl,-Bsymbolic in 0.9.6d broke shared builds
On Thu, Nov 14, 2002 at 03:00:37PM +0100, Richard Levitte via RT wrote: [[EMAIL PROTECTED] - Sun May 12 22:48:56 2002]: JFYI, when updating our package from 0.9.6c to 0.9.6d I've noticed that the new shared libcrypto library doesn't work anymore. The openssl(1) binary wouldn't recognize any of the block ciphers. I tracked this down to the addition of -Wl,-Bsymbolic. Removing that option solved the problem for us. Hmm, if you, instead of removing that option, build and check the resulting library with 'nm', what exactly do you get? It's a mystery to me why things should go wrong for you. I've now tried removing the patch from our 0.9.6g package and what I get is: 1. Both versions appear to produce a working library now, however: 2. The sizes and symbol offsets in them differ: With -Wl,-Bsymbolic (original): -rwxr-xr-x root root 827429 Nov 15 09:28 /usr/lib/libcrypto.so.0.9.6 Without -Wl,-Bsymbolic (patched): -rwxr-xr-x root root 858309 Nov 15 09:40 /usr/lib/libcrypto.so.0.9.6 3. The sets of symbols (as reported by nm -a) are exactly the same. About the -Wl,-Bsymbolic problem, have you tried downloading a recent 0.9.6 snapshot and checked that the problem is still present? I haven't tried removing the patch from our package in further updates (to 0.9.6e and 0.9.6g), but have tried so now (see above). It does appear that the problem is gone, but I'd feel more comfortable knowing what made it go. One change to our package which could be relevant is this: * Wed Sep 25 2002 Solar Designer [EMAIL PROTECTED] - Don't do an explicit make build-shared, it's not needed and could only cause harm (link libssl against libcrypto statically), but luckily didn't; pointed out by Dmitry V. Levin of ALT Linux. Basically, with 0.9.6d we used to do: # Check these against the DIRS= line and all target in top-level Makefile # when updating to a new version of OpenSSL; with 0.9.6d the Makefile says: # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools # all: clean-shared Makefile.ssl sub_all make Makefile.ssl make sub_all DIRS=crypto ssl rsaref make build-shared LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools Now this has changed to: # Check these against the DIRS= line and all target in top-level Makefile # when updating to a new version of OpenSSL; with 0.9.6g the Makefile says: # DIRS= crypto ssl rsaref $(SHLIB_MARK) apps test tools # all: clean-shared Makefile.ssl sub_all make Makefile.ssl make sub_all DIRS=crypto ssl rsaref LD_LIBRARY_PATH=`pwd` make sub_all DIRS=apps test tools -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
-Wl,-Bsymbolic in 0.9.6d broke shared builds
JFYI, when updating our package from 0.9.6c to 0.9.6d I've noticed that the new shared libcrypto library doesn't work anymore. The openssl(1) binary wouldn't recognize any of the block ciphers. I tracked this down to the addition of -Wl,-Bsymbolic. Removing that option solved the problem for us. GNU ld version 2.10.91 (with BFD 2.10.1.0.4) Supported emulations: elf_i386 i386linux Also 0.9.6d would tend to recompile some previously built files during the make install stage, resulting in static linking of openssl(1) (which is undesired in our case). I didn't investigate this too deeply but rather applied the following workaround for now: touch now find -type f -print0 | xargs -0 touch -r now -- right after unpacking the original tarball (before any patches are applied or anything is built). That helped. -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
[openssl.org #29] -Wl,-Bsymbolic in 0.9.6d broke shared builds
JFYI, when updating our package from 0.9.6c to 0.9.6d I've noticed that the new shared libcrypto library doesn't work anymore. The openssl(1) binary wouldn't recognize any of the block ciphers. I tracked this down to the addition of -Wl,-Bsymbolic. Removing that option solved the problem for us. GNU ld version 2.10.91 (with BFD 2.10.1.0.4) Supported emulations: elf_i386 i386linux Also 0.9.6d would tend to recompile some previously built files during the make install stage, resulting in static linking of openssl(1) (which is undesired in our case). I didn't investigate this too deeply but rather applied the following workaround for now: touch now find -type f -print0 | xargs -0 touch -r now -- right after unpacking the original tarball (before any patches are applied or anything is built). That helped. -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: openssl dgst ignores read errors
On Mon, Apr 29, 2002 at 03:48:48PM +0100, Ben Laurie wrote: Solar Designer wrote: it could also be nice to report the filename and strerror(errno), or it is sometimes not immediately clear what the error messages apply to: jill!solar:~/build/openssl-SNAP-20020416$ apps/openssl dgst -md5 /bin/ls /dev/log /bin /usr/bin/id MD5(/bin/ls)= d93498d9f52c3dc0330ab930fe3ffc50 Read Error Read Error MD5(/usr/bin/id)= 4b37435d0793aba2b602fd2da0d7f8c5 jill!solar:~/build/openssl-SNAP-20020416$ echo $? 0 ...and the zero exit status in that case is even worse. I am thinking something like fread: /bin: Is a directory printed to stderr and a non-zero exit if at least one of the arguments resulted in an error. Well, here's an even bigger and better patch. Thanks for the continued feedback. OK, I've now tested this patch applied to 0.9.6d (by the way, the announcement message incorrectly mentions beta 1 in the Subject line). It works as desired, thanks! I wouldn't make the messages this verbose for uses from within the openssl binary, but that's up to you. I'll be updating the OpenSSL package in Owl to 0.9.6d + this patch now. Thank you once again. -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: openssl dgst ignores read errors
On Tue, Apr 23, 2002 at 12:09:14PM +0100, Ben Laurie wrote:
Solar Designer wrote:
This sounds like a bug to me. Noticed it last year and I've just
checked that it's still not resolved in the latest snapshot.
jill!solar:~/build/openssl-SNAP-20020416$ apps/openssl dgst -md5 /bin/ls
MD5(/bin/ls)= d93498d9f52c3dc0330ab930fe3ffc50
OK.
jill!solar:~/build/openssl-SNAP-20020416$ apps/openssl dgst -md5 /bin
MD5(/bin)= d41d8cd98f00b204e9800998ecf8427e
jill!solar:~/build/openssl-SNAP-20020416$ apps/openssl dgst -md5 /dev/log
MD5(/dev/log)= d41d8cd98f00b204e9800998ecf8427e
Wrong. I'd want it to fail with a message to stderr and a non-zero
exit code. Also on any read error, not just on non-regular files.
open(/bin, O_RDONLY) = 4
[some getpid()'s]
read(4, 0x8189fb0, 8192)= -1 EISDIR (Is a directory)
[lots of getpid()'s, why?!]
write(1, MD5(/bin)= d41d8cd98f00b204e9800..., 44) = 44
Try this for size (against 0.9.7, but probably will work on other
versions)
Applied to the same snapshot as above with patch -l (probably tabs
got converted to spaces somewhere in your mail).
It's better but still not perfect:
jill!solar:~/build/openssl-SNAP-20020416$ apps/openssl dgst -md5 /dev/log
Read Error
MD5(/dev/log)= jill!solar:~/build/openssl-SNAP-20020416$ echo $?
0
I'd rather not have it output the MD5(/dev/log)= until it has the
hash value and have it exit non-zero.
ret=fread(out,1,(int)outl,(FILE *)b-ptr);
+ if(ret == 0 ferror((FILE *)b-ptr))
+ ret=-1;
I am looking at this out of context (so I don't know if the fread()
will always be re-called when ret is smaller than the requested
outl) but I'd change the ret == 0 to ret outl (possibly with
a cast) or drop this check entirely (the ferror() alone is sufficient).
size_t fread(void *ptr, size_t size, size_t nitems, FILE *stream);
Upon successful completion, fread() returns the number of members
successfully read which is less than nitems only if a read error or
end-of-file is encountered. If size or nitems is 0, fread() returns 0
and the contents of the array and the state of the stream remain
unchanged. Otherwise, if a read error occurs, the error indicator for
the stream is set and errno is set to indicate the error.
(SUSv2)
--- apps/dgst.c 2002/04/06 18:59:57 1.23.2.2
+++ apps/dgst.c 2002/04/23 11:06:14
@@ -362,7 +362,13 @@
for (;;)
{
i=BIO_read(bp,(char *)buf,BUFSIZE);
- if (i = 0) break;
+ if(i 0)
+ {
+ BIO_printf(bio_err, Read Error\n);
+ ERR_print_errors(bio_err);
+ return;
+ }
+ if (i == 0) break;
}
I'm not sure how this would handle partial reads. I don't know the
semantics of BIO_read(), but it appears weird to retry with the same
arguments on any positive return value.
Thank you for working on this!
--
/sd
__
OpenSSL Project http://www.openssl.org
Development Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
openssl dgst ignores read errors
Hi, This sounds like a bug to me. Noticed it last year and I've just checked that it's still not resolved in the latest snapshot. jill!solar:~/build/openssl-SNAP-20020416$ apps/openssl dgst -md5 /bin/ls MD5(/bin/ls)= d93498d9f52c3dc0330ab930fe3ffc50 OK. jill!solar:~/build/openssl-SNAP-20020416$ apps/openssl dgst -md5 /bin MD5(/bin)= d41d8cd98f00b204e9800998ecf8427e jill!solar:~/build/openssl-SNAP-20020416$ apps/openssl dgst -md5 /dev/log MD5(/dev/log)= d41d8cd98f00b204e9800998ecf8427e Wrong. I'd want it to fail with a message to stderr and a non-zero exit code. Also on any read error, not just on non-regular files. open(/bin, O_RDONLY) = 4 [some getpid()'s] read(4, 0x8189fb0, 8192)= -1 EISDIR (Is a directory) [lots of getpid()'s, why?!] write(1, MD5(/bin)= d41d8cd98f00b204e9800..., 44) = 44 -- /sd __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
