[Bug-wget] OpenSSL
Hi Everyone, I'm testing a new configuration. Wget 1.20.3 and OpenSSL 1.1.1d. One minor warning: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC=\"/home/jwalton/bootstrap/etc/wgetrc\" -DLOCALEDIR=\"/home/jwalton/bootstrap/share/locale\" -I. -I../lib -I../lib -I/home/jwalton/bootstrap/include -I/home/jwalton/bootstrap/include -DHAVE_LIBSSL -DNDEBUG -MT version.o -MD -MP -MF .deps/version.Tpo -c -o version.o version.c mv -f .deps/version.Tpo .deps/version.Po openssl.c: In function 'ssl_init': openssl.c:178:7: warning: 'OPENSSL_config' is deprecated [-Wdeprecated-declarations] OPENSSL_config (NULL); ^~ In file included from /home/jwalton/bootstrap/include/openssl/e_os2.h:13:0, from /home/jwalton/bootstrap/include/openssl/ssl.h:15, from openssl.c:40: /home/jwalton/bootstrap/include/openssl/conf.h:91:25: note: declared here DEPRECATEDIN_1_1_0(void OPENSSL_config(const char *config_name)) ^ /home/jwalton/bootstrap/include/openssl/opensslconf.h:127:37: note: in definition of macro 'DECLARE_DEPRECATED' # define DECLARE_DEPRECATED(f)f __attribute__ ((deprecated)); ^ /home/jwalton/bootstrap/include/openssl/conf.h:91:1: note: in expansion of macro 'DEPRECATEDIN_1_1_0' DEPRECATEDIN_1_1_0(void OPENSSL_config(const char *config_name)) ^~
[Bug-wget] OpenSSL, random seeding and init_prng
Hi Everyone, Looking at https://git.savannah.gnu.org/cgit/wget.git/tree/src/openssl.c and init_prng, there could be four small issues, but I don't think they pose significant risk. I think its mostly splitting hairs. First, checking RAND_status could setup a race condition for most software. Wget is not general purpose, so I'm not sure if it applies here. That is, will another Wget thread drain entropy just after this thread initializes the prng: if (RAND_status ()) return; Second, there are several of these sprinkled throughout init_prng. if (RAND_status ()) return; If random_file is plentiful but fixed (maybe burned into ROM), then entropy does not really change in the way one would expect on a desktop. The early-out effectively means the same blob might be used over and over again: if (random_file && *random_file) RAND_load_file (random_file, 16384); if (RAND_status ()) return; Third, if a users wants to add EGD entropy via --egd-file=file, then the early out from item (2) means the user's choice may not be honored in some instances. That is, the same blob is used over and over again and the early out short circuits the desired EGD code: if (RAND_status ()) return; #ifdef HAVE_RAND_EGD /* Get random data from EGD if opt.egd_file was used. */ if (opt.egd_file && *opt.egd_file) RAND_egd (opt.egd_file); #endif Fourth, before an OpenSSL routine is called that uses a random numbers, the prng could be re-seeded/stirred to help avoid some entropy based attacks, like VM rollbacks. Also see the following references. Even NIST is recommending a stir in their approved generators for prediction resistance and back tracking resistance: * When Virtual is Harder than Real: Security Challenges in Virtual Machine Based Computing Environments, https://www.usenix.org/legacy/event/hotos05/final_papers/full_papers/garfinkel/garfinkel.pdf * When Good Randomness Goes Bad: Virtual Machine Reset Vulnerabilities and Hedging Deployed Cryptography, http://pages.cs.wisc.edu/~rist/papers/sslhedge.pdf * NIST SP 800-90A, http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-90Ar1.pdf Jeff
Re: [Bug-wget] OpenSSL 1.1.0
On Wednesday 29 June 2016 13:22:07 Tim Ruehsen wrote: > On Wednesday 29 June 2016 00:10:34 Ángel González wrote: > > On 28/06/16 22:16, Tim Rühsen wrote: > > > Patching src/openssl.c for 1.1.0 (see below) let it compile. > > > But the HTTPS tests fail due to > > > > > > ERROR: cannot verify localhost's certificate, issued by > > > 'O=GNU,OU=Wget,CN=GNU> > > > > > > Wget': > > >unsupported certificate purpose > > > > > > Any idea ? > > > > server-cert.pem has the following extensions: > > Key Usage > > Usages:Revocation list signature > > Critical:Yes > > > > Extended Key Usage > > Allowed Purposes:Server Authentication > > Critical:No > > > > > > Looks like the second extension isn't supported by OpenSSL 1.1.0, and > > Server Authentication not being in Key Usage, it is rejected. > > > > Recreate this certificate with no Key Usage at all would probably fix > > it. I'm not sure about the required steps, though. > > Just pushed a commit with a shell script to automatically generate the files > in testenv/certs. Built with GnuTLS, wget passes the tests. > > With OpenSSL 1.1.0 (+ my patch + freshly generated certs), wget spins at all > HTTPS tests, eating up 100% CPU. > > With OpenSSL 1.1.0 (+ my patch + old certs), wget spins only in Test- > pinnedpubkey-der-no-check-https.py. The other HTTPS tests fail. > > With a little debug output, I verified that SSL_peek() does not return (and > spins). Here is wget / valgrind output: > > Setting --no-config (noconfig) to 1 > Setting --check-certificate (checkcertificate) to 0 > Setting --pinnedpubkey (pinnedpubkey) to > /usr/oms/src/wget1.x/testenv/certs/server-pubkey.der > DEBUG output created by Wget 1.18.7-4335 on linux-gnu. > > Reading HSTS entries from /usr/oms/.wget-hsts > URI encoding = ‘UTF-8’ > Converted file name 'File1' (UTF-8) -> 'File1' (UTF-8) > --2016-06-29 13:15:01-- https://127.0.0.1:34755/File1 > Connecting to 127.0.0.1:34755... connected. > Created socket 3. > Releasing 0x093d49d0 (new refcount 0). > Deleting unused 0x093d49d0. > Initiating SSL handshake. > Handshake successful; connected socket 3 to SSL handle 0x093d4b90 > certificate: > subject: O=GNU,OU=Wget,CN=127.0.0.1 > issuer: O=GNU,OU=Wget,CN=GNU Wget > WARNING: cannot verify 127.0.0.1's certificate, issued by > ‘O=GNU,OU=Wget,CN=GNU Wget’: > Unable to locally verify the issuer's authority. > > ---request begin--- > GET /File1 HTTP/1.1 > User-Agent: Wget/1.18.7-4335 (linux-gnu) > Accept: */* > Accept-Encoding: identity > Host: 127.0.0.1:34755 > Connection: Keep-Alive > > ---request end--- > 127.0.0.1 - - [29/Jun/2016 13:15:02] "GET /File1 HTTP/1.1" 200 - > HTTP request sent, awaiting response... > [Here is spins - killing memcheck process after a while:] > ==560== > ==560== Process terminating with default action of signal 15 (SIGTERM) > ==560==at 0x54D802A: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1) > ==560==by 0x54DDFB5: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1) > ==560==by 0x54E7B56: SSL_peek (in > /usr/lib/x86_64-linux-gnu/libssl.so.1.1) ==560==by 0x4360BC: > openssl_peek (openssl.c:420) > ==560==by 0x429BEC: fd_read_hunk (retr.c:513) > ==560==by 0x41D546: read_http_response_head (http.c:575) > ==560==by 0x41D546: gethttp (http.c:3162) > ==560==by 0x42074F: http_loop (http.c:3975) > ==560==by 0x42AB75: retrieve_url (retr.c:817) > ==560==by 0x406C72: main (main.c:1947) > ==560== > > This kind of error could be anything... but OpenSSL should not behave like > that at all... any ideas ? Just want to say that going back to OpenSSL 1.0.2h-1 (Debian unstable), all tests work fine, even with new, auto-generated cert and keys. Tim signature.asc Description: This is a digitally signed message part.
Re: [Bug-wget] OpenSSL 1.1.0
On Wednesday 29 June 2016 00:10:34 Ángel González wrote: > On 28/06/16 22:16, Tim Rühsen wrote: > > Patching src/openssl.c for 1.1.0 (see below) let it compile. > > But the HTTPS tests fail due to > > > > ERROR: cannot verify localhost's certificate, issued by > > 'O=GNU,OU=Wget,CN=GNU> > > Wget': > >unsupported certificate purpose > > > > Any idea ? > > server-cert.pem has the following extensions: > Key Usage > Usages:Revocation list signature > Critical:Yes > > Extended Key Usage > Allowed Purposes:Server Authentication > Critical:No > > > Looks like the second extension isn't supported by OpenSSL 1.1.0, and > Server Authentication not being in Key Usage, it is rejected. > > Recreate this certificate with no Key Usage at all would probably fix > it. I'm not sure about the required steps, though. Just pushed a commit with a shell script to automatically generate the files in testenv/certs. Built with GnuTLS, wget passes the tests. With OpenSSL 1.1.0 (+ my patch + freshly generated certs), wget spins at all HTTPS tests, eating up 100% CPU. With OpenSSL 1.1.0 (+ my patch + old certs), wget spins only in Test- pinnedpubkey-der-no-check-https.py. The other HTTPS tests fail. With a little debug output, I verified that SSL_peek() does not return (and spins). Here is wget / valgrind output: Setting --no-config (noconfig) to 1 Setting --check-certificate (checkcertificate) to 0 Setting --pinnedpubkey (pinnedpubkey) to /usr/oms/src/wget1.x/testenv/certs/server-pubkey.der DEBUG output created by Wget 1.18.7-4335 on linux-gnu. Reading HSTS entries from /usr/oms/.wget-hsts URI encoding = ‘UTF-8’ Converted file name 'File1' (UTF-8) -> 'File1' (UTF-8) --2016-06-29 13:15:01-- https://127.0.0.1:34755/File1 Connecting to 127.0.0.1:34755... connected. Created socket 3. Releasing 0x093d49d0 (new refcount 0). Deleting unused 0x093d49d0. Initiating SSL handshake. Handshake successful; connected socket 3 to SSL handle 0x093d4b90 certificate: subject: O=GNU,OU=Wget,CN=127.0.0.1 issuer: O=GNU,OU=Wget,CN=GNU Wget WARNING: cannot verify 127.0.0.1's certificate, issued by ‘O=GNU,OU=Wget,CN=GNU Wget’: Unable to locally verify the issuer's authority. ---request begin--- GET /File1 HTTP/1.1 User-Agent: Wget/1.18.7-4335 (linux-gnu) Accept: */* Accept-Encoding: identity Host: 127.0.0.1:34755 Connection: Keep-Alive ---request end--- 127.0.0.1 - - [29/Jun/2016 13:15:02] "GET /File1 HTTP/1.1" 200 - HTTP request sent, awaiting response... [Here is spins - killing memcheck process after a while:] ==560== ==560== Process terminating with default action of signal 15 (SIGTERM) ==560==at 0x54D802A: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1) ==560==by 0x54DDFB5: ??? (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1) ==560==by 0x54E7B56: SSL_peek (in /usr/lib/x86_64-linux-gnu/libssl.so.1.1) ==560==by 0x4360BC: openssl_peek (openssl.c:420) ==560==by 0x429BEC: fd_read_hunk (retr.c:513) ==560==by 0x41D546: read_http_response_head (http.c:575) ==560==by 0x41D546: gethttp (http.c:3162) ==560==by 0x42074F: http_loop (http.c:3975) ==560==by 0x42AB75: retrieve_url (retr.c:817) ==560==by 0x406C72: main (main.c:1947) ==560== This kind of error could be anything... but OpenSSL should not behave like that at all... any ideas ? Regards, Tim signature.asc Description: This is a digitally signed message part.
Re: [Bug-wget] OpenSSL 1.1.0
On Wednesday 29 June 2016 00:10:34 Ángel González wrote: > On 28/06/16 22:16, Tim Rühsen wrote: > > Patching src/openssl.c for 1.1.0 (see below) let it compile. > > But the HTTPS tests fail due to > > > > ERROR: cannot verify localhost's certificate, issued by > > 'O=GNU,OU=Wget,CN=GNU> > > Wget': > >unsupported certificate purpose > > > > Any idea ? > > server-cert.pem has the following extensions: > Key Usage > Usages:Revocation list signature > Critical:Yes > > Extended Key Usage > Allowed Purposes:Server Authentication > Critical:No > > > Looks like the second extension isn't supported by OpenSSL 1.1.0, and > Server Authentication not being in Key Usage, it is rejected. > > Recreate this certificate with no Key Usage at all would probably fix > it. I'm not sure about the required steps, though. Thanks for the hint, I'll check it out. BTW, I documented the creation of the test certs in testenv/certs/README. Meanwhile I saw that certtool supports also has a non-interactive mode... so it would be possible to write a small shell script to automate the process of creating the test keys/certs/crl etc. Regards signature.asc Description: This is a digitally signed message part.
Re: [Bug-wget] OpenSSL 1.1.0
On 28/06/16 22:16, Tim Rühsen wrote: Patching src/openssl.c for 1.1.0 (see below) let it compile. But the HTTPS tests fail due to ERROR: cannot verify localhost's certificate, issued by 'O=GNU,OU=Wget,CN=GNU Wget': unsupported certificate purpose Any idea ? server-cert.pem has the following extensions: Key Usage Usages:Revocation list signature Critical:Yes Extended Key Usage Allowed Purposes:Server Authentication Critical:No Looks like the second extension isn't supported by OpenSSL 1.1.0, and Server Authentication not being in Key Usage, it is rejected. Recreate this certificate with no Key Usage at all would probably fix it. I'm not sure about the required steps, though. Regards
[Bug-wget] OpenSSL 1.1.0
Patching src/openssl.c for 1.1.0 (see below) let it compile.
But the HTTPS tests fail due to
ERROR: cannot verify localhost's certificate, issued by 'O=GNU,OU=Wget,CN=GNU
Wget':
unsupported certificate purpose
Any idea ?
Regards, Tim
The (quick) patch to compile would be
diff --git a/src/openssl.c b/src/openssl.c
index c6549ea..389f3ba 100644
--- a/src/openssl.c
+++ b/src/openssl.c
@@ -204,18 +204,20 @@ ssl_init (void)
#endif
SSL_library_init ();
SSL_load_error_strings ();
+#if OPENSSL_VERSION_NUMBER < 0x1010L
SSLeay_add_all_algorithms ();
SSLeay_add_ssl_algorithms ();
+#endif
switch (opt.secure_protocol)
{
-#ifndef OPENSSL_NO_SSL2
+#if !defined OPENSSL_NO_SSL2 && OPENSSL_VERSION_NUMBER < 0x1010L
case secure_protocol_sslv2:
meth = SSLv2_client_method ();
break;
#endif
-#ifndef OPENSSL_NO_SSL3
+#ifndef OPENSSL_NO_SSL3_METHOD
case secure_protocol_sslv3:
meth = SSLv3_client_method ();
break;
@@ -583,7 +585,7 @@ ssl_connect_wget (int fd, const char *hostname, int
*continue_session)
DEBUGP (("SSL handshake timed out.\n"));
goto timeout;
}
- if (scwt_ctx.result <= 0 || SSL_state(conn) != SSL_ST_OK)
+ if (scwt_ctx.result <= 0 || SSL_get_state(conn) != TLS_ST_OK)
goto error;
ctx = xnew0 (struct openssl_transport_context);
signature.asc
Description: This is a digitally signed message part.
