Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )
> You don't have to explain the code to me. :-) ^^. Ok. I was also trying to explain what I understood so you can correct me if I'm wrong. Everything is Cristal clear now. Thanks a million. On Sun, 3 Oct 2021, 22:25 Viktor Dukhovni, wrote: > On Sun, Oct 03, 2021 at 09:33:29PM +0200, Alex Robuchon wrote: > > > > Not quite, a candidate chain is constructed from whatever certificates > the > > > peer (server in your case) provided, and then passed to the callback > with > > > "preverify_ok" set to false (for the top certificate), because the > chain is > > > not trusted. > > > > This confuses me a little bit because I thought the callback function set > > with *SSL_set_verify* would have the "trusted first" valid chain. > > But there is no trust store configured, so trust first is a no-op. > The constructed chain consists entirely of certificates from the > peer, and last certificate in the chain (the ultimate issuing CA) > is still not trusted, so preverify_ok = 0. > > > > Are you sure you are not speaking as if eventmachine was using the > > *SSL_CTX_set_cert_verify_callback* ? > > I rewrote the OpenSSL chain construction code for OpenSSL 1.1.0. Yes, I > am sure. > > > - *build_chain* that will apply the trusted first algorithm and replace > the > > certificate chain passed by the server with the valid one ( if found ). > > The code in eventmachine does not configure any trusted certificates for > the SSL_CTX handles used to create the SSL connections. So > trusted-first is a no-op. > > > - *internal_verify *which now has the new chain and will call > *verify_callback > > *( the callback function passed to SSL_set_verify ) for every certificate > > in this new chain in reverse order. > > You don't have to explain the code to me. :-) > > > > But given all the evidence before me, I'd want to delete that code and > > > never see it again. > > > > I hear you :). > > That code is beyond repair, it needs to be thrown out, and replaced with > ground up by the book TLS connection establishment. > > #1. No compiled in private keys > > #2. Configure the SSL_CTX with the desired list of trust anchors > > #3. Reuse the SSL_CTX for multiple connections that share the same > trust and other general settings > > #4. Prior to SSL_connect(), tell the SSL library what hostname you want > to verify via SSL_set1_host(3). Let OpenSSL do all the heavy > lifting of certificate and name checks. > > #5. DO NOT attempt to override certificate chain verifiction in the > verify callback. Use the verify callback only for logging or > similar diagnostic purposes. > > #6. Return "preverify_ok" unmodified, unless your application is doing > opportunistic unauthenticated TLS, or wants to complete the > handshake even on verification failure, and then gracefully tear > down the application-layer session with the peer (after checking the > verification status at connection completion). > > If preverify_ok is false (0), OpenSSL chain verification failed, > your application is unlikely to do better. Return 0, and the > TLS handshake will be aborted, you can log the error reason, > chain depth, ... in the verify callback. > > -- > Viktor. >
Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )
> > Not quite, a candidate chain is constructed from whatever certificates the > peer (server in your case) provided, and then passed to the callback with > "preverify_ok" set to false (for the top certificate), because the chain is > not trusted. > This confuses me a little bit because I thought the callback function set with *SSL_set_verify* would have the "trusted first" valid chain. Are you sure you are not speaking as if eventmachine was using the *SSL_CTX_set_cert_verify_callback* ? >From what I see from the source code ( master ) The *tls_post_process_server_certificate* function calls ssl/ssl_cert.c#ssl_verify_cert_chain <https://github.com/openssl/openssl/blob/19e277dd19f2897f6a7b7eb236abe46655e575bf/ssl/ssl_cert.c#L426> if (s->ctx->app_verify_callback != NULL) i = s->ctx->app_verify_callback(ctx, s->ctx->app_verify_arg); else i = X509_verify_cert(ctx); return i Since eventmachine do not set the *app_verify_callback* so the default *X509_verify_cert* is used from crypto/x509/x509_vfy.c (L261) <https://github.com/openssl/openssl/blob/19e277dd19f2897f6a7b7eb236abe46655e575bf/crypto/x509/x509_vfy.c#L261> The function *X509_verify_cert* calls *verify_chain(ctx)* line 295 which in turns calls: - *build_chain* that will apply the trusted first algorithm and replace the certificate chain passed by the server with the valid one ( if found ). At the point the ctx has the new chain - *internal_verify *which now has the new chain and will call *verify_callback *( the callback function passed to SSL_set_verify ) for every certificate in this new chain in reverse order. During the build_chain process I notice the ctx->get_issuer ( which actually points to X509_STORE_CTX_get1_issuer which I suppose return 0 because eventmachine do not properly set the store ) But the evenmachine callback ignores "preverify_ok" and goes through the > motions of doing some sort of verification of each certificate. > yes indeed But given all the evidence before me, I'd want to delete that code and > never see it again. > I hear you :). On Sun, Oct 3, 2021 at 6:48 PM Viktor Dukhovni wrote: > > On 3 Oct 2021, at 12:33 pm, Alex Robuchon > wrote: > > > > So I suppose openssl skip the part that is supposed to build the chain > when no store is configured. > > Not quite, a candidate chain is constructed from whatever certificates > the peer (server in your case) provided, and then passed to the callback > with "preverify_ok" set to false (for the top certificate), because the > chain is not trusted. > > But the evenmachine callback ignores "preverify_ok" and goes through the > motions of doing some sort of verification of each certificate. > > Ultimately, it will attempt to "verify" the leaf certificate, and if it > can somehow do a fair job of that (by building its own chain, checking > all the signatures, doing the name checks (for a name that does not > appear to be passed to the verification function), then in theory > the checks at depths above 0 are just silly opportunities to fail and > the EE cert check would be enough. > > But given all the evidence before me, I'd want to delete that code and > never see it again. > > -- > Viktor. > >
Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )
I just had a big laugh Reading this. I never had to dig into openssl before so I unfortunately had to try to understand your API with eventmachine as an example. Silly me. > On the other hand I suppose if we do not call this it would pick the > "default" trust store from the system which seems to be the case here > because it can find /usr/lib/ssl/certs/2e5ac55d.0 . > Actually the part responsible for this lookup is not part of the openssl library but from the ruby callback function registered through SSL_set_verify which as you said tries to verify each certificate in isolation ( with a configured store this time ) So I suppose openssl skip the part that is supposed to build the chain when no store is configured. Do whatever it takes to never rely on this code again. Even abandon Ruby > if that's what it takes... Hopefully we do not have a lot of code relying on eventmachine so we'll see if we can improve the library or move away from it. Thanks so much for your time, explanation and responsiveness. On Sun, 3 Oct 2021, 17:37 Viktor Dukhovni, wrote: > On Sun, Oct 03, 2021 at 01:54:44PM +0200, Alex Robuchon wrote: > > > Thanks for the detailed answer. > > > > From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1 > > > > When I use the eventmachine lib that uses the wrong cert chain I can see > > with strace : > > Run as far away from eventmachine as you can. The very first thing you > see in the code is a compiled in default "private key" (for all the > world to share). > > > https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L30-L123 > > Though applications can override these, and supply actual private keys, > it does not get better from there... The comments in the code show the > author punting on understanding the OpenSSL API and just guessing what > to do. > > Indeed the code creates SSL_CTX objects without specifying either the > default or explicit trust stores. > > The real disaster is in: > > > https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L675-L698 > > with a completely broken SSL verification callback, that completely > ignores all errors from the chain construction and signature > verification code, and just attempts to "verify" each certificate in > *isolation*. > > > https://github.com/eventmachine/eventmachine/blob/5cac87805f26b5cdc29eca713871c3374131d786/ext/ssl.cpp#L693-L697 > > This means: > > * No verification of chain signatures > * No verification of path constraints > * No verification of name constraints > * No hostname checks. > * ... > > This code was written by someone too clueless to know what they're > doing and too lazy to bother to learn. DO NOT USE IT. > > Do whatever it takes to never rely on this code again. Even abandon > Ruby if that's what it takes... > > -- > Viktor. >
Re: SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )
Thanks for the detailed answer.
>From strace I can see that I'm using /lib/x86_64-linux-gnu/libssl.so.1.1
When I use the eventmachine lib that uses the wrong cert chain I can see
with strace :
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/2e5ac55d.0", {st_mode=S_IFREG|0644, st_size=1200,
...}) = 0
openat(AT_FDCWD, "/usr/lib/ssl/certs/2e5ac55d.0", O_RDONLY) = 8
stat("/usr/lib/ssl/certs/2e5ac55d.1", 0x7ffe1a8e5d80) = -1 ENOENT (No such
file or directory)
When I use another lib that uses the correct cert chain I can see with
strace :
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/8d33f237.0", 0x7fff1b4b0f90) = -1 ENOENT (No such
file or directory)
stat("/usr/lib/ssl/certs/4042bcee.0", {st_mode=S_IFREG|0644, st_size=1939,
...}) = 0
openat(AT_FDCWD, "/usr/lib/ssl/certs/4042bcee.0", O_RDONLY) = 8
In the second case I can see it tries to load the R3 certificate
( 8d33f237.0 ). I wonder why in the first case it does not try to find each
certificate in the chain from the trust store at all. I wonder if it can be
related to the fact I do not see any call to SSL_CTX_set_cert_store in the
lib. On the other hand I suppose if we do not call this it would pick the
"default" trust store from the system which seems to be the case here
because it can find /usr/lib/ssl/certs/2e5ac55d.0 .
This looks like to be an issue in the eventmachine lib itself. I need to
brush up my C skills and have a deeper look at this :).
Thanks
On Sat, Oct 2, 2021 at 8:11 PM Viktor Dukhovni
wrote:
> On Sat, Oct 02, 2021 at 06:21:00PM +0100, Angus Robertson - Magenta
> Systems Ltd wrote:
>
> > > Yes. To make things even more complex, a few sites also have an
> > > older version of R3 that is directly signed by the DST root:
> > >
> > > - leaf <- R3 <- DST Root CA X3 (self-signed)
> > >
> > > but that's far from common at this point.
> >
> > That old R3 [CA] was issued last winter and got installed in Windows
> > Server 2018 intermediate stores then, and was still being sent out on
> > 29th and 30th, despite expiring on 29th.
>
> Not just Windows, at least one Unix system running Postfix is still
> vending a chain with the R3 signed by DST that expired on 2021-09-29:
>
> issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
> subject=C = US, O = Let's Encrypt, CN = R3
> notBefore=Oct 7 19:21:40 2020 GMT
> notAfter=Sep 29 19:21:40 2021 GMT
> SHA256
> Fingerprint=73:0C:1B:DC:D8:5F:57:CE:5D:C0:BB:A7:33:E5:F1:BA:5A:92:5B:2A:77:1D:64:0A:26:F7:A4:54:22:4D:AD:3B
> -BEGIN CERTIFICATE-
> MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
> MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
> DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
> MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
> AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
> jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
> Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
> U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
> gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
> /xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
> oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
> BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
> ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
> p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
> AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
> Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
> LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
> r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
> AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
> ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
> S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
> qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
> O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
> UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
> -END CERTIFICATE-
>
> The EE (depth 0 server) certificate is not expired, and yet somehow the
> server is building a chain with a fresh leaf cert, and a rather stale
> issuer CA.
>
> It verifies via the DANE implementation in OpenSSL, because its "2 1 1"
> record with a fresh RRSIG specifies the R3 CA as trusted, and its
> expiration date is not in scope since it was signed by an entity outside
> the effective trust chain.
>
> Validation would fail for the same chain with WebPKI, unless there's a
> new improved R3 in the trust store (not just the roots).
>
> My DANE survey scan engine checks trust-anchor cert expiration date,
> even when an intermediate CA,
SSL_CTX_set_verify uses the "wrong" certificate chain (cross signed certificate )
Hello Openssl community, I've encountered an issue with em-http-request ( https://github.com/igrigorik/em-http-request) based on top of eventmachine ( https://github.com/eventmachine/eventmachine) since let's encrypt Root certificate expired the 30th of September. The project has a callback function registered with SSL_CTX_set_verify and failed to verify DST Root CA X3 since it expired. >From what I understood about the let's encrypt certificate chain, R3 is cross signed and two chained could be built: - leaf <- R3 <- ISRG Root X1 <- DST Root CA X3 (self-signed) - leaf <- R3 <- ISRG Root X1 (self-signed) The servers by default return the first chain but from what I understand depending on the openssl version/flags it should use the second path if ISRG Root X1 is in the store My config : openssl version : OpenSSL 1.1.1f 31 Mar 2020 os : 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ISRG Root X1 self signed is in my cert store. So from what I understand, trusted first is default in this version of openssl and the second path should be taken. For the record s_client can valide the second path on my machine : $ echo | openssl s_client -connect retouche-pro.ch:443 -name retouche-pro.ch -servername retouche-pro.ch CONNECTED(0003) depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = retouche-pro.ch verify return:1 --- Certificate chain 0 s:CN = retouche-pro.ch i:C = US, O = Let's Encrypt, CN = R3 1 s:C = US, O = Let's Encrypt, CN = R3 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1 i:O = Digital Signature Trust Co., CN = DST Root CA X3 --- So I suppose the problem relies on the use of openssl in the ruby libs. However what I'm a bit surprised with is the fact that SSL_CTX_set_verify is called with the "wrong" certificate chain. The documentation says it's going to be called for each certificate in the chain but do not specify if it's the chain specified by the server or the one built by openssl with trusted first algorithm. I also tried adding a verify callback on the ruby net http library ( https://github.com/ruby/net-http) which is passed to SSL_CTX_set_verify under the hood and I also noticed that it's using the "wrong" chain. Is it normal to have the callback defined in SSL_CTX_set_verify to be called with the wrong chain ? Or do you think something is not configured correctly on these gems ? If it's normal behavior how can I have additional certificate verification on the trusted first chain. Thanks for reading, I hope it was not too boring or confusing. Alex
