Re: configuring callbacks (or not) and SNI vs not... no shared cipher from server end
On Wed, May 08, 2019 at 05:23:38PM -0500, Benjamin Kaduk via openssl-users wrote: > > > In Postfix, it is configured with the same settings as the initial > > > SSL_CTX, *but* no server certificates. During the SNI callback I > > > interpose the certificate-less context, and then set the certificate > > > chain on the connection handle (SSL *) instead. > > > > okay, I'll use Postfix as my reference :-) > > For "how to use and switch SSL_CTXs" I'm sure it's admirable, but my > understanding is that it's still using the legacy server_name callback > (as opposed to the new client_hello_cb), and the new callback has a lot > of advantages for architectural cleanliness and avoiding some surprising > behavior with respect to the ordering of certain processing in the > server. So for a greenfield application I'd still suggest using the > client_hello_cb (not that I'm entirely unbiased...). The reason for the choice in Postfix is that we still support OpenSSL 1.0.2, which does not have the new interface. So for now I'm using the older interface which works with both 1.0.2 and 1.1.1. Since in Postfix we not also doing anything exciting like ALPN, or other exciting extensions, I don't expect any trouble from the original callback, but perhaps I've not looked closely enough at the potential drawbacks. If there's good reason to expect trouble, I'd like to hear what specifically might go wrong. -- Viktor.
Re: configuring callbacks (or not) and SNI vs not... no shared cipher from server end
On Wed, May 08, 2019 at 04:40:07PM -0400, Michael Richardson wrote: > > Viktor Dukhovni wrote: > >> Diversionary issue: > >> > https://www.openssl.org/docs/manmaster/man3/SSL_set_tlsext_host_name.html > >> and: > >> > https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_client_hello_cb.html > >> > >> are pretty vague. I think that SSL_set_tlsext_host_name() is probably > >> intended to be used on the client to set the SNI, but I'm not sure. > > > Yes, e.g. in the Postfix TLS client: > > > > https://github.com/vdukhovni/postfix/blob/master/postfix/src/tls/tls_client.c#L1035-L1045 > > So, okay. > Either this URL can go into the man page, or some short code extract could go > in. Probably better to have a code snippet (filing a github issue or sending a pull request would probably be good). > >> The legacy cb function returns int, but it's values are not > >> documented. > > > On the server side I'm using SSL_CTX_set_tlsext_servername_callback(): > > > > https://github.com/vdukhovni/postfix/blob/2399e9e179ee025d03155fa3637cccab0a23ddce/postfix/src/tls/tls_misc.c#L1040-L1043 > > > https://github.com/vdukhovni/postfix/blob/master/postfix/src/tls/tls_misc.c#L668 > > >> I guess the point is that CB can set the server certificate to > >> something appropriate, or I think, it could just decide to ignore the > >> SNI value completely and force the certificate regardless. > > > Yes. > > I can see that the CB provides comprehensive functionality, but I worry about > applications trying to parse ClientHello extensions themselves and getting it > wrong. It turns out that the server_name TLS extension is something of an unfortunate exception in terms of the unneeded complexity in its encoding. When I wrote the client_hello_cb functionality (at the time, know as the early_cb), I thought about whether I wanted to add a dedicated API just for the SNI value, due to the level of complexity involved. I ended up not doing so in the initial submission, both because I figured it could safely be added later as an incremental change, and because I was worried (IIRC) about being tempted to expose some of the PACKET_* APIs in the process, which is not really the right architectural choice for OpenSSL. There is, however, an existing implementation for extracting the SNI value in the test code at https://github.com/openssl/openssl/blob/master/test/handshake_helper.c#L150-L187 that has been successfully extracted and used in a couple places I know of. > >> What is the SNI functionality otherwise on the server? > > > You can interpose a secondary "virtual-host-specific" SSL_CTX for for > > the rest of the handshake. This carries the server certificate, but > > also the trust store settings for validating client certificates, the > > settings to request (or not) client certificates, the verification > > callbacks, ... It is a rather heavyweight object, best cached and > > re-used for multiple connections. > > So, it's okay to change the SSL_CTX for an SSL* after creation. > That is rather surprising to me, but I guess it's okay. > I suppose I feel that there ought to be reference counts, but this is C, not > Rust. There *are* reference counts. > > In Postfix, it is configured with the same settings as the initial > > SSL_CTX, *but* no server certificates. During the SNI callback I > > interpose the certificate-less context, and then set the certificate > > chain on the connection handle (SSL *) instead. > > okay, I'll use Postfix as my reference :-) For "how to use and switch SSL_CTXs" I'm sure it's admirable, but my understanding is that it's still using the legacy server_name callback (as opposed to the new client_hello_cb), and the new callback has a lot of advantages for architectural cleanliness and avoiding some surprising behavior with respect to the ordering of certain processing in the server. So for a greenfield application I'd still suggest using the client_hello_cb (not that I'm entirely unbiased...). -Ben > >> Is there any support for picking a certificate based upon the SNI > >> name? > > > The application does the "picking"... The application sets one or more > > certificate chains (one per supported public key algorithm) that best > > match the SNI name, and then OpenSSL chooses one of these based on the > > client's advertised supported signature algorithms, ... > > What I was observing (wrongly) was that maybe the server was doing something > itself if there was no callback, and it was failing. This was from looking > at the code around the error code that came out. > This (see other email) proved to wildly incorrect. > > -- > ] Never tell me the odds! | ipv6 mesh networks [ > ] Michael Richardson, Sandelman Software Works|IoT architect [ > ] m...@sandelman.ca
Re: configuring callbacks (or not) and SNI vs not... no shared cipher from server end
On Wed, May 08, 2019 at 04:40:07PM -0400, Michael Richardson wrote: > > You can interpose a secondary "virtual-host-specific" SSL_CTX for for > > the rest of the handshake. This carries the server certificate, but > > also the trust store settings for validating client certificates, the > > settings to request (or not) client certificates, the verification > > callbacks, ... It is a rather heavyweight object, best cached and > > re-used for multiple connections. > > So, it's okay to change the SSL_CTX for an SSL* after creation. > That is rather surprising to me, but I guess it's okay. > I suppose I feel that there ought to be reference counts, but this is C, not > Rust. It is not that sort of change "change", there's a call to insert a an additional CTX that interposes between the SSL handle and its parent context for most of the relevant data and function pointers. The SSL handle is still ultimately tied to the same parent context. > > In Postfix, it is configured with the same settings as the initial > > SSL_CTX, *but* no server certificates. During the SNI callback I > > interpose the certificate-less context, and then set the certificate > > chain on the connection handle (SSL *) instead. > > okay, I'll use Postfix as my reference :-) Fine by me. :-) -- Viktor.
Re: configuring callbacks (or not) and SNI vs not... no shared cipher from server end
Viktor Dukhovni wrote: >> Diversionary issue: >> https://www.openssl.org/docs/manmaster/man3/SSL_set_tlsext_host_name.html >> and: >> https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_client_hello_cb.html >> >> are pretty vague. I think that SSL_set_tlsext_host_name() is probably >> intended to be used on the client to set the SNI, but I'm not sure. > Yes, e.g. in the Postfix TLS client: > https://github.com/vdukhovni/postfix/blob/master/postfix/src/tls/tls_client.c#L1035-L1045 So, okay. Either this URL can go into the man page, or some short code extract could go in. >> The legacy cb function returns int, but it's values are not >> documented. > On the server side I'm using SSL_CTX_set_tlsext_servername_callback(): > https://github.com/vdukhovni/postfix/blob/2399e9e179ee025d03155fa3637cccab0a23ddce/postfix/src/tls/tls_misc.c#L1040-L1043 > https://github.com/vdukhovni/postfix/blob/master/postfix/src/tls/tls_misc.c#L668 >> I guess the point is that CB can set the server certificate to >> something appropriate, or I think, it could just decide to ignore the >> SNI value completely and force the certificate regardless. > Yes. I can see that the CB provides comprehensive functionality, but I worry about applications trying to parse ClientHello extensions themselves and getting it wrong. >> What is the SNI functionality otherwise on the server? > You can interpose a secondary "virtual-host-specific" SSL_CTX for for > the rest of the handshake. This carries the server certificate, but > also the trust store settings for validating client certificates, the > settings to request (or not) client certificates, the verification > callbacks, ... It is a rather heavyweight object, best cached and > re-used for multiple connections. So, it's okay to change the SSL_CTX for an SSL* after creation. That is rather surprising to me, but I guess it's okay. I suppose I feel that there ought to be reference counts, but this is C, not Rust. > In Postfix, it is configured with the same settings as the initial > SSL_CTX, *but* no server certificates. During the SNI callback I > interpose the certificate-less context, and then set the certificate > chain on the connection handle (SSL *) instead. okay, I'll use Postfix as my reference :-) >> Is there any support for picking a certificate based upon the SNI >> name? > The application does the "picking"... The application sets one or more > certificate chains (one per supported public key algorithm) that best > match the SNI name, and then OpenSSL chooses one of these based on the > client's advertised supported signature algorithms, ... What I was observing (wrongly) was that maybe the server was doing something itself if there was no callback, and it was failing. This was from looking at the code around the error code that came out. This (see other email) proved to wildly incorrect. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature
Re: configuring callbacks (or not) and SNI vs not... no shared cipher from server end
> On May 8, 2019, at 4:23 PM, Michael Richardson wrote: > > My questions about the documentation of the callbacks remain. > Having solved the problem, I'm pretty certain the the "no shared cipher" > error message is way too overloaded. It sounds like you failed to load a matching key pair into the server's SSL context (something that you would typically check as part of setting the certificate chain and private key). Once the server context has no signing keys, it can only negotiate anon-DHE and anon-ECDHE ciphers, but the client did not offer these, so you got "no shared cipher", which is fact correct. > Some piece of code is clearly doing something useful, which is to check if > the public/private key match. Unfortunately, that code is not announcing > the mismatch in a useful way. The check is done at configuration time. You're likely not doing the key setup "by the book": https://github.com/vdukhovni/postfix/blob/master/postfix/src/tls/tls_certkey.c#L600-L623 -- Viktor. -- Viktor.
Re: configuring callbacks (or not) and SNI vs not... no shared cipher from server end
My questions about the documentation of the callbacks remain. Having solved the problem, I'm pretty certain the the "no shared cipher" error message is way too overloaded. Some piece of code is clearly doing something useful, which is to check if the public/private key match. Unfortunately, that code is not announcing the mismatch in a useful way. My provisioning script, due to a typo, was generating new CSRs, but sending an ancient CSR with an old public key. Writing up the problem, I eventually noticed the public key dump from the private key file did not match the dump from the certificate. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature
Re: configuring callbacks (or not) and SNI vs not... no shared cipher from server end
On Wed, May 08, 2019 at 02:15:43PM -0400, Michael Richardson wrote: > Diversionary issue: > https://www.openssl.org/docs/manmaster/man3/SSL_set_tlsext_host_name.html > and: > https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_client_hello_cb.html > > are pretty vague. I think that SSL_set_tlsext_host_name() is probably > intended to be used on the client to set the SNI, but I'm not sure. Yes, e.g. in the Postfix TLS client: https://github.com/vdukhovni/postfix/blob/master/postfix/src/tls/tls_client.c#L1035-L1045 > The legacy cb function returns int, but it's values are not documented. On the server side I'm using SSL_CTX_set_tlsext_servername_callback(): https://github.com/vdukhovni/postfix/blob/2399e9e179ee025d03155fa3637cccab0a23ddce/postfix/src/tls/tls_misc.c#L1040-L1043 https://github.com/vdukhovni/postfix/blob/master/postfix/src/tls/tls_misc.c#L668 > I guess the point is that CB can set the server certificate to something > appropriate, or I think, it could just decide to ignore the SNI value > completely and force the certificate regardless. Yes. > What is the SNI functionality otherwise on the server? You can interpose a secondary "virtual-host-specific" SSL_CTX for for the rest of the handshake. This carries the server certificate, but also the trust store settings for validating client certificates, the settings to request (or not) client certificates, the verification callbacks, ... It is a rather heavyweight object, best cached and re-used for multiple connections. In Postfix, it is configured with the same settings as the initial SSL_CTX, *but* no server certificates. During the SNI callback I interpose the certificate-less context, and then set the certificate chain on the connection handle (SSL *) instead. > Is there any support for picking a certificate based upon the SNI name? The application does the "picking"... The application sets one or more certificate chains (one per supported public key algorithm) that best match the SNI name, and then OpenSSL chooses one of these based on the client's advertised supported signature algorithms, ... -- Viktor.
configuring callbacks (or not) and SNI vs not... no shared cipher from server end
Diversionary issue:
https://www.openssl.org/docs/manmaster/man3/SSL_set_tlsext_host_name.html
and:
https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_client_hello_cb.html
are pretty vague. I think that SSL_set_tlsext_host_name() is probably
intended to be used on the client to set the SNI, but I'm not sure.
The legacy cb function returns int, but it's values are not documented.
The newer cb function is better documented, but I'm a bit at a loss as to
what SSL_client_hello_get0_ext() extracts. Is the CB expected to parse the
extensions itself?
I guess the point is that CB can set the server certificate to something
appropriate, or I think, it could just decide to ignore the SNI value
completely and force the certificate regardless.
What is the SNI functionality otherwise on the server?
Is there any support for picking a certificate based upon the SNI name?
EXEC SUMMARY
I am asking because I seem to have run into a situation where it does not
seem to do the right thing, but I'm not sure that the error that I'm getting
is really about selecting the right certificate, or if there is something
else going on. Well, I'm pretty sure that the "no shared cipher" (on server)
is wrong. This comes out stderr upon receipt/processing of ClientHello.
Things I have tried (described below):
1) making sure that I'm running 1.1.1, which has ECDSA support, and not
getting 1.0 shared object by mistake (this has happened before)
2) making sure that the SubjectName contains the target SNI.
(with working certificate, it does not matter to server if I use wrong name)
3) observed private key was in different (SEC1 vs PKCS8) format, tried
switching that.
Many details at: http://www.sandelman.ca/tmp/certprob201905/
A longer story
--
In testing of a rails-based HTTPS server I have typically just configured a
keypair without a lot of thought to the DN used for the server. As I have
some (experimental) patches to openssl and ruby-openssl, I often struggle
with having the wrong shared object pulled in and then some things do not
work. In particular, I would get a message about no shared cipher on the
server emitted when something linked in openssl 1.0.x rather than 1.1.x,
and I had configured an ECDSA keypair. I'm not sure if it was precisely:
140639813764864:error:1417A0C1:SSL routines:tls_post_process_client_hello:no
shared cipher:ssl/statem/statem_srvr.c:2131:
(1) which I'm now getting, but it was close to that. I seem to recall that the
file name mentioned was one that was in 1.0.x, but not in 1.1.x, which was
the clue that I had made a linking error. I got this again yesterday while
testing, and wondered if I could excise 1.0.x completely from my laptop, and
finally found that it was only an old version of libpq5 that linked against
it, and an upgrade (via postgresql.org debian package) eliminated my ruby
process from ever linking 1.0.x.
Yet the error persisted. I test with the hostname target of
"fountain-test.example.com", which I put into my /etc/hosts as ::2
(an alias on lo), and bind against.
The private keys are test keys, and I could share them if that was useful.
NOTE that both working and failing certificates are generated by ruby code.
(2) I went back to a known working situation where a locally generated
certificate with that name as the CN was present:
(file: http://www.sandelman.ca/tmp/certprob201905/working-cert.txt and .pem)
Issuer: DC = ca, DC = sandelman, CN = fountain-test.example.com\0A
Unstrung Fountain Root CA
Validity
Not Before: May 7 22:56:23 2019 GMT
Not After : Jun 7 08:56:23 2019 GMT
Subject: DC = ca, DC = sandelman, CN = fountain-test.example.com
And this one works regardless of what name I use to access it. That is,
given:
::2 fountain-test.sandelman.ca fountain-test.example.com
n3CE618.router.securehomegateway.ca
all three of:
%curl -k https://fountain-test.sandelman.ca:8443/version.json
{"version":"0.7","revision":"devel"}%
%curl -k https://n3CE618.router.securehomegateway.ca:8443/version.json
{"version":"0.7","revision":"devel"}%
%curl -k https://fountain-test.example.com:8443/version.json
{"version":"0.7","revision":"devel"}%
work. Using what I think is a similar certificate:
/corp/projects/shg/shg_mud_supervisor/spec/files/product/Smarkaklink-n3ce618/jrc_prime256v1.crt
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 840664151 (0x321b8457)
Signature Algorithm: ecdsa-with-SHA256
Issuer: C = Canada, ST = Ontario, OU = Sandelman, CN =
highway-test.example.com CA
Validity
Not Before: May 8 17:18:37 2019 GMT
Not After : Dec 31 00:00:00 2999 GMT
Subject: CN = n3CE618.router.securehomegateway.ca
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
