Re: Use SNI with healthchecks
On Fri, Apr 27, 2018 at 06:39:07AM +0200, Willy Tarreau wrote: > I think that a few operators like strcmp() and concat() should be > implemented to cover the short-term needs. I forgot that I finally implemented concat() after talking about it for about a year :-) It is a good starting point to see how to implement strcmp() if you're interested. Willy
Re: Use SNI with healthchecks
Hi Tim,
On Fri, Apr 27, 2018 at 12:16:15AM +0200, Tim Düsterhus wrote:
> The solution I got from "Holger Just" was:
>
> > http-request set-header X-CHECKSNI %[req.hdr(host)]==%[ssl_fc_sni] if
> > { ssl_fc_has_sni }
> > http-request deny if { ssl_fc_has_sni } ! {
> > hdr(X-CHECKSNI) -m reg -i ^(?.+)==\1$ }
> > http-request del-header X-CHECKSNI
>
> which works well, but is not intuitive and requires PCRE.
>
> I have contemplated to add a patch with a `ssl_sni_http_host_mismatch`
> fetch, but I wasn't quite sure where to place it (it requires both http
> and ssl and thus is misplaced in both) and what parameters to pass
> (meaning SMP_USE_HRQHV et al). And thus I gave up.
I'd proceed differently I think. Just like we do have arithmetic operators
support variables as arguments, we could have string operators like strcmp()
which would support a variable as well. Then you could simply put your SNI
into a variable, and compare the host field with this variable.
In practice I've long wanted to have a stack-based sample expression
evaluation but while many of us think it will eventually appear, I think
that a few operators like strcmp() and concat() should be implemented to
cover the short-term needs.
> Not having this validation opens up possible security issues (validation
> of client certificates is based on SNI, routing is possibly based on the
> Host header).
For this specific case I agree.
> Personally I require SNI for everything and select the
> backends based on SNI and not the Host header.
This is dangerous because it's not what the backend servers will do,
so you need to keep this in mind to ensure you never switch to the
wrong place. Also as Lukas explained, when some certs overlap, you'll
only have the SNI matching the first Host. Apparently in your case it's
not an issue since you exactly compare them. Normally the reliable way
of doing it would be to ensure that SNI is valid (eg when you have to
check a client cert) but still rely on Host for routing.
> If you could give me some pointers on where to put such a fetch and what
> parameters to set I'd appreciate it. Or tell me that such a fetch is
> stupid, because it mixes information from different layers.
It's not stupid, it's just that I'm pretty sure that immediately after
being committed, someone will want to adjust its behaviour (ignore case,
stop on colon, etc). For this strcmp() will allow you to normalize each
party and then to compare the results.
Regards,
Willy
Re: Use SNI with healthchecks
Hi Lukas, On Fri, Apr 27, 2018 at 01:56:42AM +0200, Lukas Tribus wrote: > Hello Willy, > > > On 25 April 2018 at 12:16, Willy Tarreau wrote: > >> I'm not even sure that differentiate "Host" header from SNI values is > >> possible on softwares like Nginx or Apache. > > > > It should not, that would be a violation of HTTP over TLS. > > I think I disagree. > > > This is very possible and in fact happens everyday. When the > certificate has SAN's for www.example.org and mail.example.org, a > browser uses a single TLS connection to access both domains - SNI will > be whatever HTTP request warranted the TLS session in the first place > (as SNI is only send in the TLS handshake's client_hello). You're absolutely right. > This only breaks when people try to content switch HTTP transactions > that should be based on the Host header and look at the SNI header > instead (which I'd argue is a layering violation). SNI should imho > only be used for certificate selection at the TLS server > endpoint/layer and best not be carried to upper layers. Totally agreed, and I remember that both of us have been warning certain people against the risks of using SNI for L7 switching. > This is not limited to H2, I believe browsers do it in HTTP/1.1 as > well, but here's the H2 RFC documenting this behavior [1]. H2 also > introduces a new error code for servers unable to handle this, which > is 421 [2]. Yes, at the end of RFC6066 section 3 [3] there is a SHOULD > NOT about this. But it doesn't specifically address the HTTP use-case > and it's not a MUST NOT. It doesn't make sense for a browser to throw > away an already validated TLS sessions for which the hostname matches > a SAN from the validated certificate. I remember following a conversation on this exact subject on the HTTP WG, though I didn't catch all the details. But it's true that in general browsers will try hard to reuse existing connections as long as possible. > Let's not make the FTP mess again and pull lower layer transport (or > transport encryption) variables up into the application layer. Let the > TLS server endpoint pick the certificate based on SNI and then stop > looking at it would be my general advice. Please note that we were discussing about the other way around, how to automatically set SNI based on the Host header field, which is what browsers and proxies normally do. However, this discussion made me think about a case where we do not want to do this : if a browser connects to a proxy using TLS (to protect user authentication), the advertised SNI should not be the one from the Host header field but the one matching the proxy host name, which doesn't appear in the request and cannot be deduced. This is a good example showing that we can't have SNI always deduced from the request. > When TCP proxying TLS traffic and content switching based on SNI, one > needs to make sure that the certificates do not overlap because of > this connection reuse (just to provide another perspective), see [4] > and [5]. Agreed. In my opinion, SNI-based switching is only usable when you know that the certificates do not overlap. But in practice that's often the case initially, until someone renews the certs with multiple SANs for some purpose and starts to break everything. Cheers, Willy
Re: Use SNI with healthchecks
Hello Willy, On 25 April 2018 at 12:16, Willy Tarreau wrote: >> I'm not even sure that differentiate "Host" header from SNI values is >> possible on softwares like Nginx or Apache. > > It should not, that would be a violation of HTTP over TLS. I think I disagree. This is very possible and in fact happens everyday. When the certificate has SAN's for www.example.org and mail.example.org, a browser uses a single TLS connection to access both domains - SNI will be whatever HTTP request warranted the TLS session in the first place (as SNI is only send in the TLS handshake's client_hello). This only breaks when people try to content switch HTTP transactions that should be based on the Host header and look at the SNI header instead (which I'd argue is a layering violation). SNI should imho only be used for certificate selection at the TLS server endpoint/layer and best not be carried to upper layers. This is not limited to H2, I believe browsers do it in HTTP/1.1 as well, but here's the H2 RFC documenting this behavior [1]. H2 also introduces a new error code for servers unable to handle this, which is 421 [2]. Yes, at the end of RFC6066 section 3 [3] there is a SHOULD NOT about this. But it doesn't specifically address the HTTP use-case and it's not a MUST NOT. It doesn't make sense for a browser to throw away an already validated TLS sessions for which the hostname matches a SAN from the validated certificate. Let's not make the FTP mess again and pull lower layer transport (or transport encryption) variables up into the application layer. Let the TLS server endpoint pick the certificate based on SNI and then stop looking at it would be my general advice. When TCP proxying TLS traffic and content switching based on SNI, one needs to make sure that the certificates do not overlap because of this connection reuse (just to provide another perspective), see [4] and [5]. cheers, lukas [1] https://tools.ietf.org/html/rfc7540#section-9.1.1 [2] https://tools.ietf.org/html/rfc7540#section-9.1.2 [3] https://tools.ietf.org/html/rfc6066#section-3 [4] https://discourse.haproxy.org/t/sni-routing-to-servers-only-ever-sent-to-first-server/1265/4 [5] https://discourse.haproxy.org/t/h2-problem-with-multiple-backends/1650
Re: Use SNI with healthchecks
Willy,
Am 25.04.2018 um 12:16 schrieb Willy Tarreau:
> On Wed, Apr 25, 2018 at 09:48:13AM +, GALLISSOT VINCENT wrote:
>> I don't see a case were one would define a different check-sni or sni values
>> from the "Host" header.
>
> It definitely must match in HTTP. *snip*
>
>> I'm not even sure that differentiate "Host" header from SNI values is
>> possible on softwares like Nginx or Apache.
>
> It should not, that would be a violation of HTTP over TLS.
>
Unfortunately it’s not exactly easy to detect this condition using
haproxy. I asked on May 7th, 2017 on the list (thread: Compare against
variable string in ACL, Message-ID:
,
https://www.mail-archive.com/haproxy@formilux.org/msg26014.html).
The solution I got from "Holger Just" was:
> http-request set-header X-CHECKSNI %[req.hdr(host)]==%[ssl_fc_sni] if
> { ssl_fc_has_sni }
> http-request deny if { ssl_fc_has_sni } ! {
> hdr(X-CHECKSNI) -m reg -i ^(?.+)==\1$ }
> http-request del-header X-CHECKSNI
which works well, but is not intuitive and requires PCRE.
I have contemplated to add a patch with a `ssl_sni_http_host_mismatch`
fetch, but I wasn't quite sure where to place it (it requires both http
and ssl and thus is misplaced in both) and what parameters to pass
(meaning SMP_USE_HRQHV et al). And thus I gave up.
Not having this validation opens up possible security issues (validation
of client certificates is based on SNI, routing is possibly based on the
Host header). Personally I require SNI for everything and select the
backends based on SNI and not the Host header.
If you could give me some pointers on where to put such a fetch and what
parameters to set I'd appreciate it. Or tell me that such a fetch is
stupid, because it mixes information from different layers.
Best regards
Tim Düsterhus
RE: Use SNI with healthchecks
> It definitely must match in HTTP. However there's nothing making it mandatory > to send HTTP checks, let alone a Host header field (eg: if sending a simple > HTTP/1.0 request). However I'm noting the comment, because once we're able > to more easily configure the HTTP checks, we could imagine that we'd always > use SNI when "sni" is configured on the server line and an http check is in > use. That would be awesome ! > > I'm not even sure that differentiate "Host" header from SNI values is > > possible on softwares like Nginx or Apache. > It should not, that would be a violation of HTTP over TLS. > > We should always use SNI & check-sni as the same value as the "Host" header > > value, > Yes, when we have one :-) > > which in many cases should be the same value as the FQDN from the server > > line. > This is where it's not always the case. Very commonly you'll have the > individual server name (server1, server2, etc) and you need the application > Host header. Yes indeed, that's the point. We use HAProxy mainly to proxy traffic from devices that don't like url/domain changes (like ISP boxes) to CDNs providers so we rewrite a lot of things at HAProxy side (including Host). I totally understand that most HAProxy users need it to forward traffic as it is and therefore to keep SNI and other values from front clients requests. > > I also think SNI should be defaulted when using ssl over server backend. > We could possibly consider making it a default in a future version, indeed, > but only once we have the ability to automatically retrieve correct values. Would definitely be awesome! > > Check-sni should default to the same value as sni but re-writable. > Except that SNI comes from forwarded traffic and there's no forwarded traffic > with the checks to extract the information from. > > CDN providers like CloudFront or Akamai are using more and more SNI > > by blowing up prices on non-sni configurations (CloudFront bills > > 600$/month/vhost to avoid SNI) > I can easily understand why they're doing this. IP addresses are becoming > rare and it's a shame that some sites continue to waste them. With the > rapid growth of SSL over the last few years, I can easily imagine that > many sites have started to request their own IP addresses for no really > valid reason (I believe that MSIE6 was the last browser not to support SNI > but I could be wrong). Absolutely, what for use of default sni also in haproxy would be royal ;) Vincent > Willy
Re: Use SNI with healthchecks
On Wed, Apr 25, 2018 at 09:48:13AM +, GALLISSOT VINCENT wrote: > I don't see a case were one would define a different check-sni or sni values > from the "Host" header. It definitely must match in HTTP. However there's nothing making it mandatory to send HTTP checks, let alone a Host header field (eg: if sending a simple HTTP/1.0 request). However I'm noting the comment, because once we're able to more easily configure the HTTP checks, we could imagine that we'd always use SNI when "sni" is configured on the server line and an http check is in use. > I'm not even sure that differentiate "Host" header from SNI values is > possible on softwares like Nginx or Apache. It should not, that would be a violation of HTTP over TLS. > We should always use SNI & check-sni as the same value as the "Host" header > value, Yes, when we have one :-) > which in many cases should be the same value as the FQDN from the server line. This is where it's not always the case. Very commonly you'll have the individual server name (server1, server2, etc) and you need the application Host header. > I also think SNI should be defaulted when using ssl over server backend. We could possibly consider making it a default in a future version, indeed, but only once we have the ability to automatically retrieve correct values. > Check-sni should default to the same value as sni but re-writable. Except that SNI comes from forwarded traffic and there's no forwarded traffic with the checks to extract the information from. > CDN providers like CloudFront or Akamai are using more and more SNI > by blowing up prices on non-sni configurations (CloudFront bills > 600$/month/vhost to avoid SNI) I can easily understand why they're doing this. IP addresses are becoming rare and it's a shame that some sites continue to waste them. With the rapid growth of SSL over the last few years, I can easily imagine that many sites have started to request their own IP addresses for no really valid reason (I believe that MSIE6 was the last browser not to support SNI but I could be wrong). Willy
RE: Use SNI with healthchecks
I'll test the "setenv" approach soon. I don't see a case were one would define a different check-sni or sni values from the "Host" header. I'm not even sure that differentiate "Host" header from SNI values is possible on softwares like Nginx or Apache. We should always use SNI & check-sni as the same value as the "Host" header value, which in many cases should be the same value as the FQDN from the server line. I also think SNI should be defaulted when using ssl over server backend. Check-sni should default to the same value as sni but re-writable. CDN providers like CloudFront or Akamai are using more and more SNI by blowing up prices on non-sni configurations (CloudFront bills 600$/month/vhost to avoid SNI) Vincent De : Willy Tarreau Envoyé : mercredi 25 avril 2018 09:16 À : Jonathan Matthews Cc : GALLISSOT VINCENT; Lukas Tribus; haproxy@formilux.org Objet : Re: Use SNI with healthchecks On Tue, Apr 24, 2018 at 06:50:13PM +, Jonathan Matthews wrote: > [Top post; fight me] Grrr > You could either read an environment variable inherited from outside the > process, or use "setenv" or "presetenv" as appropriate to DRY your config > out. > > The fine manual describes how you would refer to this envvar in section > 2.3, regardless of which of those options you use to set it. That's indeed a possibility. In Vincent's case I'm seeing that he uses the same name as the FQDN one. That makes me think we could possibly have a special check-sni value to use the FQDN from the server line. That would have the benefit of making it easier to place in a default-server statement. I just don't know how often it happens that the SNI used for checks has to match the FQDN declared in the configuration. Willy
Re: Use SNI with healthchecks
On Tue, Apr 24, 2018 at 06:50:13PM +, Jonathan Matthews wrote: > [Top post; fight me] Grrr > You could either read an environment variable inherited from outside the > process, or use "setenv" or "presetenv" as appropriate to DRY your config > out. > > The fine manual describes how you would refer to this envvar in section > 2.3, regardless of which of those options you use to set it. That's indeed a possibility. In Vincent's case I'm seeing that he uses the same name as the FQDN one. That makes me think we could possibly have a special check-sni value to use the FQDN from the server line. That would have the benefit of making it easier to place in a default-server statement. I just don't know how often it happens that the SNI used for checks has to match the FQDN declared in the configuration. Willy
Re: Use SNI with healthchecks
[Top post; fight me] You could either read an environment variable inherited from outside the process, or use "setenv" or "presetenv" as appropriate to DRY your config out. The fine manual describes how you would refer to this envvar in section 2.3, regardless of which of those options you use to set it. J On Tue, 24 Apr 2018 at 16:45, GALLISSOT VINCENT wrote: > I migrated to 1.8 and sni + check-sni *are working fine* with the > following code: > > > > 88 > > backend cloudfront > http-request set-header Host 123456789abcde.cloudfront.net > option httpchk HEAD /check HTTP/1.1\r\nHost:\ > 123456789abcde.cloudfront.net > server applaunch 123456789abcde.cloudfront.net:443 check resolvers > mydns no-sslv3 ssl verify required ca-file ca-certificates.crt sni > req.hdr(host) > check-sni 123456789abcde.cloudfront.net > > 88 > > > Obviously I cannot use %[req.hdr(host)] for "option httpchk" nor for > "check-sni" directives. > > > Do you know how can I define only one time my Host header in the code > above ? > > > Thanks, > > Vincent > > > ------ > *De :* GALLISSOT VINCENT > *Envoyé :* lundi 23 avril 2018 17:33 > *À :* Lukas Tribus > *Cc :* haproxy@formilux.org > *Objet :* RE: Use SNI with healthchecks > > > Thank you very much for your answers, > > I'll migrate to 1.8 asap to fix this. > > > Vincent > > > > ---------- > *De :* lu...@ltri.eu de la part de Lukas Tribus < > lu...@ltri.eu> > *Envoyé :* lundi 23 avril 2018 17:18 > *À :* GALLISSOT VINCENT > *Cc :* haproxy@formilux.org > *Objet :* Re: Use SNI with healthchecks > > Hello Vincent, > > > On 23 April 2018 at 16:38, GALLISSOT VINCENT > wrote: > > Does anybody know how can I use healthchecks over HTTPS with SNI support > ? > > You need haproxy 1.8 for this, it contains the check-sni directive > which allows to set SNI to a specific string for the health check: > > http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-check-sni > > > > > Regards, > > Lukas > -- Jonathan Matthews London, UK http://www.jpluscplusm.com/contact.html
RE: Use SNI with healthchecks
I migrated to 1.8 and sni + check-sni are working fine with the following code: 88 backend cloudfront http-request set-header Host 123456789abcde.cloudfront.net option httpchk HEAD /check HTTP/1.1\r\nHost:\ 123456789abcde.cloudfront.net server applaunch 123456789abcde.cloudfront.net:443 check resolvers mydns no-sslv3 ssl verify required ca-file ca-certificates.crt sni req.hdr(host) check-sni 123456789abcde.cloudfront.net 88 Obviously I cannot use %[req.hdr(host)] for "option httpchk" nor for "check-sni" directives. Do you know how can I define only one time my Host header in the code above ? Thanks, Vincent De : GALLISSOT VINCENT Envoyé : lundi 23 avril 2018 17:33 À : Lukas Tribus Cc : haproxy@formilux.org Objet : RE: Use SNI with healthchecks Thank you very much for your answers, I'll migrate to 1.8 asap to fix this. Vincent De : lu...@ltri.eu de la part de Lukas Tribus Envoyé : lundi 23 avril 2018 17:18 À : GALLISSOT VINCENT Cc : haproxy@formilux.org Objet : Re: Use SNI with healthchecks Hello Vincent, On 23 April 2018 at 16:38, GALLISSOT VINCENT wrote: > Does anybody know how can I use healthchecks over HTTPS with SNI support ? You need haproxy 1.8 for this, it contains the check-sni directive which allows to set SNI to a specific string for the health check: http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-check-sni Regards, Lukas
RE: Use SNI with healthchecks
Thank you very much for your answers, I'll migrate to 1.8 asap to fix this. Vincent De : lu...@ltri.eu de la part de Lukas Tribus Envoyé : lundi 23 avril 2018 17:18 À : GALLISSOT VINCENT Cc : haproxy@formilux.org Objet : Re: Use SNI with healthchecks Hello Vincent, On 23 April 2018 at 16:38, GALLISSOT VINCENT wrote: > Does anybody know how can I use healthchecks over HTTPS with SNI support ? You need haproxy 1.8 for this, it contains the check-sni directive which allows to set SNI to a specific string for the health check: http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-check-sni Regards, Lukas
Re: Use SNI with healthchecks
Hello Vincent, On 23 April 2018 at 16:38, GALLISSOT VINCENT wrote: > Does anybody know how can I use healthchecks over HTTPS with SNI support ? You need haproxy 1.8 for this, it contains the check-sni directive which allows to set SNI to a specific string for the health check: http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-check-sni Regards, Lukas
Re: Use SNI with healthchecks
Hi Vincent,
On Mon, Apr 23, 2018 at 02:38:32PM +, GALLISSOT VINCENT wrote:
> Hi all,
>
>
> I want to use SNI with httpchk on HAProxy 1.7.10 to connect to CloudFront
> distributions as backend servers.
>
> I saw in this mailing-list archives that SNI is not used by default even when
> using the ssl directive.
>
> We don't pay for SNI on that distribution, that means CloudFront doesn't
> provide a certificate on its default vhost.
>
> Because of that, all healthchecks fail with "handshake failure".
>
>
> I temporarily by-passed the issue by adding "port 80" to allow healthchecks
> over HTTP:
>
>
> option httpchk HEAD /check HTTP/1.1\r\nHost:\
> mydistribution.cloudfront.net
> server mydistribution mydistribution.cloudfront.net:443 check resolvers
> mydns port 80 cookie no-sslv3 ssl verify required ca-file ca-certificates.crt
>
>
> Does anybody know how can I use healthchecks over HTTPS with SNI support ?
>
Prior to 1.8 if you want SNI in the health checks you have to use something
along these lines:
backend moo
mode http
option httpchk GET / HTTP/1.0
server s1 my.example.host:443 check addr 127.0.0.1 port 1234 ssl sni
str("my.example.host")
listen foo
bind 127.0.0.1:1234
server s1 my.example.host:443 sni str("my.example.host") ssl
That's because sni keyword only applies to proxied traffic, and not checks, so
you check through a listener that will add the sni.
With 1.8 and later, you just use check-sni on server lines.
cheers,
Jérôme
Use SNI with healthchecks
Hi all, I want to use SNI with httpchk on HAProxy 1.7.10 to connect to CloudFront distributions as backend servers. I saw in this mailing-list archives that SNI is not used by default even when using the ssl directive. We don't pay for SNI on that distribution, that means CloudFront doesn't provide a certificate on its default vhost. Because of that, all healthchecks fail with "handshake failure". I temporarily by-passed the issue by adding "port 80" to allow healthchecks over HTTP: option httpchk HEAD /check HTTP/1.1\r\nHost:\ mydistribution.cloudfront.net server mydistribution mydistribution.cloudfront.net:443 check resolvers mydns port 80 cookie no-sslv3 ssl verify required ca-file ca-certificates.crt Does anybody know how can I use healthchecks over HTTPS with SNI support ? Many thanks, Vincent
