Re: MINOR: fixes haiku linkage
On Sat, Jun 19, 2021 at 02:47:52PM +0100, David CARLIER wrote: > Hi here a little change proposal to fix haproxy at runtime in this platform. > Cheers. Thanks David, now merged. Willy
Re: Fix small bug in srv_parse_agent_check
Hi Dirkjan, On Fri, Jun 18, 2021 at 10:03:17PM +0200, Dirkjan Bussink wrote: > Hi all, > > I was building HAProxy using scan-build to see if there were any issues and > it called out an unused variable write. I think this was due to a bug that > the err_code was not used in srv_parse_agent_check. I agree with your analysis, I've now merged your patch. Thank you! Willy
Re: SSL Labs says my server isn't doing ssl session resumption
On Sun, Jun 20, 2021 at 05:20:41PM -0600, Shawn Heisey wrote: > On 6/20/2021 3:16 PM, Lukas Tribus wrote: > > It's a haproxy bug, affecting 2.4 releases, I've filed an issue in our > > tracker: > > > > https://github.com/haproxy/haproxy/issues/1297 > > Almost always when I report a problem I'm having with a mature piece of > software, I expect the issue to be PEBCAK, not an actual bug. Either way, I > really enjoy being a part of an open source community. Hehe we have no business trying to hide bugs, as they'll affect all of us. It's much better to humbly recognize them and fix them :-) > It took me a while to figure out how to get the proper code branch with git > for 2.4 development. 2.4 is not in development anymore, it's stable (and LTS -- long term supported). Please don't try to do any development on 2.4, you'll only waste your time, because if you want to propose some changes, you'll first be requested to rebase your changes on top of latest development and that could take you a while in certain areas. The right way to participate is to pull the development branch, and update it once in a while (e.g. every time you're going back to your developments, such as once a week). This way if something breaks, chances are that the person who did the breaking change still has fresh memories about it and will be able to help, and instantly fix the trouble or explain how to adapt your code. Do not worry too much for the risk of instabilities, the published code is tested and even the haproxy.org site runs on it. It's quite sufficient for development and even for either a personal site, or one that is properly monitored and operated by skilled admins. > I don't think it's on github. This is the first time > I've seen completely separate repositories for different versions. Not > complaining ... I bet there's a very good reason for it. It precisely is because we don't want to receive pull reqs nor issues on the wrong branch, and github doesn't offer sufficient flexibility to allow to disable PRs on certain branches nor repos. I can't blame them too much for this, it's their core business, they have no reason to offer free hosting and resources with no hopes to see forks and PRs in return that indirectly bring them new users. But overall the current model looks like a good balance offering a good efficiency to everyone. > I wish I knew C a lot better than I do ... That's what about half of the long time contributors started to say before become regular ones, welcome! You'll get back to it easily, do not worry. C is easy to learn (or it wouldn't have lasted 50+ years), it just has a number of traps, but those not using it all day long are extra careful and generally do not fall into them. It's like with parachutes, accidents only affect those routinely using them and who are less careful. > though I expect that for TLS > code, even very strong C skills would leave me clueless without spending > several weeks or months doing a deep dive. I agree. Cryptography code is cryptic, that's the part I understand the least. In addition, OpenSSL is even older than haproxy and has 2.5 decades of evolving API, naming schemes, coding styles and incompatible features behind, that inevitably leave some confusion at plenty of places. HAProxy itself is not exempt from this (it's only in 2.4 that we finally renamed all "sess*" functions that were acting on streams and not sessions). But that's why there are comments in the code to explain what the developer tries to do and why. If you find some areas that are too cryptic and not commented enough, feel free to ask here for precisions. You can even use "git blame" on the file then "git show" on the commit IDs to see who worked last on the lines you don't understand. Sometimes you'll find more explanations in the commit message. If you don't, feel free to ask, that's how it works! Willy
Re: Line 47 in src/queue.c "s * queue's lock."
On Thu, Jun 24, 2021 at 11:35:51PM +0200, Aleksandar Lazic wrote: > Hi. > > when someone works again on src/queue.c could be this typo fixed. > > http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/queue.c;h=6d3aa9a12bcd6078d1b5a76969da4104a6adb1bd;hb=HEAD#l47 > > ``` > 44 * - a pendconn_add() is only performed by the stream which will own > the > 45 * pendconn ; the pendconn is allocated at this moment and returned > ; it is > 46 * added to either the server or the proxy's queue while holding this > 47 s * queue's lock. > 48 * > ``` Indeed, thanks Aleks! Willy
Line 47 in src/queue.c "s * queue's lock."
Hi. when someone works again on src/queue.c could be this typo fixed. http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/queue.c;h=6d3aa9a12bcd6078d1b5a76969da4104a6adb1bd;hb=HEAD#l47 ``` 44 * - a pendconn_add() is only performed by the stream which will own the 45 * pendconn ; the pendconn is allocated at this moment and returned ; it is 46 * added to either the server or the proxy's queue while holding this 47 s * queue's lock. 48 * ``` Regards Alex
Re: SNI spoofing in HAproxy?
On Thu, Jun 24, 2021 at 04:07:33PM +0200, Tim Düsterhus wrote:
> Julien,
>
> On 6/24/21 3:40 PM, Julien Pivotto wrote:
> > >use_backend bob if { hdr(host) -m dom bob.com }
> > >use_backend alice if { hdr(host) -m dom alice.com }
> >
> > Thanks for taking the time to write this report.
> >
> > SNI and host header are indeed different.
> >
> > You should consider using req.ssl_sni instead of hdr(host).
> >
>
> NO! Using req.ssl_sni for request routing is unsafe and it will break more
> setups than it fixes.
>
> Browsers can and will open a single TCP / TLS connection for multiple
> unrelated hosts if the certificate presented on the first connection is
> valid for the other host. This is especially true for HTTP/2.
>
> To prevent the follow-up requests from being routed to the wrong backend you
> must use the 'host' header for routing -- or your certificates need to be
> non-overlapping.
>
> In fact HTTP 421 Misdirected Request was invented for the specific case of a
> web browser reusing an existing TCP connection for an unrelated domain.
FWIW I entirely second all what Tim said.
Willy
Re: SNI spoofing in HAproxy?
Julien,
On 6/24/21 3:40 PM, Julien Pivotto wrote:
use_backend bob if { hdr(host) -m dom bob.com }
use_backend alice if { hdr(host) -m dom alice.com }
Thanks for taking the time to write this report.
SNI and host header are indeed different.
You should consider using req.ssl_sni instead of hdr(host).
NO! Using req.ssl_sni for request routing is unsafe and it will break
more setups than it fixes.
Browsers can and will open a single TCP / TLS connection for multiple
unrelated hosts if the certificate presented on the first connection is
valid for the other host. This is especially true for HTTP/2.
To prevent the follow-up requests from being routed to the wrong backend
you must use the 'host' header for routing -- or your certificates need
to be non-overlapping.
In fact HTTP 421 Misdirected Request was invented for the specific case
of a web browser reusing an existing TCP connection for an unrelated domain.
Best regards
Tim Düsterhus
Re: SNI spoofing in HAproxy?
Dominik,
On 6/24/21 3:29 PM, Froehlich, Dominik wrote:
Not sure if you would call this a security issue, hence I am asking this on the
mailing list prior to opening a github issue:
This is also known as "Domain Fronting"
(https://en.wikipedia.org/wiki/Domain_fronting). It's not necessarily a
security issue and in fact might be the desired behavior in certain
circumstances.
I’ve noticed that it is really easy to bypass the check on client certificates
of a domain when the client can present a valid certificate for another domain.
Indeed. I also use client certificates for authentication and I'm aware
of this issue. That's why I validate that the SNI and Host match up for
my set up. In fact I added the 'strcmp' converter to HAProxy just for
this purpose. The documentation for 'strcmp' also gives an explicit
example on how to use it to prevent domain fronting:
http-request set-var(txn.host) hdr(host)
# Check whether the client is attempting domain fronting.
acl ssl_sni_http_host_match ssl_fc_sni,strcmp(txn.host) eq 0
https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#7.3.1-strcmp
My full rules look like this:
# Verify that SNI and Host header match
http-request set-var(txn.host)hdr(host)
http-request deny deny_status 400 unless { req.hdr_cnt(host) eq 1 }
http-request deny deny_status 421 unless {
ssl_fc_sni,strcmp(txn.host) eq 0 }
-
[…]
My questions:
* HAproxy does seem to treat SNI (L5) and HTTP Host Header (L7) as
unrelated. Is this true?
Yes.
* Applications offloading TLS to HAproxy usually trust that mTLS requests
coming in are validated correctly. They usually don’t revalidate the entire
certificate again and only check for the subject’s identity. Is there a way to
make SNI vs host header checking more strict?
Yes, see above.
* What’s the best practice to dispatch mTLS requests to backends? I’ve
used a host header based approach here but it shows the above vulnerabilities.
You *must* use the 'Host' header for routing. Using the SNI value for
routing is even more unsafe. But you also must validate that the SNI
matches up or that the client presented a valid certificate.
Best regards
Tim Düsterhus
Re: SNI spoofing in HAproxy?
On 24 Jun 13:29, Froehlich, Dominik wrote:
> Hi,
>
> Not sure if you would call this a security issue, hence I am asking this on
> the mailing list prior to opening a github issue:
>
> I’ve noticed that it is really easy to bypass the check on client
> certificates of a domain when the client can present a valid certificate for
> another domain.
>
> Consider this HAproxy config:
>
> global
> log /dev/log len 4096 format rfc3164 syslog info
>
> defaults
> log global
> mode http
> timeout connect 5s
> timeout client 1h
> timeout server 5s
>
> frontend myfrontend
> bind :443 ssl crt /etc/cert/server.pem crt-list /crt-list
> log-format "%ci:%cp [%tr] (%ID) %ft %b/%s %TR/%Tw/%Tc/%Tr/%Tt %ST %B %CC
> %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslc %sslv"
>
> capture request header Host len 256
>
> http-request set-header X-SSL-Client %[ssl_c_used]
> if { ssl_c_used }
> http-request set-header X-SSL-Client-Session-ID %[ssl_fc_session_id,hex]
> if { ssl_c_used }
> http-request set-header X-SSL-Client-Verify %[ssl_c_verify]
> if { ssl_c_used }
> http-request set-header X-SSL-Client-Subject-DN %{+Q}[ssl_c_s_dn]
> if { ssl_c_used }
> http-request set-header X-SSL-Client-Subject-CN %{+Q}[ssl_c_s_dn(cn)]
> if { ssl_c_used }
> http-request set-header X-SSL-Client-Issuer-DN%{+Q}[ssl_c_i_dn]
> if { ssl_c_used }
> http-request set-header X-SSL-Client-NotBefore%{+Q}[ssl_c_notbefore]
> if { ssl_c_used }
> http-request set-header X-SSL-Client-NotAfter %{+Q}[ssl_c_notafter]
> if { ssl_c_used }
>
> use_backend bob if { hdr(host) -m dom bob.com }
> use_backend alice if { hdr(host) -m dom alice.com }
Thanks for taking the time to write this report.
SNI and host header are indeed different.
You should consider using req.ssl_sni instead of hdr(host).
Regards,
>
> default_backend alice
>
> backend alice
> server alice localhost:8080 check
>
> backend bob
> server bob localhost:8081 check
>
> ---
> So this HAproxy hosts two domains alice.com and bob.com. It uses the
> following crt-list to make TLS connections:
>
> /etc/cert/server.pem [ca-file /alice.ca.pem verify required] *.alice.com
> /etc/cert/server.pem [ca-file /bob.ca.pem verify required] *.bob.com
>
> ---
> So any client connecting to alice.com must present a certificate signed by
> the Alice CA and any client connecting to bob.com must present a certificate
> signed by the Bob CA.
>
>
> However, I noticed that HAproxy does allow me to “spoof” the host header to
> bob.com even though I did a TLS handshake with alice.com. The request will be
> forwarded to bob.com with the alice.com certificate:
>
> curl -v -k --cert alice.com.crt --key alice.com.key --resolve
> www.alice.com:9443:127.0.0.1
> https://www.alice.com:9443/headers -H "host: www.bob.com"
>
> * Added www.alice.com:9443:127.0.0.1 to DNS cache
> * Hostname www.alice.com was found in DNS cache
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to www.alice.com (127.0.0.1) port 9443 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> * CAfile: /etc/ssl/cert.pem
> CApath: none
> (…)
> > GET /headers HTTP/1.1
> > Host: www.bob.com
> > User-Agent: curl/7.64.1
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < date: Thu, 24 Jun 2021 13:07:17 GMT
> < content-length: 578
> < content-type: text/plain; charset=utf-8
> <
> Hello from Bob!
>
> - Headers Received -
> Accept : [*/*]
> User-Agent : [curl/7.64.1]
> X-Ssl-Client : [1]
> X-Ssl-Client-Issuer-Dn :
> [/C=US/ST=California/L=AliceLand/O=Alice.com/CN=Alice.com Root
> CA/emailAddress=ad...@alice.com]
> X-Ssl-Client-Notafter : [220624125634Z]
> X-Ssl-Client-Notbefore : [210624125634Z]
> X-Ssl-Client-Session-Id :
> [D941ECCAACAFEBC5CB3AE17794B54DC3DFC7549C401DB20D7EC5ADC48244D3D0]
> X-Ssl-Client-Subject-Cn : [Alice]
> X-Ssl-Client-Subject-Dn :
> [/C=US/ST=Michigan/L=Detroit/O=Alice.com/CN=Alice/emailAddress=al...@alice.com]
> X-Ssl-Client-Verify : [0]
>
> ---
> So basically anyone who can get a client certificate from Alice.com can use
> it to also connect to Bob.com without getting validated against Bob’s CA.
>
> I’ve tested this with HAproxy 2.2.14.
>
> My questions:
>
> * HAproxy does seem to treat SNI (L5) and HTTP Host Header (L7) as
> unrelated. Is this true?
> * Applications offloading TLS to HAproxy usually trust that mTLS requests
> coming in are validated correctly. They usually don’t revalidate the entire
> certificate again and only check for the subject’s identity. Is there a way
> to make SNI vs host header checking more strict?
> * What’s the best practice to dispatch mTLS requests to backends? I’ve
> used a host header based approach here but it shows the above vulnerabilities.
>
>
> Best regards,
> Dom
--
(o-Julien Pivotto
//\
SNI spoofing in HAproxy?
Hi,
Not sure if you would call this a security issue, hence I am asking this on the
mailing list prior to opening a github issue:
I’ve noticed that it is really easy to bypass the check on client certificates
of a domain when the client can present a valid certificate for another domain.
Consider this HAproxy config:
global
log /dev/log len 4096 format rfc3164 syslog info
defaults
log global
mode http
timeout connect 5s
timeout client 1h
timeout server 5s
frontend myfrontend
bind :443 ssl crt /etc/cert/server.pem crt-list /crt-list
log-format "%ci:%cp [%tr] (%ID) %ft %b/%s %TR/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS
%tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslc %sslv"
capture request header Host len 256
http-request set-header X-SSL-Client %[ssl_c_used]if
{ ssl_c_used }
http-request set-header X-SSL-Client-Session-ID %[ssl_fc_session_id,hex] if
{ ssl_c_used }
http-request set-header X-SSL-Client-Verify %[ssl_c_verify] if
{ ssl_c_used }
http-request set-header X-SSL-Client-Subject-DN %{+Q}[ssl_c_s_dn]if
{ ssl_c_used }
http-request set-header X-SSL-Client-Subject-CN %{+Q}[ssl_c_s_dn(cn)]if
{ ssl_c_used }
http-request set-header X-SSL-Client-Issuer-DN%{+Q}[ssl_c_i_dn]if
{ ssl_c_used }
http-request set-header X-SSL-Client-NotBefore%{+Q}[ssl_c_notbefore] if
{ ssl_c_used }
http-request set-header X-SSL-Client-NotAfter %{+Q}[ssl_c_notafter]if
{ ssl_c_used }
use_backend bob if { hdr(host) -m dom bob.com }
use_backend alice if { hdr(host) -m dom alice.com }
default_backend alice
backend alice
server alice localhost:8080 check
backend bob
server bob localhost:8081 check
---
So this HAproxy hosts two domains alice.com and bob.com. It uses the following
crt-list to make TLS connections:
/etc/cert/server.pem [ca-file /alice.ca.pem verify required] *.alice.com
/etc/cert/server.pem [ca-file /bob.ca.pem verify required] *.bob.com
---
So any client connecting to alice.com must present a certificate signed by the
Alice CA and any client connecting to bob.com must present a certificate signed
by the Bob CA.
However, I noticed that HAproxy does allow me to “spoof” the host header to
bob.com even though I did a TLS handshake with alice.com. The request will be
forwarded to bob.com with the alice.com certificate:
curl -v -k --cert alice.com.crt --key alice.com.key --resolve
www.alice.com:9443:127.0.0.1
https://www.alice.com:9443/headers -H "host: www.bob.com"
* Added www.alice.com:9443:127.0.0.1 to DNS cache
* Hostname www.alice.com was found in DNS cache
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to www.alice.com (127.0.0.1) port 9443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
(…)
> GET /headers HTTP/1.1
> Host: www.bob.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< date: Thu, 24 Jun 2021 13:07:17 GMT
< content-length: 578
< content-type: text/plain; charset=utf-8
<
Hello from Bob!
- Headers Received -
Accept : [*/*]
User-Agent : [curl/7.64.1]
X-Ssl-Client : [1]
X-Ssl-Client-Issuer-Dn :
[/C=US/ST=California/L=AliceLand/O=Alice.com/CN=Alice.com Root
CA/emailAddress=ad...@alice.com]
X-Ssl-Client-Notafter : [220624125634Z]
X-Ssl-Client-Notbefore : [210624125634Z]
X-Ssl-Client-Session-Id :
[D941ECCAACAFEBC5CB3AE17794B54DC3DFC7549C401DB20D7EC5ADC48244D3D0]
X-Ssl-Client-Subject-Cn : [Alice]
X-Ssl-Client-Subject-Dn :
[/C=US/ST=Michigan/L=Detroit/O=Alice.com/CN=Alice/emailAddress=al...@alice.com]
X-Ssl-Client-Verify : [0]
---
So basically anyone who can get a client certificate from Alice.com can use it
to also connect to Bob.com without getting validated against Bob’s CA.
I’ve tested this with HAproxy 2.2.14.
My questions:
* HAproxy does seem to treat SNI (L5) and HTTP Host Header (L7) as
unrelated. Is this true?
* Applications offloading TLS to HAproxy usually trust that mTLS requests
coming in are validated correctly. They usually don’t revalidate the entire
certificate again and only check for the subject’s identity. Is there a way to
make SNI vs host header checking more strict?
* What’s the best practice to dispatch mTLS requests to backends? I’ve used
a host header based approach here but it shows the above vulnerabilities.
Best regards,
Dom
unable to load certificate chain from file error (Haproxy 2.2 )
Hello Guys, Issue - Haproxy config test is failing with below error unable to load certificate chain from file '/etc/haproxy/certs/mapersite2-vs/209037_mapersite2_default_mysite.pem'. Haproxy Version - haproxy version is 2.2.9-a947cc2 & tried 2.2.10-6a09215 also. Troubleshooting details - Validated the SSL certificate & CA Chain - Validation is passing & No issue found on chain also. - Created new file with certificate content and tried - No luck. - No other security issue like selinux context and all. - Tried to reproduce the issue from diffrent haproxy with the same certificate - No luck, it's throwing an error there also. Note: Its working fine on older Haproxy v1.8
Re: [PATCH 0/1] Replace issue templates by issue forms
Willy, On 6/24/21 4:17 AM, Willy Tarreau wrote: Thanks for this fast feedback, now merged! And now we already have the first issue created using an Issue Form: https://github.com/haproxy/haproxy/issues/1307 I think the result is looking great. I particularly like that the plaintext email no longer contains the HTML comments guiding the user through the template. Best regards Tim Düsterhus
