Re: HAProxy with native SSL support !
Hi David, On Wed, Sep 12, 2012 at 10:07:58PM +, David Torgerson wrote: haproxy SSL termination... Awesome!! I have been in the process of replacing our hardware appliances with a software based solution running in a virtualized environment. We currently have a project running in semi-beta mode to a closed set of users. Our current load is around 2500 new ssl TPS at 2048 bit ssl certs. We currently have a ssl cache of around 1,000,000. I am not sure how many ssl session reuses a second because our appliances do not capture that info ( I am guessing around 4000 because of data in other logs files ). When we open up our project to the public we expect much more traffic. Our software solution is using stunnel + haproxy (with accept-proxy) running on 10 Virtual machines per host, with 10 CPU's per virtual machine. We actually get better throughput using the VM's rather than the physical server due to software interrupts etc. I have been able to benchmark this solution using full TCP and HTTP requests at 9,500 new ssl TPS at 2048 bit certs with around 70,000 ssl reuses per second. We also get over 6Gbps of throughput. (we will have at least two physical servers for redundancy etc so double the numbers above). Impressive numbers ! Our software solution still has a pair L4 load balancers in front of the ssl terminators for redundancy, and we are using stunnel's shared ssl session cache so that we can avoid sticky TCP sessions from the L4 load balancer. In fact even without a shared session cache, you can linearly scale using SSL-ID stickiness with haproxy. Here's how to achieve this (I'm sure you'll find it fun to experiment) : - have a first layer of L3/L4 LBs - have a second layer of haproxy servers which only do TCP-based LB and NO SSL offloading. These ones point to a farm of SSL offloaders using the PROXY protocol, they stick on the SSL ID and share their SSL ID tables. Depending on the load, you can have either one or many such front LBs. - have a third layer of haproxy-based SSL offloaders with accept-proxy. These ones do not need to share their session cache because they're always accessed with either a new SSL session or an SSL session they already know. The first haproxy layer conf would probably look approximately like this : # 4 front TCP LBs with SSL stickiness peers front-peers peer 1.1.1.1:1024 peer 1.1.1.2:1024 listen ssl-front bind :443 stick-table type binary len 32 size 1m expire 30m peers front-peers acl clienthello req_ssl_hello_type 1 acl serverhello rep_ssl_hello_type 2 # use tcp content accepts to detects ssl client and server hello. tcp-request inspect-delay 5s tcp-request content accept if clienthello # no timeout on response inspect delay by default. tcp-response content accept if serverhello # SSL session ID (SSLID) may be present on a client or server hello. # Its length is coded on 1 byte at offset 43 and its value starts # at offset 44. # Match and learn on request if client hello. stick on payload_lv(43,1) if clienthello # Learn on response if server hello. stick store-response payload_lv(43,1) if serverhello option ssl-hello-chk server offload1 10.10.10.1:443 check send-proxy server offload2 10.10.10.2:443 check send-proxy server offload3 10.10.10.3:443 check send-proxy server offload4 10.10.10.4:443 check send-proxy server offload5 10.10.10.5:443 check send-proxy server offload6 10.10.10.6:443 check send-proxy server offload7 10.10.10.7:443 check send-proxy server offload8 10.10.10.8:443 check send-proxy And the second layer, you already know since you've built it, it's very basic : listen ssl-back bind :443 accept-proxy ssl That way, as you can see, nothing is shared between the SSL offloaders, which results in lower network traffic and overhead. You can even put them on multiple sites ! I tested the new haproxy SSL implementation I was able to hit closer to 12,000 new TPS (MUCH better performance) but it does not appear that haproxy currently shares the session cache across servers. I do know that it shares it across multi-process on the same box. Is this something that you are planning on implementing? Or is there some other way that I can achieve this? Yes this is something we want to implement at Exceliance. We have already done this for the Stud project, so we already have some working code to adapt to haproxy. It's just that there is no emergency for us to do it because the architecture above scales much better and already works. But we're certainly going to implement it, because it removes one layer of LBs and because it's cool :-) Thanks for your awesome work! Thanks for your detailed and interesting feedback ! Regards, Willy
Re: HAProxy with native SSL support !
A few links on our blogs related to Willy's mail and your problem: - SSLID persistence: http://blog.exceliance.fr/2011/07/04/maintain-affinity-based-on-ssl-session-id/ - Content switching based on SNI in HAProxy: http://blog.exceliance.fr/2012/04/13/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension/ - Proxy protocol and application: http://blog.exceliance.fr/2012/06/05/preserve-source-ip-address-despite-reverse-proxies/ - SSL offloading in HAProxy: http://blog.exceliance.fr/2012/09/10/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/ Scaling out SSL with Stud: http://blog.exceliance.fr/2011/11/07/scaling-out-ssl/ (but might be the same in HAProxy once applied to it) You mix a bit of these article and you can build the architecture Willy described. cheers
Re: HAProxy with native SSL support !
haproxy SSL termination... Awesome!! I have been in the process of replacing our hardware appliances with a software based solution running in a virtualized environment. We currently have a project running in semi-beta mode to a closed set of users. Our current load is around 2500 new ssl TPS at 2048 bit ssl certs. We currently have a ssl cache of around 1,000,000. I am not sure how many ssl session reuses a second because our appliances do not capture that info ( I am guessing around 4000 because of data in other logs files ). When we open up our project to the public we expect much more traffic. Our software solution is using stunnel + haproxy (with accept-proxy) running on 10 Virtual machines per host, with 10 CPU's per virtual machine. We actually get better throughput using the VM's rather than the physical server due to software interrupts etc. I have been able to benchmark this solution using full TCP and HTTP requests at 9,500 new ssl TPS at 2048 bit certs with around 70,000 ssl reuses per second. We also get over 6Gbps of throughput. (we will have at least two physical servers for redundancy etc so double the numbers above). Our software solution still has a pair L4 load balancers in front of the ssl terminators for redundancy, and we are using stunnel's shared ssl session cache so that we can avoid sticky TCP sessions from the L4 load balancer. I tested the new haproxy SSL implementation I was able to hit closer to 12,000 new TPS (MUCH better performance) but it does not appear that haproxy currently shares the session cache across servers. I do know that it shares it across multi-process on the same box. Is this something that you are planning on implementing? Or is there some other way that I can achieve this? Thanks for your awesome work! David T.
Re: HAProxy with native SSL support !
Hi Guillaume, On Tue, Sep 04, 2012 at 09:16:17AM +0200, Willy Tarreau wrote: Hi, On Tue, Sep 04, 2012 at 09:12:53AM +0200, Guillaume Castagnino wrote: Hi, Great news ! Just one question: is SNI support planned ? This would be great to allow one certificate per named vhost. Yes it's planned but not done yet. Emeric sees how to implement this but we wanted to propose an early pre-release so that we can get feedback while we're developing. Just an update on the subject, Emeric managed to get a working version yesterday evening with SNI and wildcard certs. We're still facing a bit too many #ifdefs to support the various openssl versions, and we're try to simplify this, but I think that by the end of the week-end I'll be able to release -dev12 with a working SSL+SNI implementation. If I get enough time, I'll try to also add some SNI-based ACLs. Regards, Willy
RE: HAProxy with native SSL support !
-(C)yassl doesn't support - by design - renegotiation. They also don't implement RFC4756 (secure renegotiation), see [3]. While this is not a security problem (from a server point of view), it will become an interoperability problem sooner or later, once browser vendors make the switch, and threat non-RFC4756 capable servers as broken [4], [5], [6]. I wonder how this is going to be fixed, if at all. typo: I obviously meant RFC5746, not RFC4756.
Re: HAProxy with native SSL support !
Hey Willy and the rest of Exceliance team, Awesome work, you guys rock! So looking forward to trying this on my systems. .pelle On Tue, Sep 4, 2012 at 1:37 AM, Willy Tarreau w...@1wt.eu wrote: Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's at least as fast. 7) other A lot remains to be done, mainly some of the aforementionned structres are still included in other ones, which simplified the split Once all the work is over, we should end up with less memory used per connection. This is important to better handle DDoS. At the moment, everything we could try seems to work fine. The SSL stacks well on top of the PROXY protocol, which is very important to build SSL offload farms (I'm sure Baptiste will want to write a blog article on the subject of using sub-$1000 machines to build large 100k+tps farms). Stats work over https too. Right now we're missing ACLs to match whether the traffic was SSL or clear, as well as logs. Both can be worked around by using distinct bind lines or even frontends. The doc is still clearly lacking, but we think that the config will change a little bit. Only the GNU makefile was updated, neither the BSD nor OSX were, they're a little trickier. If someone with one of these systems wants to update them, I'll happily accept the patches. What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL connections per second and 300 Mbps is what we got out of a dual-core Atom D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was enabled) :-) This is a bit more than stud and obviously much better than stunnel (which doesn't scale to more than a few hundred connections before the performance quickly drops). And older tests seem to indicate that with YaSSL we can get 30-40% more, maybe even more. We need to work with the YaSSL guys to slightly improve their cache management before this can become a default build option. Enough speaking, for those who want to test or even have the hardware to run more interesting benchmarks, the code was merged into the master branch and is in today's snapshot (20120904) here : http://haproxy.1wt.eu/download/1.5/src/snapshot/ Build
Re: HAProxy with native SSL support !
Just for the few who have already downloaded it, I have re-uploaded the snapshot with a fix (I failed my attempt at automatically renaming it so it ended up with the same name). There was a bug affecting the combination of accept-proxy + ssl which I just fixed. Regards, Willy
Re: HAProxy with native SSL support !
What a great news ! Let's go testing on internal applications. Congrats to the Exceliance team ! Hervé. On 09/04/2012 08:12 AM, Willy Tarreau wrote: Just for the few who have already downloaded it, I have re-uploaded the snapshot with a fix (I failed my attempt at automatically renaming it so it ended up with the same name). There was a bug affecting the combination of accept-proxy + ssl which I just fixed. Regards, Willy -- Hervé COMMOWICK Ingénieur systèmes et réseaux. http://www.rezulteo.com by Lizeo Online Media Group http://www.lizeo-online-media-group.com/ 42 quai Rambaud - 69002 Lyon (France) ⎮ ☎ +33 (0)4 63 05 95 30
Re: HAProxy with native SSL support !
On Tuesday, September 04, 2012 01:37:17 AM Willy Tarreau wrote: After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working on both sides of HAProxy. Very cool. Since HAProxy is event-driven, is anything done to avoid expensive crypto operations from blocking the event loop? Maybe that is what you intend for remote sockets ? Justin
Re: HAProxy with native SSL support !
On Mon, Sep 03, 2012 at 11:21:51PM -0700, Justin Karneges wrote: On Tuesday, September 04, 2012 01:37:17 AM Willy Tarreau wrote: After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working on both sides of HAProxy. Very cool. Since HAProxy is event-driven, is anything done to avoid expensive crypto operations from blocking the event loop? Maybe that is what you intend for remote sockets ? Exactly ! That said, as I've always said, the problem is not the relative cost of crypto but the computation time. On small devices, if an operation can take 10ms, it clearly is a problem. However on larger devices, if no operation takes more than 100 microseconds, then the added latency is minimal compared to system latency. That does not mean it's not a problem, just that when you're at 100% CPU, you might still have enough scheduling capacity to process the rest of the traffic without too much impact. But clearly when we can offload SSL to other processes, that will be much better. This can already be done BTW, but you have to chain proxies. For instance : global nbproc 4 listen front bind-process 1,2 bind :443 ssl haproxy.pem server http 127.0.0.1: send-proxy listen http bind-process 3 bind 127.0.0.1: accept-proxy mode http server back 127.0.0.1:8443 send-proxy listen back bind-process 4 bind 127.0.0.1:8443 accept-proxy server out 192.168.0.1:443 ssl You see, https is accepted on processes 1 and 2, which decipher the traffic and pass it to process 3, preserving IP information. This process does the HTTP handling and passes the traffic to process 4 which will re-cipher it before going to the server. Admittedly this is ugly and that's why we'd like to implement the remote sockets soon, so that you can write something like this : listen http bind-process 3 bind :443 ssl haproxy.pem process 1,2 mode http server out 192.168.0.1:443 ssl process 4 Also, using socketpair() internally will be much faster and have a lower overhead than TCP connect/accept. Regards, Willy
Re: HAProxy with native SSL support !
Hi Willy, congratulations to the whole Team. Thanks for this feature, now the SSL-chain is much simpler ;-) BR Aleks Am 04-09-2012 01:37, schrieb Willy Tarreau: Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's at least as fast. 7) other A lot remains to be done, mainly some of the aforementionned structres are still included in other ones, which simplified the split Once all the work is over, we should end up with less memory used per connection. This is important to better handle DDoS. At the moment, everything we could try seems to work fine. The SSL stacks well on top of the PROXY protocol, which is very important to build SSL offload farms (I'm sure Baptiste will want to write a blog article on the subject of using sub-$1000 machines to build large 100k+tps farms). Stats work over https too. Right now we're missing ACLs to match whether the traffic was SSL or clear, as well as logs. Both can be worked around by using distinct bind lines or even frontends. The doc is still clearly lacking, but we think that the config will change a little bit. Only the GNU makefile was updated, neither the BSD nor OSX were, they're a little trickier. If someone with one of these systems wants to update them, I'll happily accept the patches. What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL connections per second and 300 Mbps is what we got out of a dual-core Atom D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was enabled) :-) This is a bit more than stud and obviously much better than stunnel (which doesn't scale to more than a few hundred connections before the performance quickly drops). And older tests seem to indicate that with YaSSL we can get 30-40% more, maybe even more. We need to work with the YaSSL guys to slightly improve their cache management before this can become a default build option. Enough speaking, for those who want to test or even have the hardware to run more interesting benchmarks, the code was merged into the master branch and is in today's snapshot (20120904) here : http://haproxy.1wt.eu/download/1.5/src/snapshot/ Build it by passing USE_OPENSSL=1 on the make command
Re: HAProxy with native SSL support !
Hi, Great news ! Just one question: is SNI support planned ? This would be great to allow one certificate per named vhost. I'm currently stuck with nginx for the SSL layer because of this feature (I know that stunnel and stud recently get this feature, but not yet tested). This would allow me to strip one layer from my setup ;) Thanks ! Le mardi 04 septembre 2012 01:37:17 Willy Tarreau a écrit : Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's at least as fast. 7) other A lot remains to be done, mainly some of the aforementionned structres are still included in other ones, which simplified the split Once all the work is over, we should end up with less memory used per connection. This is important to better handle DDoS. At the moment, everything we could try seems to work fine. The SSL stacks well on top of the PROXY protocol, which is very important to build SSL offload farms (I'm sure Baptiste will want to write a blog article on the subject of using sub-$1000 machines to build large 100k+tps farms). Stats work over https too. Right now we're missing ACLs to match whether the traffic was SSL or clear, as well as logs. Both can be worked around by using distinct bind lines or even frontends. The doc is still clearly lacking, but we think that the config will change a little bit. Only the GNU makefile was updated, neither the BSD nor OSX were, they're a little trickier. If someone with one of these systems wants to update them, I'll happily accept the patches. What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL connections per second and 300 Mbps is what we got out of a dual-core Atom D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was enabled) :-) This is a bit more than stud and obviously much better than stunnel (which doesn't scale to more than a few hundred connections before the performance quickly drops). And older tests seem to indicate that with YaSSL we can get 30-40% more, maybe even more. We need to work with the YaSSL guys to slightly improve their cache management before this can become a default build option. Enough speaking, for
Re: HAProxy with native SSL support !
Hi, On Tue, Sep 04, 2012 at 09:12:53AM +0200, Guillaume Castagnino wrote: Hi, Great news ! Just one question: is SNI support planned ? This would be great to allow one certificate per named vhost. Yes it's planned but not done yet. Emeric sees how to implement this but we wanted to propose an early pre-release so that we can get feedback while we're developing. I'm currently stuck with nginx for the SSL layer because of this feature (I know that stunnel and stud recently get this feature, but not yet tested). This would allow me to strip one layer from my setup ;) I understand :-) Cheers, Willy
Re: HAProxy with native SSL support !
Le mar. 04 sept. 2012 09:12:53 CEST, Guillaume Castagnino a écrit : Hi, Great news ! Just one question: is SNI support planned ? This would be great to allow one certificate per named vhost. I'm currently stuck with nginx for the SSL layer because of this feature (I know that stunnel and stud recently get this feature, but not yet tested). This would allow me to strip one layer from my setup ;) Thanks ! Le mardi 04 septembre 2012 01:37:17 Willy Tarreau a écrit : Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's at least as fast. 7) other A lot remains to be done, mainly some of the aforementionned structres are still included in other ones, which simplified the split Once all the work is over, we should end up with less memory used per connection. This is important to better handle DDoS. At the moment, everything we could try seems to work fine. The SSL stacks well on top of the PROXY protocol, which is very important to build SSL offload farms (I'm sure Baptiste will want to write a blog article on the subject of using sub-$1000 machines to build large 100k+tps farms). Stats work over https too. Right now we're missing ACLs to match whether the traffic was SSL or clear, as well as logs. Both can be worked around by using distinct bind lines or even frontends. The doc is still clearly lacking, but we think that the config will change a little bit. Only the GNU makefile was updated, neither the BSD nor OSX were, they're a little trickier. If someone with one of these systems wants to update them, I'll happily accept the patches. What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL connections per second and 300 Mbps is what we got out of a dual-core Atom D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was enabled) :-) This is a bit more than stud and obviously much better than stunnel (which doesn't scale to more than a few hundred connections before the performance quickly drops). And older tests seem to indicate that with YaSSL we can get 30-40% more, maybe even more. We need to work with the YaSSL guys to slightly improve their cache management before this can become a default build option. Enough speaking, for those who want to test or even have
Re: HAProxy with native SSL support !
On 04/09/12 09:37, Willy Tarreau wrote: Have a lot of fun and please report your success/failures, Willy Small issue when compiling on CentOS 5.8 64bit against RPM versions of openssl-devel and e2fsprogs-devel-1.39-34.el5_8.1 I get the following: make TARGET=linux2628 USE_OPENSSL=1 gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_EPOLL -DENABLE_SEPOLL -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX -DCONFIG_HAPROXY_VERSION=\1.5-dev11\ -DCONFIG_HAPROXY_DATE=\2012/06/04\ \ -DBUILD_TARGET='linux2628' \ -DBUILD_ARCH='' \ -DBUILD_CPU='generic' \ -DBUILD_CC='gcc' \ -DBUILD_CFLAGS='-O2 -g -fno-strict-aliasing' \ -DBUILD_OPTIONS='' \ -c -o src/haproxy.o src/haproxy.c In file included from src/haproxy.c:83: include/proto/proto_http.h:111: error: conflicting types for ‘error_message’ /usr/include/et/com_err.h:37: error: previous declaration of ‘error_message’ was here make: *** [src/haproxy.o] Error 1
Re: HAProxy with native SSL support !
Great! Thanks Willy, De: Willy Tarreau w...@1wt.eu Para: haproxy@formilux.org Enviado: Martes 4 de septiembre de 2012 1:37 Asunto: HAProxy with native SSL support ! Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's at least as fast. 7) other A lot remains to be done, mainly some of the aforementionned structres are still included in other ones, which simplified the split Once all the work is over, we should end up with less memory used per connection. This is important to better handle DDoS. At the moment, everything we could try seems to work fine. The SSL stacks well on top of the PROXY protocol, which is very important to build SSL offload farms (I'm sure Baptiste will want to write a blog article on the subject of using sub-$1000 machines to build large 100k+tps farms). Stats work over https too. Right now we're missing ACLs to match whether the traffic was SSL or clear, as well as logs. Both can be worked around by using distinct bind lines or even frontends. The doc is still clearly lacking, but we think that the config will change a little bit. Only the GNU makefile was updated, neither the BSD nor OSX were, they're a little trickier. If someone with one of these systems wants to update them, I'll happily accept the patches. What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL connections per second and 300 Mbps is what we got out of a dual-core Atom D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was enabled) :-) This is a bit more than stud and obviously much better than stunnel (which doesn't scale to more than a few hundred connections before the performance quickly drops). And older tests seem to indicate that with YaSSL we can get 30-40% more, maybe even more. We need to work with the YaSSL guys to slightly improve their cache management before this can become a default build option. Enough speaking, for those who want to test or even have the hardware to run more interesting benchmarks, the code was merged into the master branch and is in today's snapshot (20120904) here : http://haproxy.1wt.eu/download/1.5/src/snapshot/ Build it by passing USE_OPENSSL=1 on the make command line. You should also
Re: HAProxy with native SSL support !
All, A small howto to play with it can be found here: http://blog.exceliance.fr/2012/09/04/howto-ssl-native-in-haproxy/ cheers
Re: HAProxy with native SSL support !
On Tue, Sep 04, 2012 at 05:56:14PM +1000, Duncan Hall wrote:
On 04/09/12 09:37, Willy Tarreau wrote:
Have a lot of fun and please report your success/failures,
Willy
Small issue when compiling on CentOS 5.8 64bit against RPM versions of
openssl-devel and e2fsprogs-devel-1.39-34.el5_8.1 I get the following:
make TARGET=linux2628 USE_OPENSSL=1
gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing
-DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY
-DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_EPOLL -DENABLE_SEPOLL
-DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX
-DCONFIG_HAPROXY_VERSION=\1.5-dev11\
-DCONFIG_HAPROXY_DATE=\2012/06/04\ \
-DBUILD_TARGET='linux2628' \
-DBUILD_ARCH='' \
-DBUILD_CPU='generic' \
-DBUILD_CC='gcc' \
-DBUILD_CFLAGS='-O2 -g -fno-strict-aliasing' \
-DBUILD_OPTIONS='' \
-c -o src/haproxy.o src/haproxy.c
In file included from src/haproxy.c:83:
include/proto/proto_http.h:111: error: conflicting types for ?error_message?
/usr/include/et/com_err.h:37: error: previous declaration of
?error_message? was here
make: *** [src/haproxy.o] Error 1
This is very strange, this one is as old as 2006 ! I suspect that the
issue comes from new includes which uncovered this conflict. Just got
it too here, it indeed only happens with USE_OPENSSL=1.
Just found another build issue in futex which does not build on RHEL
due to missing u32 declaration. Fixed too. With this it builds for me.
I'm attaching the two patches.
Thanks for the report,
Willy
From 783f25800cc0b10793817afd1d78615125be3a36 Mon Sep 17 00:00:00 2001
From: Willy Tarreau w...@1wt.eu
Date: Tue, 4 Sep 2012 12:19:04 +0200
Subject: [PATCH 1/2] BUILD: http: rename error_message http_error_message to
fix conflicts on RHEL
Duncan Hall reported a build issue on CentOS where error_message conflicts
with another system declaration when SSL is enabled. Rename the function.
---
include/proto/proto_http.h |2 +-
src/proto_http.c | 56 ++--
src/session.c |2 +-
3 files changed, 30 insertions(+), 30 deletions(-)
diff --git a/include/proto/proto_http.h b/include/proto/proto_http.h
index 69d932c..e7ad8ff 100644
--- a/include/proto/proto_http.h
+++ b/include/proto/proto_http.h
@@ -108,7 +108,7 @@ void http_reset_txn(struct session *s);
struct http_req_rule *parse_http_req_cond(const char **args, const char *file,
int linenum, struct proxy *proxy);
void free_http_req_rules(struct list *r);
-struct chunk *error_message(struct session *s, int msgnum);
+struct chunk *http_error_message(struct session *s, int msgnum);
/* to be used when contents change in an HTTP message */
#define http_msg_move_end(msg, bytes) do { \
diff --git a/src/proto_http.c b/src/proto_http.c
index 4faff52..d7208d2 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -669,7 +669,7 @@ static void http_server_error(struct session *t, struct
stream_interface *si,
* and message.
*/
-struct chunk *error_message(struct session *s, int msgnum)
+struct chunk *http_error_message(struct session *s, int msgnum)
{
if (s-be-errmsg[msgnum].str)
return s-be-errmsg[msgnum];
@@ -841,25 +841,25 @@ void http_return_srv_error(struct session *s, struct
stream_interface *si)
if (err_type SI_ET_QUEUE_ABRT)
http_server_error(s, si, SN_ERR_CLICL, SN_FINST_Q,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type SI_ET_CONN_ABRT)
http_server_error(s, si, SN_ERR_CLICL, SN_FINST_C,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type SI_ET_QUEUE_TO)
http_server_error(s, si, SN_ERR_SRVTO, SN_FINST_Q,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type SI_ET_QUEUE_ERR)
http_server_error(s, si, SN_ERR_SRVCL, SN_FINST_Q,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type SI_ET_CONN_TO)
http_server_error(s, si, SN_ERR_SRVTO, SN_FINST_C,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type SI_ET_CONN_ERR)
http_server_error(s, si, SN_ERR_SRVCL, SN_FINST_C,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else /* SI_ET_CONN_OTHER and others */
http_server_error(s, si, SN_ERR_INTERNAL, SN_FINST_C,
-
Re: HAProxy with native SSL support !
Congratulations Willy and Team... On Tue, Sep 4, 2012 at 3:59 PM, Willy Tarreau w...@1wt.eu wrote: On Tue, Sep 04, 2012 at 05:56:14PM +1000, Duncan Hall wrote: On 04/09/12 09:37, Willy Tarreau wrote: Have a lot of fun and please report your success/failures, Willy Small issue when compiling on CentOS 5.8 64bit against RPM versions of openssl-devel and e2fsprogs-devel-1.39-34.el5_8.1 I get the following: make TARGET=linux2628 USE_OPENSSL=1 gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_EPOLL -DENABLE_SEPOLL -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX -DCONFIG_HAPROXY_VERSION=\1.5-dev11\ -DCONFIG_HAPROXY_DATE=\2012/06/04\ \ -DBUILD_TARGET='linux2628' \ -DBUILD_ARCH='' \ -DBUILD_CPU='generic' \ -DBUILD_CC='gcc' \ -DBUILD_CFLAGS='-O2 -g -fno-strict-aliasing' \ -DBUILD_OPTIONS='' \ -c -o src/haproxy.o src/haproxy.c In file included from src/haproxy.c:83: include/proto/proto_http.h:111: error: conflicting types for ?error_message? /usr/include/et/com_err.h:37: error: previous declaration of ?error_message? was here make: *** [src/haproxy.o] Error 1 This is very strange, this one is as old as 2006 ! I suspect that the issue comes from new includes which uncovered this conflict. Just got it too here, it indeed only happens with USE_OPENSSL=1. Just found another build issue in futex which does not build on RHEL due to missing u32 declaration. Fixed too. With this it builds for me. I'm attaching the two patches. Thanks for the report, Willy -- -Rahul N. IT Department In2M Technologies Pvt Ltd. (Finicity) Website: www.finicity.com/india
Re: HAProxy with native SSL support !
Hi, Willy Thanks for this long time expected feature ! Have a lot of fun and please report your success/failures, There is an include issue in this snapshot on FreeBSD (witch is not I think ssl related) : gmake TARGET=freebsd USE_OPENSSL=1 gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_OPENSSL -DCONFIG_HAPROXY_VERSION=\1.5-dev11\ -DCONFIG_HAPROXY_DATE=\2012/06/04\ \ -DBUILD_TARGET='freebsd' \ -DBUILD_ARCH='' \ -DBUILD_CPU='generic' \ -DBUILD_CC='gcc' \ -DBUILD_CFLAGS='-O2 -g -fno-strict-aliasing' \ -DBUILD_OPTIONS='' \ -c -o src/haproxy.o src/haproxy.c gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_OPENSSL -DCONFIG_HAPROXY_VERSION=\1.5-dev11\ -DCONFIG_HAPROXY_DATE=\2012/06/04\ -c -o src/sessionhash.o src/sessionhash.c gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_OPENSSL -DCONFIG_HAPROXY_VERSION=\1.5-dev11\ -DCONFIG_HAPROXY_DATE=\2012/06/04\ -c -o src/base64.o src/base64.c gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing -DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE -DUSE_OPENSSL -DCONFIG_HAPROXY_VERSION=\1.5-dev11\ -DCONFIG_HAPROXY_DATE=\2012/06/04\ -c -o src/protocols.o src/protocols.c In file included from src/protocols.c:20: include/common/standard.h: In function 'is_addr': include/common/standard.h:572: error: 'AF_INET' undeclared (first use in this function) include/common/standard.h:572: error: (Each undeclared identifier is reported only once include/common/standard.h:572: error: for each function it appears in.) include/common/standard.h:574: error: 'AF_INET6' undeclared (first use in this function) include/common/standard.h: In function 'get_net_port': include/common/standard.h:586: error: 'AF_INET' undeclared (first use in this function) include/common/standard.h:588: error: 'AF_INET6' undeclared (first use in this function) include/common/standard.h: In function 'get_host_port': include/common/standard.h:598: error: 'AF_INET' undeclared (first use in this function) include/common/standard.h:600: error: 'AF_INET6' undeclared (first use in this function) include/common/standard.h: In function 'get_addr_len': include/common/standard.h:610: error: 'AF_INET' undeclared (first use in this function) include/common/standard.h:612: error: 'AF_INET6' undeclared (first use in this function) include/common/standard.h:614: error: 'AF_UNIX' undeclared (first use in this function) include/common/standard.h: In function 'set_net_port': include/common/standard.h:624: error: 'AF_INET' undeclared (first use in this function) include/common/standard.h:626: error: 'AF_INET6' undeclared (first use in this function) include/common/standard.h: In function 'set_host_port': include/common/standard.h:636: error: 'AF_INET' undeclared (first use in this function) include/common/standard.h:638: error: 'AF_INET6' undeclared (first use in this function) gmake: *** [src/protocols.o] Erreur 1 A workaround is to include sys/socket.h in include/common/standard.h. Once corrected, ssl support successfully build and run on FreeBSD 8.3. Joris Willy
Re: HAProxy with native SSL support !
Hi Joris, On Tue, Sep 04, 2012 at 01:45:29PM +0200, joris dedieu wrote: Hi, Willy Thanks for this long time expected feature ! Have a lot of fun and please report your success/failures, There is an include issue in this snapshot on FreeBSD (witch is not I think ssl related) : (...) A workaround is to include sys/socket.h in include/common/standard.h. Once corrected, ssl support successfully build and run on FreeBSD 8.3. Joris Thanks for the report, patch merged and attached. Willy From d50265aa0ed40969f2659c001e261e4ca78d011f Mon Sep 17 00:00:00 2001 From: Willy Tarreau w...@1wt.eu Date: Tue, 4 Sep 2012 14:18:33 +0200 Subject: [PATCH] BUILD: include sys/socket.h to fix build failure on FreeBSD Joris Dedieu reported that include/common/standard.h needs this. --- include/common/standard.h |1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/include/common/standard.h b/include/common/standard.h index 511a1dd..93b9e36 100644 --- a/include/common/standard.h +++ b/include/common/standard.h @@ -26,6 +26,7 @@ #include string.h #include time.h #include sys/types.h +#include sys/socket.h #include sys/un.h #include netinet/in.h #include common/config.h -- 1.7.1
RE: HAProxy with native SSL support !
Willy, this is huge! Great, great work! A few comments/questions: - are you running latest and greatest openssl on demo.1wt.eu? I am asking because Secure Renegotiation doesn't seem to be supported [1]. Older (1.0.0?) releases seem to have a higher memory overhead as well, iirc. - I see you have implemented [nosslv3] [notlsv1] to limit the SSL/TLS protocols. I do prefer the nginx approach [2], positively specifying the protocols used. This is much more flexible. Even if today the 2 options are fine for generic HTTPS proxies in the Internet, there may be corner cases or future needs to specify the protocols in a more flexible manner, like nginx does. - about ssl ciphers, we should probably steal the default from nginx, by using HIGH:!aNULL:!MD5 [3], so we don't enable obsolete ciphers like DES by default [1]. - any way to validate certificates of the backend? Perhaps a insecure/secure flag should be introduced. The exact host/domainname is needed as well to validate the backend certificate. Consider configurations where the backend server is not local, but can be contacted only over the big bad internet. I'm also thinking of a certificate validation in the health checks. - probably not a much requested feature, but for the record: any plan to make client based certificates work? Could be interesting for example to authenticate haproxy on the backend servers. There are probably people with frontend client certificate use-cases as well. However general interest in this feature is probably low. - you seem to be interested in supporting native yassl. Would you then drop openssl support or would the idea be to support both native yassl and the openssl api? Congratulations to this big step in haproxy development! Lukas [1] https://www.ssllabs.com/ssltest/analyze.html?d=https%3A%2F%2Fdemo.1wt.eu%2F [2] http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols [3] http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers Date: Tue, 4 Sep 2012 01:37:17 +0200 From: w...@1wt.eu To: haproxy@formilux.org Subject: HAProxy with native SSL support ! Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's
Re: HAProxy with native SSL support !
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, On 04/Sep - 01:37, Willy Tarreau w...@1wt.eu wrote: | Have a lot of fun and please report your success/failures, | Willy Thanks a lot for this useful feature. It works well on a dual PPC64 Linux server. I wrote a small path to add the SSL_OP_CIPHER_SERVER_PREFERENCE OpenSSL option to frontend, if the 'prefer-server-ciphers' keyword is set. https://0x1.fr/files/patchs/haproxy-ss-20120904_prefer_server_ciphers.patch Example : bind 10.11.12.13 ssl /etc/haproxy/ssl/cert.pem ciphers RC4:HIGH:!aNULL:!MD5 prefer-server-ciphers This option mitigate the effect of the BEAST Attack (as I understand), and it equivalent to : - Apache HTTPd SSLHonorCipherOrder option. - Nginx ssl_prefer_server_ciphers option. Maybe it can be useful to add OpenSSL option directly in the haproxy configuration as the 'options' keyword in stunnel. Best regards. - -- David BERARD contact(at)davidberard.fr GPG|PGP KeyId 0xC8533354 GPG|PGP Key http://davidberard.fr/C8533354.gpgkey * No electrons were harmed in the transmission of this email * -BEGIN PGP SIGNATURE- iEYEARECAAYFAlBF/uAACgkQOL7fhchTM1Q0PQCgqbxmjbxKokJ2dFX28dbfjml4 KOcAnja+g7reSbHJVub8P4HYrcz1Q/TG =PD86 -END PGP SIGNATURE-
Re: HAProxy with native SSL support ! = fix for ssl_cert
Emeric reported that the build fails without USE_OPENSSL, which is caused
by a last-minute change I did yesterday evening. It shows up as ssl_cert
not being part of a structure.
If you get this, please use the attached patch.
Regards,
Willy
From ff9f7698fcefef66bceb1ec32a3da8b14947a594 Mon Sep 17 00:00:00 2001
From: Willy Tarreau w...@1wt.eu
Date: Tue, 4 Sep 2012 15:13:20 +0200
Subject: [PATCH] BUILD: fix build error without SSL (ssl_cert)
One last-minute optimization broke the build without SSL support.
Move ssl_cert out of the #ifdef/#endif and it's OK.
---
include/types/protocols.h |2 +-
src/haproxy.c |2 --
2 files changed, 1 insertions(+), 3 deletions(-)
diff --git a/include/types/protocols.h b/include/types/protocols.h
index 4e40a67..b075ef6 100644
--- a/include/types/protocols.h
+++ b/include/types/protocols.h
@@ -130,8 +130,8 @@ struct listener {
char *interface;/* interface name or NULL */
int maxseg; /* for TCP, advertised MSS */
-#ifdef USE_OPENSSL
char *ssl_cert; /* ssl certificate */
+#ifdef USE_OPENSSL
struct {
SSL_CTX *ctx;
char *ciphers; /* cipher suite to use if non-null */
diff --git a/src/haproxy.c b/src/haproxy.c
index adf2614..4e75080 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -996,9 +996,7 @@ void deinit(void)
l_next = l-next;
unbind_listener(l);
delete_listener(l);
-#ifdef USE_OPENSSL
free(l-ssl_cert);
-#endif
free(l-name);
free(l-counters);
free(l);
--
1.7.1
Re: HAProxy with native SSL support !
Hi Lukas, On Tue, Sep 04, 2012 at 03:05:14PM +0200, Lukas Tribus wrote: Willy, this is huge! Great, great work! A few comments/questions: - are you running latest and greatest openssl on demo.1wt.eu? I am asking because Secure Renegotiation doesn't seem to be supported [1]. Older (1.0.0?) releases seem to have a higher memory overhead as well, iirc. Just checked and no, it's 0.9.8 only on this box. Anyway this is not very important on this box since it gets very low load. Also this was more for the test than anything else, so I expected that someone would even complain about the self-signed cert :-) - I see you have implemented [nosslv3] [notlsv1] to limit the SSL/TLS protocols. I do prefer the nginx approach [2], positively specifying the protocols used. This is much more flexible. Even if today the 2 options are fine for generic HTTPS proxies in the Internet, there may be corner cases or future needs to specify the protocols in a more flexible manner, like nginx does. Point taken. However, it's important to know that SSL uses the negative form, which is why I preferred to use the same one. You have options to *disable* use of V2/V3/TLS, not to enable them. Thus I find it more durable to stay on the same logics because if openssl 1.2 comes with support for TLSv5, it will be offered by default until we have an option to disable it. With the positive approach, it will be offered by default until unknown, and once known, it will suddenly be disabled for the same configs unless users explicitly ask for it. I'm rather for long-term config compatibility you know :-) - about ssl ciphers, we should probably steal the default from nginx, by using HIGH:!aNULL:!MD5 [3], so we don't enable obsolete ciphers like DES by default [1]. OK, right now nothing is set by default and indeed it could be an idea. - any way to validate certificates of the backend? Perhaps a insecure/secure flag should be introduced. The exact host/domainname is needed as well to validate the backend certificate. Consider configurations where the backend server is not local, but can be contacted only over the big bad internet. I'm also thinking of a certificate validation in the health checks. We're considering adding some verify options exactly for this, in order to only accept some backend certificates as well as to validate some users. They are not implemented yet, but this is planned. - probably not a much requested feature, but for the record: any plan to make client based certificates work? Could be interesting for example to authenticate haproxy on the backend servers. There are probably people with frontend client certificate use-cases as well. However general interest in this feature is probably low. This was our primary concern, but in order to get these, we needed native SSL :-) So yes this will come, maybe soon, maybe later. We hope no later than the end of the year depending on our available time, but maybe much sooner too. I also want to be able to pass client cert info into HTTP headers, logs and ACLs. - you seem to be interested in supporting native yassl. Would you then drop openssl support or would the idea be to support both native yassl and the openssl api? No, we wouldn't drop it, openssl is the de-facto standard. However if we see a much higher performance level by using the native API, we'd probably write a 3rd data layer dedicated to yassl, and would probably rename the current SSL data layer so that we can choose the one we want at build time. Congratulations to this big step in haproxy development! Thanks :-) Regards, willy
Re: HAProxy with native SSL support !
Hi David, On Tue, Sep 04, 2012 at 03:15:13PM +0200, David BERARD wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, On 04/Sep - 01:37, Willy Tarreau w...@1wt.eu wrote: | Have a lot of fun and please report your success/failures, | Willy Thanks a lot for this useful feature. It works well on a dual PPC64 Linux server. Excellent, thanks for the report. I wrote a small path to add the SSL_OP_CIPHER_SERVER_PREFERENCE OpenSSL option to frontend, if the 'prefer-server-ciphers' keyword is set. https://0x1.fr/files/patchs/haproxy-ss-20120904_prefer_server_ciphers.patch Example : bind 10.11.12.13 ssl /etc/haproxy/ssl/cert.pem ciphers RC4:HIGH:!aNULL:!MD5 prefer-server-ciphers This option mitigate the effect of the BEAST Attack (as I understand), and it equivalent to : - Apache HTTPd SSLHonorCipherOrder option. - Nginx ssl_prefer_server_ciphers option. OK I did not know there was such a workaround. Thanks for the patch, I'm going to apply it now. Maybe it can be useful to add OpenSSL option directly in the haproxy configuration as the 'options' keyword in stunnel. No, your current solution is better, as we'd like to get such options for each bind line. So your patch is perfect :-) Best regards, Willy
RE: HAProxy with native SSL support !
Point taken. However, it's important to know that SSL uses the negative form, which is why I preferred to use the same one. You have options to *disable* use of V2/V3/TLS, not to enable them. Thus I find it more durable to stay on the same logics because if openssl 1.2 comes with support for TLSv5, it will be offered by default until we have an option to disable it. With the positive approach, it will be offered by default until unknown, and once known, it will suddenly be disabled for the same configs unless users explicitly ask for it. I'm rather for long-term config compatibility you know :-) Right, I fully agree with you. However if we see a much higher performance level by using the native API, we'd probably write a 3rd data layer dedicated to yassl, and would probably rename the current SSL data layer so that we can choose the one we want at build time. Ok, thats great. I have another comment about yassl: It seems that CyaSSL, the ANSI C based edition of yaSSL is recommended over yasll (see [1]). For example TLS1.2 doesn't seem to be supported on yasll yet, while CyaSSL already supports it. I do not have any programming knowlegde, so please execuse my probably stupid question, but whats the reason to impement yasll instead of CyaSSL? Can't a C-lib be implemented in a C++ application? Regards, Lukas [1] http://www.yassl.com/yaSSL/Products-yassl.html Date: Tue, 4 Sep 2012 15:26:24 +0200 From: w...@1wt.eu To: luky...@hotmail.com CC: haproxy@formilux.org Subject: Re: HAProxy with native SSL support ! Hi Lukas, On Tue, Sep 04, 2012 at 03:05:14PM +0200, Lukas Tribus wrote: Willy, this is huge! Great, great work! A few comments/questions: - are you running latest and greatest openssl on demo.1wt.eu? I am asking because Secure Renegotiation doesn't seem to be supported [1]. Older (1.0.0?) releases seem to have a higher memory overhead as well, iirc. Just checked and no, it's 0.9.8 only on this box. Anyway this is not very important on this box since it gets very low load. Also this was more for the test than anything else, so I expected that someone would even complain about the self-signed cert :-) - I see you have implemented [nosslv3] [notlsv1] to limit the SSL/TLS protocols. I do prefer the nginx approach [2], positively specifying the protocols used. This is much more flexible. Even if today the 2 options are fine for generic HTTPS proxies in the Internet, there may be corner cases or future needs to specify the protocols in a more flexible manner, like nginx does. Point taken. However, it's important to know that SSL uses the negative form, which is why I preferred to use the same one. You have options to *disable* use of V2/V3/TLS, not to enable them. Thus I find it more durable to stay on the same logics because if openssl 1.2 comes with support for TLSv5, it will be offered by default until we have an option to disable it. With the positive approach, it will be offered by default until unknown, and once known, it will suddenly be disabled for the same configs unless users explicitly ask for it. I'm rather for long-term config compatibility you know :-) - about ssl ciphers, we should probably steal the default from nginx, by using HIGH:!aNULL:!MD5 [3], so we don't enable obsolete ciphers like DES by default [1]. OK, right now nothing is set by default and indeed it could be an idea. - any way to validate certificates of the backend? Perhaps a insecure/secure flag should be introduced. The exact host/domainname is needed as well to validate the backend certificate. Consider configurations where the backend server is not local, but can be contacted only over the big bad internet. I'm also thinking of a certificate validation in the health checks. We're considering adding some verify options exactly for this, in order to only accept some backend certificates as well as to validate some users. They are not implemented yet, but this is planned. - probably not a much requested feature, but for the record: any plan to make client based certificates work? Could be interesting for example to authenticate haproxy on the backend servers. There are probably people with frontend client certificate use-cases as well. However general interest in this feature is probably low. This was our primary concern, but in order to get these, we needed native SSL :-) So yes this will come, maybe soon, maybe later. We hope no later than the end of the year depending on our available time, but maybe much sooner too. I also want to be able to pass client cert info into HTTP headers, logs and ACLs. - you seem to be interested in supporting native yassl. Would you then drop openssl support or would the idea be to support both native yassl and the openssl api? No, we wouldn't drop it, openssl is the de-facto standard. However if we see a much higher
Re: HAProxy with native SSL support !
On Tue, Sep 04, 2012 at 04:12:43PM +0200, Lukas Tribus wrote: However if we see a much higher performance level by using the native API, we'd probably write a 3rd data layer dedicated to yassl, and would probably rename the current SSL data layer so that we can choose the one we want at build time. Ok, thats great. I have another comment about yassl: It seems that CyaSSL, the ANSI C based edition of yaSSL is recommended over yasll (see [1]). For example TLS1.2 doesn't seem to be supported on yasll yet, while CyaSSL already supports it. I do not have any programming knowlegde, so please execuse my probably stupid question, but whats the reason to impement yasll instead of CyaSSL? Can't a C-lib be implemented in a C++ application? In fact when I say yassl, I really mean CyaSSL. Regards, Willy
Re: HAProxy with native SSL support !
Great ! Thanks to the team ! :-) 2012/9/4 Willy Tarreau w...@1wt.eu On Tue, Sep 04, 2012 at 04:12:43PM +0200, Lukas Tribus wrote: However if we see a much higher performance level by using the native API, we'd probably write a 3rd data layer dedicated to yassl, and would probably rename the current SSL data layer so that we can choose the one we want at build time. Ok, thats great. I have another comment about yassl: It seems that CyaSSL, the ANSI C based edition of yaSSL is recommended over yasll (see [1]). For example TLS1.2 doesn't seem to be supported on yasll yet, while CyaSSL already supports it. I do not have any programming knowlegde, so please execuse my probably stupid question, but whats the reason to impement yasll instead of CyaSSL? Can't a C-lib be implemented in a C++ application? In fact when I say yassl, I really mean CyaSSL. Regards, Willy -- Falco SCHMUTZ http://fr.linkedin.com/in/fschmutz E-mail : fschm...@premaccess.com Mobile : +33 6 80 22 00 29 Fixe Suisse : +41 44 586 77 74 Fax Suisse : +41 26 347 28 90 Fax France : +33 9 72 12 19 25 premaccess sàrl Route des Arsenaux 41 - CP 132 CH-1705 Fribourg, Suisse http://www.premaccess.com Attention : le présent message et toutes les pièces jointes sont confidentiels et établis a l'attention exclusive du ou des destinataire(s) indique(s). Toute autre diffusion ou utilisation non autorisée est interdite. Si vous recevez ce message par erreur, veuillez immédiatement en avertir l'expéditeur par e-mail en retour, détruire le message et vous abstenir de toute référence aux informations qui y figurent afin d'éviter les sanctions attachées a la divulgation et a l'utilisation d'informations confidentielles. Les messages électroniques sont susceptibles d'altération. Nous déclinons toute responsabilité en cas d'altération ou de falsification du présent message. Warning : this e-mail and any files attached are confidential and intended solely to the named addressee(s). Any unauthorised dissemination or use is strictly prohibited. If you received this e-mail in error, please immediately notify the sender by reply e-mail and then delete the e-mail from your system. Please do not copy, use or make reference to it for any purpose, or disclose its contents to any person : to do so could expose you to sanctions. E-mails can be altered or falsified. We shall not be liable for any alteration or falsification on this e-mail.
Re: HAProxy with native SSL support !
On Tuesday, September 04, 2012 08:41:44 AM Willy Tarreau wrote: On Mon, Sep 03, 2012 at 11:21:51PM -0700, Justin Karneges wrote: On Tuesday, September 04, 2012 01:37:17 AM Willy Tarreau wrote: After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working on both sides of HAProxy. Very cool. Since HAProxy is event-driven, is anything done to avoid expensive crypto operations from blocking the event loop? Maybe that is what you intend for remote sockets ? Exactly ! That said, as I've always said, the problem is not the relative cost of crypto but the computation time. On small devices, if an operation can take 10ms, it clearly is a problem. However on larger devices, if no operation takes more than 100 microseconds, then the added latency is minimal compared to system latency. That does not mean it's not a problem, just that when you're at 100% CPU, you might still have enough scheduling capacity to process the rest of the traffic without too much impact. But clearly when we can offload SSL to other processes, that will be much better. This can already be done BTW, but you have to chain proxies. For instance : global nbproc 4 listen front bind-process 1,2 bind :443 ssl haproxy.pem server http 127.0.0.1: send-proxy listen http bind-process 3 bind 127.0.0.1: accept-proxy mode http server back 127.0.0.1:8443 send-proxy listen back bind-process 4 bind 127.0.0.1:8443 accept-proxy server out 192.168.0.1:443 ssl You see, https is accepted on processes 1 and 2, which decipher the traffic and pass it to process 3, preserving IP information. This process does the HTTP handling and passes the traffic to process 4 which will re-cipher it before going to the server. Admittedly this is ugly and that's why we'd like to implement the remote sockets soon, so that you can write something like this : listen http bind-process 3 bind :443 ssl haproxy.pem process 1,2 mode http server out 192.168.0.1:443 ssl process 4 Also, using socketpair() internally will be much faster and have a lower overhead than TCP connect/accept. Usually, the most expensive operations in TLS are the public key ones at the start of a negotiation (and possibly a renegotiation, though I'm not sure of protocol details there). However, pretty much all other TLS traffic should be fast to execute. I wonder if there is a way to differentiate these steps? This is how you'd ensure that a bunch of new https connections do not cause slowdown of existing https connections. I have tried in the past to use OpenSSL in an event-driven application in a way that does not block the event loop. This was for a local desktop application that used smart cards, where an OpenSSL call might block for 5 seconds while it hits the hardware. Since the expected number of TLS sessions in the application was small, I simply made one thread per TLS session, and funneled all results back to the event-driven main thread. Of course a thread per connection approach would not work if you want to handle thousands of connections like HAProxy. One day I wanted to go back and rewrite my code such that I could send operations to separate threads or keep them in the main thread based on a given TLS session state. This way, I could make it so only the TLS handshakes (with the expensive RSA operations and such) would need concurrency. Then the number of threads needed would be the number of expected simultaneous handshakes, which would be a far more manageable number (20?) than expected simultaneous sessions (thousands). This is not an answer for you, but just a brain dump of my thoughts on the subject. Justin
RE: HAProxy with native SSL support !
Hi, In fact when I say yassl, I really mean CyaSSL. Ok, great. A few more comments about (C)yassl: - development of new features is obviously not as fast as in OpenSSL. For example TLS SNI is not supported yet (ETA: next release) [1]. This feature was introduced in 2007 (0.9.8f) in the openssl implementation, and it was enabled by default in 2009 (0.9.8j), [2]. That doesn't mean yassl isn't suited for haproxy, one just needs to be aware that yassl is NOT a faster openssl with lower memory overhead. Its a different implementation with different goals and OpenSSL will remain the full-featured reference, with company's like Google introducing new features and providing bugfixes for it. - (C)yassl doesn't support - by design - renegotiation. They also don't implement RFC4756 (secure renegotiation), see [3]. While this is not a security problem (from a server point of view), it will become an interoperability problem sooner or later, once browser vendors make the switch, and threat non-RFC4756 capable servers as broken [4], [5], [6]. I wonder how this is going to be fixed, if at all. - I also believe that most of the haproxy maintainers will compile haproxy with openssl, and yassl will be more or less available only to those compiling haproxy on their own. In fact, libyassl/libcyassl is not even available in debian for example [7], and they don't like libs linked statically to their packages. Anyway, time will tell and the mentioned issues are no showstoppers. Regards, Lukas [1] http://www.yassl.com/yaSSL/Blog/Entries/2012/8/15_CyaSSL_to_include_SNI_%28Server_Name_Indication%29_in_Upcoming_Release.html [2] http://www.openssl.org/news/changelog.html [3] http://yassl.com/yaSSL/Docs-cyassl-manual-9-library-design.html [4] https://wiki.mozilla.org/Security:Renegotiation#security.ssl.treat_unsafe_negotiation_as_broken [5] https://wiki.mozilla.org/Security:Renegotiation#security.ssl.require_safe_negotiation [6] http://my.opera.com/securitygroup/blog/2010/11/04/a-few-results-from-the-tls-prober [7] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=664553
HAProxy with native SSL support !
Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's at least as fast. 7) other A lot remains to be done, mainly some of the aforementionned structres are still included in other ones, which simplified the split Once all the work is over, we should end up with less memory used per connection. This is important to better handle DDoS. At the moment, everything we could try seems to work fine. The SSL stacks well on top of the PROXY protocol, which is very important to build SSL offload farms (I'm sure Baptiste will want to write a blog article on the subject of using sub-$1000 machines to build large 100k+tps farms). Stats work over https too. Right now we're missing ACLs to match whether the traffic was SSL or clear, as well as logs. Both can be worked around by using distinct bind lines or even frontends. The doc is still clearly lacking, but we think that the config will change a little bit. Only the GNU makefile was updated, neither the BSD nor OSX were, they're a little trickier. If someone with one of these systems wants to update them, I'll happily accept the patches. What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL connections per second and 300 Mbps is what we got out of a dual-core Atom D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was enabled) :-) This is a bit more than stud and obviously much better than stunnel (which doesn't scale to more than a few hundred connections before the performance quickly drops). And older tests seem to indicate that with YaSSL we can get 30-40% more, maybe even more. We need to work with the YaSSL guys to slightly improve their cache management before this can become a default build option. Enough speaking, for those who want to test or even have the hardware to run more interesting benchmarks, the code was merged into the master branch and is in today's snapshot (20120904) here : http://haproxy.1wt.eu/download/1.5/src/snapshot/ Build it by passing USE_OPENSSL=1 on the make command line. You should also include support for linux-2.6 options for better results : make TARGET=linux2628 USE_OPENSSL=1 If all goes well by the end of the week, I'll issue -dev12, but I expect that we'll have some bugs to fix
Re: HAProxy with native SSL support !
Great day indeed, can't wait to do some tests. Thanks On 3 September 2012 20:37, Willy Tarreau w...@1wt.eu wrote: Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's at least as fast. 7) other A lot remains to be done, mainly some of the aforementionned structres are still included in other ones, which simplified the split Once all the work is over, we should end up with less memory used per connection. This is important to better handle DDoS. At the moment, everything we could try seems to work fine. The SSL stacks well on top of the PROXY protocol, which is very important to build SSL offload farms (I'm sure Baptiste will want to write a blog article on the subject of using sub-$1000 machines to build large 100k+tps farms). Stats work over https too. Right now we're missing ACLs to match whether the traffic was SSL or clear, as well as logs. Both can be worked around by using distinct bind lines or even frontends. The doc is still clearly lacking, but we think that the config will change a little bit. Only the GNU makefile was updated, neither the BSD nor OSX were, they're a little trickier. If someone with one of these systems wants to update them, I'll happily accept the patches. What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL connections per second and 300 Mbps is what we got out of a dual-core Atom D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was enabled) :-) This is a bit more than stud and obviously much better than stunnel (which doesn't scale to more than a few hundred connections before the performance quickly drops). And older tests seem to indicate that with YaSSL we can get 30-40% more, maybe even more. We need to work with the YaSSL guys to slightly improve their cache management before this can become a default build option. Enough speaking, for those who want to test or even have the hardware to run more interesting benchmarks, the code was merged into the master branch and is in today's snapshot (20120904) here : http://haproxy.1wt.eu/download/1.5/src/snapshot/ Build it by passing USE_OPENSSL=1 on the make command line. You should also include
Re: HAProxy with native SSL support !
Awesome news ! I have been waiting for this for a while. :) On Sep 3, 2012, at 4:37 PM, Willy Tarreau wrote: Hi all, today is a great day (could say night considering the time I'm posting) ! After several months of efforts by the Exceliance team, we managed to rework all the buffer and connection layers in order to get SSL working bon both sides of HAProxy. The code is still in preview, we can't break it anymore but considering that we've fixed some bugs today, I'm sure that some still remain in the 100+ patches and 16000 lines of patches this work required (not counting the many ones that were abandonned or re-merged multiple times). The code is still going to change because we're getting closer to something which will allow outgoing connections to be reused, resulting in keep-alive on both sides. But not yet, be patient. What's done right now ? 1) connections Connections are independant entities which can be instanciated without allocating a full session and its buffers. Connections are responsible for handshakes and control, and pass data to buffers. Connection-level TCP-request rules, the PROXY protocol and SSL handshakes are processed at the connection level. 2) buffers buffers have been split in three: channel (the tube where the data flows), buffer (where data is temporarily stored for analysis or forwarding) and optionally the pipe (stored in kernel area for forwarding only). New buffers only handle data without consideration for what it's used for. Health checks are currently being migrated to use this with connections. 3) data I/O data I/O are now performed between a connection and a buffer. We have two data-layer operations now : raw and ssl. It is very easy to add new ones now, we're even wondering whether it would make sense to write one dedicated to yassl in native mode (without the openssl API). 4) socket I/O at the moment we only support normal sockets, but the design considered remote sockets so that we could off-load heavy processing to external processes (eg: HTTP on one process, SSL on two other). Remote sockets have not been started yet but surely will. SHMs have also been considered to emulate sockets. 5) configuration Configuration has been extended to support the ssl keyword on bind lines and on server lines. For both, the syntax is : ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1] cert.pem is a PEM file made by concatenating the .crt and the .key of a certificate. eg: bind :443 ssl /etc/haproxy/pub.pem server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1 6) session management SSL sessions are stored in a shared memory cache, allowing haproxy to run with nbproc 1 and still work correctly. This is the session cache we developped for stunnel then stud, it was time to adopt it in haproxy. It's so fast that we don't use openssl's cache at all, since even at one single process, it's at least as fast. 7) other A lot remains to be done, mainly some of the aforementionned structres are still included in other ones, which simplified the split Once all the work is over, we should end up with less memory used per connection. This is important to better handle DDoS. At the moment, everything we could try seems to work fine. The SSL stacks well on top of the PROXY protocol, which is very important to build SSL offload farms (I'm sure Baptiste will want to write a blog article on the subject of using sub-$1000 machines to build large 100k+tps farms). Stats work over https too. Right now we're missing ACLs to match whether the traffic was SSL or clear, as well as logs. Both can be worked around by using distinct bind lines or even frontends. The doc is still clearly lacking, but we think that the config will change a little bit. Only the GNU makefile was updated, neither the BSD nor OSX were, they're a little trickier. If someone with one of these systems wants to update them, I'll happily accept the patches. What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL connections per second and 300 Mbps is what we got out of a dual-core Atom D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was enabled) :-) This is a bit more than stud and obviously much better than stunnel (which doesn't scale to more than a few hundred connections before the performance quickly drops). And older tests seem to indicate that with YaSSL we can get 30-40% more, maybe even more. We need to work with the YaSSL guys to slightly improve their cache management before this can become a default build option. Enough speaking, for those who want to test or even have the hardware to run more interesting benchmarks, the code was merged into the master branch and is in today's snapshot (20120904) here : http://haproxy.1wt.eu/download/1.5/src/snapshot/ Build it by passing USE_OPENSSL=1 on the make command line. You
