Re: SSL Performance increase?
On 2/6/2015 5:48 AM, Dennis Jacobfeuerborn wrote: Has somebody ever posted a working example configuration for haproxy that applies the improvements mentioned in the video? I tried to implement these recommendations but didn't seem to get results I was expecting. How exactly does one reliably test that the 1-RTT handshake is actually working? Running on Ubuntu 14, I have used a bind configuration like this: bind W.X.Y.Z:443 ssl crt testcert.pem npn http/1.1 The system includes an hourly cronjob that grabs the ocsp response, which haproxy (or maybe it's openssl) automatically staples into the certificate sent to the client. This config reduced the SSL negotiation time greatly when compared to haproxy on a CentOS 6 install, which uses a very old openssl version and cannot do NPN. I do not remember whether it was 1-RTT, but I don't think it was. I have these options in global: ssl-default-bind-ciphers ALL:!DH:!ADH:!EXP:!LOW:!RC2:!3DES:!SEED:!RC4:+HIGH:+MEDIUM ssl-default-server-ciphers RC4-MD5 Thanks, Shawn
Re: SSL Performance increase?
Shawn Heisey wrote on 02/06/2015 07:38 AM: [CUT] The current haproxy version implements almost every performance-enhancing method mentioned in that video, as long as your openssl is new enough. It's on CentOS 6.. is that new enough? We'll start testing different scenarios and see if we find the reason for the problem. Thank you for the video. -- Regards, Klavs Klavsen, GSEC - k...@vsen.dk - http://www.vsen.dk - Tlf. 61281200 Those who do not understand Unix are condemned to reinvent it, poorly. --Henry Spencer
Re: SSL Performance increase?
On 06.02.2015 18:50, Dennis Jacobfeuerborn wrote: On 06.02.2015 14:13, Lukas Tribus wrote: I tried to implement these recommendations but didn't seem to get results I was expecting. How exactly does one reliably test that the 1-RTT handshake is actually working? Enable TFO and announce http/1.1 via NPN and ALPN, that should do it. But your client will have to support all those features as-well (for example TFO can't possibly work in Windows). You will have to capture the TLS handshake in wireshark to see how fast it was (in terms of time and RTT's). This is really what I'm trying to get at. What is a specific way to test this? Which clients do support a 1-RTT handshake and what would a Wireshark session look like where the 1-RTT handshake succeeds compared to one which doesn't. There is a lot of information about this on the internet. All of it extremely vague. Surely there must be a way to come up with a test scenario that can verify such a setup more deterministically? Case in point: In the attached capture it looks like tls false start is working yet I actually haven't activated npn/alpn on the haproxy side which means tls false start should fail, no? Regards, Dennis No. Time SourceDestination Protocol Length Info 4 0.00021900010.99.0.1 10.99.0.202 TLSv1.2 583 Client Hello 5 0.0004910.99.0.202 10.99.0.1 TLSv1.2 227 Server Hello, Change Cipher Spec, Encrypted Handshake Message 7 0.00150300010.99.0.1 10.99.0.202 TLSv1.2 141 Change Cipher Spec, Encrypted Handshake Message 8 0.00159400010.99.0.1 10.99.0.202 TLSv1.2 727 Application Data 10 0.00231700010.99.0.202 10.99.0.1 TLSv1.2 231 Application Data
Re: SSL Performance increase?
On 06.02.2015 07:38, Shawn Heisey wrote: On 2/5/2015 5:54 AM, Klavs Klavsen wrote: Adding nbproc 4, improved performance of https from 511 req/s to 1296 req/s.. not quite an exponential scaling.. We tested with 8 cores and got 1328 req/s.. so it seems we're hitting something else already after 2,5 core.. vmstat 1 - also reveals a lot of cpu-idle time.. For cleartext performance I really don't know for sure what you can do, except maybe using bare metal rather than a virtual machine. Other people have been around this community a lot longer than I have and may have better ideas. Getting that cleartext performance up to a reasonable level will be your first step. Once that's done, there are a lot of things that will help with performance using SSL. This is an AWESOME video on that subject: https://www.youtube.com/watch?v=0EB7zh_7UE4 The current haproxy version implements almost every performance-enhancing method mentioned in that video, as long as your openssl is new enough. Has somebody ever posted a working example configuration for haproxy that applies the improvements mentioned in the video? I tried to implement these recommendations but didn't seem to get results I was expecting. How exactly does one reliably test that the 1-RTT handshake is actually working? Regards, Dennis
RE: SSL Performance increase?
I tried to implement these recommendations but didn't seem to get results I was expecting. How exactly does one reliably test that the 1-RTT handshake is actually working? Enable TFO and announce http/1.1 via NPN and ALPN, that should do it. But your client will have to support all those features as-well (for example TFO can't possibly work in Windows). You will have to capture the TLS handshake in wireshark to see how fast it was (in terms of time and RTT's). Lukas
Re: SSL Performance increase?
On 06.02.2015 14:13, Lukas Tribus wrote: I tried to implement these recommendations but didn't seem to get results I was expecting. How exactly does one reliably test that the 1-RTT handshake is actually working? Enable TFO and announce http/1.1 via NPN and ALPN, that should do it. But your client will have to support all those features as-well (for example TFO can't possibly work in Windows). You will have to capture the TLS handshake in wireshark to see how fast it was (in terms of time and RTT's). This is really what I'm trying to get at. What is a specific way to test this? Which clients do support a 1-RTT handshake and what would a Wireshark session look like where the 1-RTT handshake succeeds compared to one which doesn't. There is a lot of information about this on the internet. All of it extremely vague. Surely there must be a way to come up with a test scenario that can verify such a setup more deterministically? Regards, Dennis
Re: SSL Performance increase?
On 2/5/2015 5:54 AM, Klavs Klavsen wrote: Adding nbproc 4, improved performance of https from 511 req/s to 1296 req/s.. not quite an exponential scaling.. We tested with 8 cores and got 1328 req/s.. so it seems we're hitting something else already after 2,5 core.. vmstat 1 - also reveals a lot of cpu-idle time.. For cleartext performance I really don't know for sure what you can do, except maybe using bare metal rather than a virtual machine. Other people have been around this community a lot longer than I have and may have better ideas. Getting that cleartext performance up to a reasonable level will be your first step. Once that's done, there are a lot of things that will help with performance using SSL. This is an AWESOME video on that subject: https://www.youtube.com/watch?v=0EB7zh_7UE4 The current haproxy version implements almost every performance-enhancing method mentioned in that video, as long as your openssl is new enough. Thanks, Shawn
Re: SSL Performance increase?
Hi, On Thu, Feb 05, Klavs Klavsen wrote: Jarno Huuskonen wrote on 02/05/2015 01:28 PM: Hi, On Thu, Feb 05, Klavs Klavsen wrote: Hi guys, I'm testing our haproxy setup in regards to SSL performance - by simply using ab, and fetching a favicon.ico file.. over http haproxy delivers 3.000 req/s. over https haproxy delivers 511 req/s. I tried giving haproxy more cores (it's a virtual server) -but this did not help at all :( Silly question: when you added more vCPUs to the virtual machine did you change haproxy nbproc setting (make haproxy use more than 1process) ? Not silly at all.. I had naive figured it would scale on cpu's per default :) Adding daemon and nbproc setting = number-of-cores and testing again. I tried also setting cpu-map - but apperaently that needs to be specificly enabled at build time.. Is that an unsafe feature? Also - in regards to stats, I can understand that the stats will no longer be accurate? You can bind different procs to use different stats sockets. Something like: stats socket /path/to/stats level admin process 1 stats socket /path/to/stats2 level admin process 2 stats socket /path/to/stats3 level admin process 3 stats socket /path/to/stats4 level admin process 4 We currently fetch stats and insert in out graphite system.. which will be useless then.. :( You could collect stats from all processes and aggregate results. Or have dedicated processes for https that send traffic to http frontend (something like): listen HTTPS_in bind-process 2 3 4 mode tcp # (or http if you want http acls here) bind address:443 crt ... server http_in 127.0.0.1:666 send-proxy-v2 frontend HTTP_in bind-process 1 bind 127.0.0.1:666 accept-proxy bind *:80 ... mode http ... default_backend ... And then you can collect stats from socket thats bound to process 1. I think you can find better examples from list archives. (Or think this describes similar setup: http://brokenhaze.com/blog/2014/03/25/how-stack-exchange-gets-the-most-out-of-haproxy/) -Jarno Is it in the works, for haproxy to perhaps used some shared-memory section, with a semaphore locking ofcourse, to collect stats for all in one ? -- Jarno Huuskonen
Re: SSL Performance increase?
Adding nbproc 4, improved performance of https from 511 req/s to 1296 req/s.. not quite an exponential scaling.. We tested with 8 cores and got 1328 req/s.. so it seems we're hitting something else already after 2,5 core.. vmstat 1 - also reveals a lot of cpu-idle time.. Jarno Huuskonen wrote on 02/05/2015 01:28 PM: Hi, On Thu, Feb 05, Klavs Klavsen wrote: Hi guys, I'm testing our haproxy setup in regards to SSL performance - by simply using ab, and fetching a favicon.ico file.. over http haproxy delivers 3.000 req/s. over https haproxy delivers 511 req/s. I tried giving haproxy more cores (it's a virtual server) -but this did not help at all :( Silly question: when you added more vCPUs to the virtual machine did you change haproxy nbproc setting (make haproxy use more than 1process) ? -Jarno I can't seem to find anything on haproxy actually hitting a bottleneck.. I did try to do the https test from 2 clients simultaneously- and then they just get half the req/s - so total is the same. What should I look at, to improve https performance in haproxy? -- Regards, Klavs Klavsen, GSEC - k...@vsen.dk - http://www.vsen.dk - Tlf. 61281200 Those who do not understand Unix are condemned to reinvent it, poorly. --Henry Spencer
Re: SSL Performance increase?
Jarno Huuskonen wrote on 02/05/2015 01:28 PM: Hi, On Thu, Feb 05, Klavs Klavsen wrote: Hi guys, I'm testing our haproxy setup in regards to SSL performance - by simply using ab, and fetching a favicon.ico file.. over http haproxy delivers 3.000 req/s. over https haproxy delivers 511 req/s. I tried giving haproxy more cores (it's a virtual server) -but this did not help at all :( Silly question: when you added more vCPUs to the virtual machine did you change haproxy nbproc setting (make haproxy use more than 1process) ? Not silly at all.. I had naive figured it would scale on cpu's per default :) Adding daemon and nbproc setting = number-of-cores and testing again. I tried also setting cpu-map - but apperaently that needs to be specificly enabled at build time.. Is that an unsafe feature? Also - in regards to stats, I can understand that the stats will no longer be accurate? We currently fetch stats and insert in out graphite system.. which will be useless then.. :( Is it in the works, for haproxy to perhaps used some shared-memory section, with a semaphore locking ofcourse, to collect stats for all in one ? -- Regards, Klavs Klavsen, GSEC - k...@vsen.dk - http://www.vsen.dk - Tlf. 61281200 Those who do not understand Unix are condemned to reinvent it, poorly. --Henry Spencer
Re: SSL Performance increase?
Baptiste wrote on 02/05/2015 04:44 PM: [CUT] 3000 req/s in clear is low and a so rounded number is not normal :) Move (far far) away from this provider. You're wasting your time investigating perfomance issue while the limitation is in the hypervisor and multitenancy of your supplier. it's running on vmware 5.5 on local hardware - nowhere else to go :( If I set haproxy to just send a 301 response (ie. not relay to varnish delivering the favicon.ico) - I get approx 15k req/s.. -- Regards, Klavs Klavsen, GSEC - k...@vsen.dk - http://www.vsen.dk - Tlf. 61281200 Those who do not understand Unix are condemned to reinvent it, poorly. --Henry Spencer
Re: SSL Performance increase?
On Thu, Feb 5, 2015 at 2:03 PM, Jarno Huuskonen jarno.huusko...@uef.fi wrote: Hi, On Thu, Feb 05, Klavs Klavsen wrote: Jarno Huuskonen wrote on 02/05/2015 01:28 PM: Hi, On Thu, Feb 05, Klavs Klavsen wrote: Hi guys, I'm testing our haproxy setup in regards to SSL performance - by simply using ab, and fetching a favicon.ico file.. over http haproxy delivers 3.000 req/s. over https haproxy delivers 511 req/s. Hi, 3000 req/s in clear is low and a so rounded number is not normal :) Move (far far) away from this provider. You're wasting your time investigating perfomance issue while the limitation is in the hypervisor and multitenancy of your supplier. Baptiste
Re: SSL Performance increase?
On Thu, Feb 5, 2015 at 4:54 PM, Klavs Klavsen k...@vsen.dk wrote: Baptiste wrote on 02/05/2015 04:44 PM: [CUT] 3000 req/s in clear is low and a so rounded number is not normal :) Move (far far) away from this provider. You're wasting your time investigating perfomance issue while the limitation is in the hypervisor and multitenancy of your supplier. it's running on vmware 5.5 on local hardware - nowhere else to go :( If I set haproxy to just send a 301 response (ie. not relay to varnish delivering the favicon.ico) - I get approx 15k req/s.. this is very low We can get more than 50K conn/s in our VMWare lab using our HAProxy based ALOHA appliance. you must have an issue somwhere. Baptiste
Re: SSL Performance increase?
On 05.02.2015 20:09, Baptiste wrote: On Thu, Feb 5, 2015 at 4:54 PM, Klavs Klavsen k...@vsen.dk wrote: Baptiste wrote on 02/05/2015 04:44 PM: [CUT] 3000 req/s in clear is low and a so rounded number is not normal :) Move (far far) away from this provider. You're wasting your time investigating perfomance issue while the limitation is in the hypervisor and multitenancy of your supplier. it's running on vmware 5.5 on local hardware - nowhere else to go :( If I set haproxy to just send a 301 response (ie. not relay to varnish delivering the favicon.ico) - I get approx 15k req/s.. this is very low We can get more than 50K conn/s in our VMWare lab using our HAProxy based ALOHA appliance. you must have an issue somwhere. Are there any best practices on how to test haproxy in this regard? What are the recommended tools and settings to make a realistic conn/s test? Regards, Dennis
Re: SSL Performance increase?
Hi, On Thu, Feb 05, Klavs Klavsen wrote: Hi guys, I'm testing our haproxy setup in regards to SSL performance - by simply using ab, and fetching a favicon.ico file.. over http haproxy delivers 3.000 req/s. over https haproxy delivers 511 req/s. I tried giving haproxy more cores (it's a virtual server) -but this did not help at all :( Silly question: when you added more vCPUs to the virtual machine did you change haproxy nbproc setting (make haproxy use more than 1process) ? -Jarno I can't seem to find anything on haproxy actually hitting a bottleneck.. I did try to do the https test from 2 clients simultaneously- and then they just get half the req/s - so total is the same. What should I look at, to improve https performance in haproxy? -- Jarno Huuskonen
