Re: Support for SSL
On Wed, Nov 17, 2010 at 09:46:05AM -0500, John Marrett wrote: Bedis, Cause using the cores to decrypt traffic would reduce drastically overall performance. Well, this is what we saw on our HTTP cache server (running CentOS) on 8 cores hardware: when enabling SSL, the performance were so bad that So we kept our old Nortel vpn 3050 to handle the SSL traffic. I'm astonished to hear that you had these kinds of issues on modern hardware. We stopped using dedicated SSL hardware quite some time ago. I'm not surprized at all. The issue generally lies in mixing high latency processing (eg: SSL) with low latency (eg: HTTP processing). When your CPUs are stuck for 200 microseconds processing an SSL connection, you can try to do whatever you want, all pending HTTP processing will be stuck that long, which will considerably limit the request rate. One of the solution sometimes is to dedicate some CPUs to slow processing and others to fast processing, but this is not always possible. Cheers, Willy
Re: Support for SSL
Here's an interesting blog post by a Google engineer about how they rolled out SSL for many of their services with very little additional CPU and network overhead. Specifically, he claims that On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead. http://www.imperialviolet.org/2010/06/25/overclocking-ssl.html -Bryan On Fri, Nov 19, 2010 at 4:54 AM, Willy Tarreau w...@1wt.eu wrote: On Wed, Nov 17, 2010 at 09:46:05AM -0500, John Marrett wrote: Bedis, Cause using the cores to decrypt traffic would reduce drastically overall performance. Well, this is what we saw on our HTTP cache server (running CentOS) on 8 cores hardware: when enabling SSL, the performance were so bad that So we kept our old Nortel vpn 3050 to handle the SSL traffic. I'm astonished to hear that you had these kinds of issues on modern hardware. We stopped using dedicated SSL hardware quite some time ago. I'm not surprized at all. The issue generally lies in mixing high latency processing (eg: SSL) with low latency (eg: HTTP processing). When your CPUs are stuck for 200 microseconds processing an SSL connection, you can try to do whatever you want, all pending HTTP processing will be stuck that long, which will considerably limit the request rate. One of the solution sometimes is to dedicate some CPUs to slow processing and others to fast processing, but this is not always possible. Cheers, Willy
RE: Support for SSL
Bedis, Cause using the cores to decrypt traffic would reduce drastically overall performance. Well, this is what we saw on our HTTP cache server (running CentOS) on 8 cores hardware: when enabling SSL, the performance were so bad that So we kept our old Nortel vpn 3050 to handle the SSL traffic. I'm astonished to hear that you had these kinds of issues on modern hardware. We stopped using dedicated SSL hardware quite some time ago. Of course, everyone's traffic is different. May I ask what volume of traffic (Connections / second, Megabits) you are dealing with that saturated an 8 core machine? we should have ordered more chassis and licences to handle the same traffic... leading to earn less money :) What web/ssl server were you using and what version of CentOS. The use of the word licenses is interesting :) Were you already high in CPU consumption without the SSL traffic on the machine? -JohnF
Re: Support for SSL
Hi John, Without entering too much in details, we have a mutualized reverse proxy cache platform in order to accelerate HTTP content (you can call it CDN ;) ) on which we use an HTTP reverse proxy caches coded by a third party company. The reverse proxy software run over a centos linux and has a cost (licence). The hardware is HP server (2U) with 2 quad cores, 32G of memory and a lot of hard drives. I can't remember the numbers exactly, but when we tested the SSL capacity inside the HTTP accelerator, we decreased a lot the overall performance (maybe because of the code) and to keep the same capacity (HTTP throughput) on our CDN we should have bought more servers and more licences. Without SSL enabled, we tested the HTTP accelerator with live traffic at more than 700MB/s, more than 20K HTTP Req/s and 80% of CPU... With only a small percentage of this traffic encrypted the performance decreased a lot, but I can't remember how much :/ Note that in a normal day, our caches run only at 10% of CPU and 100 Mb... Note that we did not saturated our servers with SSL, I'm just saying that to keep enough free capacity to absord customer's pike traffic, we should have bought more servers and licences and the cost would have been too much. It was cheaper to let our old vpn3050 in the racks doing the job :) Maybe it's related to the code of our supplier ;) Anyway, they were working on improving their SSL capacity by taking advantage of offloading the SSL computation to a daughter card into the chassis and keep on using the CPUs and memory to do HTTP acceleration, URL rewrite, ACLs, etc All the SMART stuff :) Note: the software is not Varnish ;) On Wed, Nov 17, 2010 at 3:46 PM, John Marrett jmarr...@mediagrif.com wrote: Bedis, Cause using the cores to decrypt traffic would reduce drastically overall performance. Well, this is what we saw on our HTTP cache server (running CentOS) on 8 cores hardware: when enabling SSL, the performance were so bad that So we kept our old Nortel vpn 3050 to handle the SSL traffic. I'm astonished to hear that you had these kinds of issues on modern hardware. We stopped using dedicated SSL hardware quite some time ago. Of course, everyone's traffic is different. May I ask what volume of traffic (Connections / second, Megabits) you are dealing with that saturated an 8 core machine? we should have ordered more chassis and licences to handle the same traffic... leading to earn less money :) What web/ssl server were you using and what version of CentOS. The use of the word licenses is interesting :) Were you already high in CPU consumption without the SSL traffic on the machine? -JohnF
RE: Support for SSL
Bedis, At that kind of connection volume (I assume that your 20k/s includes a certain quantity of keepalive, but a large volume of new connections as well) I'm not that surprised that you needed dedicated hardware. That said, I wouldn't expect the load to necessarily be that bad. I have little experience at this level of transactional volume. If you have doubts about your application you could test using an open source web server and see what kind of performance you're able to obtain on the SSL component on the same hardware. We handle SSL using an open source web server that acts as a reverse proxy sending the underlying http requests to haproxy which does the actual load balancing. Seperating the two in this way might let you avoid licensing costs, though it's probably unnecessary if the Nortel box is holding up well for you. Note: the software is not Varnish ;) Even if you pay for varnish support I doubt it's per node :) -JohnF
Re: Support for SSL
I wish I could use OpenSource solution. But my company refused so I had to follow their requirements (actually, the requirement was to use this specific software :D) and yes, our oldies do their job on SSL :) (If it works, don't fix it!!!) On Wed, Nov 17, 2010 at 5:05 PM, John Marrett jmarr...@mediagrif.com wrote: Bedis, At that kind of connection volume (I assume that your 20k/s includes a certain quantity of keepalive, but a large volume of new connections as well) I'm not that surprised that you needed dedicated hardware. That said, I wouldn't expect the load to necessarily be that bad. I have little experience at this level of transactional volume. If you have doubts about your application you could test using an open source web server and see what kind of performance you're able to obtain on the SSL component on the same hardware. We handle SSL using an open source web server that acts as a reverse proxy sending the underlying http requests to haproxy which does the actual load balancing. Seperating the two in this way might let you avoid licensing costs, though it's probably unnecessary if the Nortel box is holding up well for you. Note: the software is not Varnish ;) Even if you pay for varnish support I doubt it's per node :) -JohnF
Re: Support for SSL
You might take a look at one of these: http://www.caviumnetworks.com/processor_security_nitroxLite.htm They ship a modified OpenSSL stack to take advantage of the card. Cavium is what's inside most of the commercial load balancers...including I believe F5. -J Sent via iPhone Is your e-mail Premiere? On Nov 17, 2010, at 8:33, Bedis 9 bed...@gmail.com wrote: Hi John, Without entering too much in details, we have a mutualized reverse proxy cache platform in order to accelerate HTTP content (you can call it CDN ;) ) on which we use an HTTP reverse proxy caches coded by a third party company. The reverse proxy software run over a centos linux and has a cost (licence). The hardware is HP server (2U) with 2 quad cores, 32G of memory and a lot of hard drives. I can't remember the numbers exactly, but when we tested the SSL capacity inside the HTTP accelerator, we decreased a lot the overall performance (maybe because of the code) and to keep the same capacity (HTTP throughput) on our CDN we should have bought more servers and more licences. Without SSL enabled, we tested the HTTP accelerator with live traffic at more than 700MB/s, more than 20K HTTP Req/s and 80% of CPU... With only a small percentage of this traffic encrypted the performance decreased a lot, but I can't remember how much :/ Note that in a normal day, our caches run only at 10% of CPU and 100 Mb... Note that we did not saturated our servers with SSL, I'm just saying that to keep enough free capacity to absord customer's pike traffic, we should have bought more servers and licences and the cost would have been too much. It was cheaper to let our old vpn3050 in the racks doing the job :) Maybe it's related to the code of our supplier ;) Anyway, they were working on improving their SSL capacity by taking advantage of offloading the SSL computation to a daughter card into the chassis and keep on using the CPUs and memory to do HTTP acceleration, URL rewrite, ACLs, etc All the SMART stuff :) Note: the software is not Varnish ;) On Wed, Nov 17, 2010 at 3:46 PM, John Marrett jmarr...@mediagrif.com wrote: Bedis, Cause using the cores to decrypt traffic would reduce drastically overall performance. Well, this is what we saw on our HTTP cache server (running CentOS) on 8 cores hardware: when enabling SSL, the performance were so bad that So we kept our old Nortel vpn 3050 to handle the SSL traffic. I'm astonished to hear that you had these kinds of issues on modern hardware. We stopped using dedicated SSL hardware quite some time ago. Of course, everyone's traffic is different. May I ask what volume of traffic (Connections / second, Megabits) you are dealing with that saturated an 8 core machine? we should have ordered more chassis and licences to handle the same traffic... leading to earn less money :) What web/ssl server were you using and what version of CentOS. The use of the word licenses is interesting :) Were you already high in CPU consumption without the SSL traffic on the machine? -JohnF
Re: Support for SSL
Hello, On Sun, Nov 07, 2010 at 04:15:18PM +0100, Sebastien Estienne wrote: Hello, Is there any news about SSL support? Yes there are some news, we'll have to work on it at Exceliance. With current server's hardware having 8 cores or more, offering SSL is quite cheap. Hehe one thing at a time : haproxy right now only uses one core. Let's first have SSL and only then see how we can make use of more cores. The really difficult part is to try to use more cores without slowing down the overall processing due to very common synchronization. This implies massive changes to ensure that there's almost no shared data between processes or threads. Moreover with tools like firesheep getting widespread offering SSL to our users become an important feature Firesheep is doing nothing more than what has been done for decades with many other tools. The same people who believe their traffic cannot be sniffed by their coworker because they connect via a switch won't care about having their SSL session hijacked with an invalid certificate. I know that it's possible to use stunnel, but it would be better to have SSL support built in haproxy Yes indeed. At least stunnel already lets us assemble the bricks to build whatever we want, eventhough the configs are sometimes tough ! Regards, Willy
Re: Support for SSL
Le 16 nov. 2010 à 12:27, Willy Tarreau w...@1wt.eu a écrit : Hello, On Sun, Nov 07, 2010 at 04:15:18PM +0100, Sebastien Estienne wrote: Hello, Is there any news about SSL support? Yes there are some news, we'll have to work on it at Exceliance. this is great news, any early timeframe even fuzzy? With current server's hardware having 8 cores or more, offering SSL is quite cheap. Hehe one thing at a time : haproxy right now only uses one core. Let's first have SSL and only then see how we can make use of more cores. The really difficult part is to try to use more cores without slowing down the overall processing due to very common synchronization. This implies massive changes to ensure that there's almost no shared data between processes or threads. i thought haproxy could use more than one core with a prefork model like nginx? Moreover with tools like firesheep getting widespread offering SSL to our users become an important feature Firesheep is doing nothing more than what has been done for decades with many other tools. The same people who believe their traffic cannot be sniffed by their coworker because they connect via a switch won't care about having their SSL session hijacked with an invalid certificate. We all agree with this like irc and newsgroups existed before emule and biittorent :) But with easy tools like this we can t hide the problem anymore. (like adobe does with rtmpe) I know that it's possible to use stunnel, but it would be better to have SSL support built in haproxy Yes indeed. At least stunnel already lets us assemble the bricks to build whatever we want, eventhough the configs are sometimes tough ! Regards, Willy thanx.
Re: Support for SSL
On Tue, Nov 16, 2010 at 01:03:01PM +0100, Sebastien Estienne wrote: Le 16 nov. 2010 à 12:27, Willy Tarreau w...@1wt.eu a écrit : Hello, On Sun, Nov 07, 2010 at 04:15:18PM +0100, Sebastien Estienne wrote: Hello, Is there any news about SSL support? Yes there are some news, we'll have to work on it at Exceliance. this is great news, any early timeframe even fuzzy? 2011 if everything runs fine. With current server's hardware having 8 cores or more, offering SSL is quite cheap. Hehe one thing at a time : haproxy right now only uses one core. Let's first have SSL and only then see how we can make use of more cores. The really difficult part is to try to use more cores without slowing down the overall processing due to very common synchronization. This implies massive changes to ensure that there's almost no shared data between processes or threads. i thought haproxy could use more than one core with a prefork model like nginx? That's precisely the goal. But even with the prefork model, there are two ways to run : - processes which don't share anything. Haproxy already supports that but it makes sharing of stats, checks, stick tables, etc... very hard - threads which share everything : easier to share information, but requires locking everywhere. Some experimentations will have to be done anyway. Regards, Willy
Support for SSL
Hello, Is there any news about SSL support? With current server's hardware having 8 cores or more, offering SSL is quite cheap. Moreover with tools like firesheep getting widespread offering SSL to our users become an important feature I know that it's possible to use stunnel, but it would be better to have SSL support built in haproxy thanx, Sebastien Estienne
