On Thu, May 22, 2014 at 8:55 AM, Willy Tarreau <[email protected]> wrote: > Hi John, > > On Thu, May 22, 2014 at 08:28:55AM +0200, John-Paul Bader wrote: >> Hey, >> >> over the last 10 days we tested how much traffic one server with >> haproxy-1.5dev25 could take and yesterday we've managed to saturate one >> machine to the point where the load was equal to the number of real >> cores in the machine and the listening queues began to fill up. > > OK. > >> We're running on a 12 core Intel Xeon E5-2620 v2 2.10GHz with 32GB of >> RAM and with two Intel i350 dual port network cards. One for public IP, >> the other one for the LAN. >> >> At the point when the listening queues filled up we had about 76000 open >> sockets, a connection rate of 4800/s and about 68Mbit/s of traffic. >> >> System and interrupt load was quite low all the time and the haproxy >> processes were at about 80% CPU >> >> Our traffic is half HTTPS, half HTTP - coming from all over the world >> with all kinds of latencies and with rather short lived but burst like >> sessions. >> >> I guess there are still optimizations possible and we'll try to find >> them but so far we're happy with the results. >> >> Maybe someone here can comment whether they think these are good number >> or if we should be able to push much more through one machine. > > When SSL is involved, it's very hard to say whether it's normal or not, > because the CPU load can vary ten-fold depending on the rekeying rate. > For example, if you're running with few visitors doing a lot of requests, > there are rarely new key renegotiation so the handshake are cheap. However > if you have many visitors doing request in a while, the renegotiation rate > will be much higher. I don't know if we can easily know whether a handshake > involved a key renegotiation, in which case we could maintain separate stats > for these. > > Given you're talking about 80% CPU for haproxy, I'm assuming you're talking > about user CPU then, which would indicate it's mostly rekeying that is > happening. > > For more normal connections, you can try to play with the various ciphers, > tls versions (eg: 1.1 is more expensive than 1.2), and maybe even try to > disable tickets to see if anything changes. > > I don't remember the exact numbers we got on our appliances, but I'm > having vague memories of something like 3000 keys/s/core on a core i7 > 3.3 GHz for 1024 bit keys, so if we consider we have 1000 keys/s per > gigahertz, it should be about 25k/s on your system. If you're running > in 2048 bits, then you can divide performance by about 4.5 and that > would give about what you're observing. > > Maybe users with high traffic SSL production servers can provide some > input here. > > Best regards, > Willy > >
Hi, It also depends on your L7 configuration. option http-close of force-close will consume more resources than keep-alive or server-close. Also, how have you configured your HAProxy from a process point of view, I mean: - how many processes - how many processes dedicated to SSL processing - have you enable 'tune.ssl.cachesize' (or have you tunned it or tried to increase its value?) etc... Can you run a 'show info' on your HAProxy socket and paste here the SSL related lines? Note to Willy and Emeric: the 'show info' don't report the number of sessions stored in the cache. It may be interesting to get it, if possible. Baptiste
