Hi Elias,
On Thu, Dec 01, 2016 at 02:53:25PM +0100, Elias Abacioglu wrote:
> Hi,
>
> Similar to what Christian asked about a few days ago I would like help
> to summarize the recommendations for running a haproxy as a SSL LB on
> a multi cpu, multi core machine.
>
> I have a machine with two sockets equipped with Intel Xeon E5-2680 v4.
> 56 cores in total with HT enabled, 28 with HT disabled.
>
> Summary(correct me where I'm wrong):
> Hyperthreading - should be disabled
HT is useful with SSL, just like it performs well for most NIC drivers,
provided you never mix two different workloads on two threads of the
same core (typically you put to SSL processes on the same core, or two
network interrupts).
For regular L7 traffic, it doesn't bring anything and can even sometimes
have a slightly negative effect, so it's better not to use the second
thread of cores that are used for non-SSL traffic.
My recommendation here remains the same : just have dedicated SSL
offloading processes running on their own cores, and forward all this
traffic to the L7 running on a single process. This allows you to get
better stats, better checks, etc.
> As for network IRQ pinning, should I not pin the NIC to core 0 of each
> CPU? Should I split it up between the cores when doing SSL termination
> on most cores?
It's more complicated and it depends what workload you can achieve on
a single core. For my use cases, I tend to assign network IRQs on pairs
of threads and assign them "enough cores" so that I don't see ksoftirqd
pop up anymore. But there's no one-size-fits-all. For example on our
ALOHA appliances, we've enough of a single CPU core (2 threads) for the
kernel NICs to let our anti-ddos PacketShield deal with line-rate SYN
floods at 10 Gbps, or to forward 10 Gbps of traffic. On another machine
I tested two years ago on which we reached ~60 Gbps of L7 traffic, we
needed about 4 cores to reach the highest performance level but
unfortunately we had only 4 :-/ I tend to think that one rough rule of
thumb could be about 1 core per 10 Gbps of traffic.
(...)
> # (didn't see any cpu-map in Christians configuration, is it not
> needed anymore?)
It has never been "needed", use it if you want to pin specific processes
to specific CPUs (ie useful for SSL vs L7).
> frontend https
> bind :443 ssl crt /etc/ssl/xx.pem process 1
> ....
> bind :443 ssl crt /etc/ssl/xx.pem process 28
> # should I omitt process 1/2 and 15/16 here?
No reason for this.
> # Should I use core 0 on each CPU for backends (proc 1+15) or should I
> use core 1(proc 2+16)?
Backends are processed on the same CPU as the frontend which passes them
the traffic, so the bind-process has no effect there. In fact bind-process
on a backend means "at least on these processes".
That's why it's better to proceed like this (stupid numbers, just so that
you get the idea):
listen ssl-offload
bind-proess 2-50
bind :443 ssl .... process 2
...
bind :443 ssl .... process 50
server clear 127.0.0.1:1 send-proxy-v2
frontend clear
bind-process 1
bind 127.0.0.1:1 accept-proxy
...
> I just read the release notes for haproxy 1.7. There was mention of
> mixing different certificate types under "support of multi-certs".
>
> Previously I would have a tcp-ssl that would send traffic to a ecc
> backend and a rsa backend depending on req.ssl_ec_ext, which in turn
> would send traffic to a frontend with two sockets, one for each
> certificate type.
> To something like this (hope my ascii doesn't get messed up).
it was messed up, you need to have fixed size fonts for ascii art
and I suspect you were using a proportional font to read it in your
mailer :-)
> +-->backend01
> +--->backend-ecc+
> |
> frontend-tcp-ssl:443 |
> +--->frontend-https+-->backend02
> +--->backend-rsa +
> |
>
> +-->backend03
>
> Does this mean I can now mix RSA and ECC certificates on the same
> frontend and skip the entire frontend-backend-frontend trick?
Absolutely! That was the purpose.
Regards,
Willy
