Re: Theoretical limits for a HAProxy instance

2023-01-24 Thread Willy Tarreau
Hi Iago, On Tue, Jan 24, 2023 at 04:45:54PM +0100, Iago Alonso wrote: > We are happy to report that after downgrading to OpenSSL 1.1.1s (from > 3.0.7), our performance problems are solved, and now looks like > HAProxy scales linearly with the available resources. Excellent, thanks for this nice f

Re: Theoretical limits for a HAProxy instance

2023-01-24 Thread Iago Alonso
We are happy to report that after downgrading to OpenSSL 1.1.1s (from 3.0.7), our performance problems are solved, and now looks like HAProxy scales linearly with the available resources. For reference, in a synthetic load test with a request payload of 2k, and a 32-core server (128GB RAM) with 10

Re: Theoretical limits for a HAProxy instance

2022-12-21 Thread Iago Alonso
Interesting, so you have conntrack enabled. With 5M conns, there's no reason to fill your table. However, have you checked your servers' kernel logs to see if you find any "conntrack table full" message, that might be caused by too fast connection recycling ? We can't see any message mentioning c

Re: Theoretical limits for a HAProxy instance

2022-12-16 Thread Willy Tarreau
On Fri, Dec 16, 2022 at 05:42:50PM +0100, Iago Alonso wrote: > Hi, > > > Ah that's pretty useful :-) It's very likely dealing with the handshake. > > Could you please run "perf top" on this machine and list the 10 top-most > > lines ? I'm interested in seeing if you're saturating on crypto functio

Re: Theoretical limits for a HAProxy instance

2022-12-16 Thread Iago Alonso
Hi, > Ah that's pretty useful :-) It's very likely dealing with the handshake. > Could you please run "perf top" on this machine and list the 10 top-most > lines ? I'm interested in seeing if you're saturating on crypto functions > or locking functions (e.g. "native_queued_spin_lock_slowpath"), th

Re: Theoretical limits for a HAProxy instance

2022-12-14 Thread Willy Tarreau
On Wed, Dec 14, 2022 at 02:04:44PM +0100, Iago Alonso wrote: > Also, our connections are renewed constantly, with a ssl rate of about 3000 > per second, same as our connection rate. Very useful, that answers one of my previous questions. Then please check "perf top", it will be very instructive I

Re: Theoretical limits for a HAProxy instance

2022-12-14 Thread Willy Tarreau
On Wed, Dec 14, 2022 at 12:50:10PM +0100, Iago Alonso wrote: > Hi, > > We are not sure what element produces the errors, in the haproxy logs we > don't see them. Then they don't pass through haproxy nor do they come from haproxy. > What does it happen with the new connections when we hit the > l

Re: Theoretical limits for a HAProxy instance

2022-12-14 Thread Iago Alonso
Also, our connections are renewed constantly, with a ssl rate of about 3000 per second, same as our connection rate. On 14/12/22 12:50, Iago Alonso wrote: Hi, We are not sure what element produces the errors, in the haproxy logs we don't see them. What does it happen with the new connections

Re: Theoretical limits for a HAProxy instance

2022-12-14 Thread Iago Alonso
Hi, We are not sure what element produces the errors, in the haproxy logs we don't see them. What does it happen with the new connections when we hit the limit? What server resource should be affected by it, if any? We have our logs in `warning` level so we do not see the response time on th

Re: Theoretical limits for a HAProxy instance

2022-12-14 Thread Iago Alonso
us metrics. https://discourse.haproxy.org/t/theoretical-limits-for-a-haproxy-instance/8168 Custom kernel parameters: net.ipv4.ip_local_port_range = "12768    60999" net.nf_conntrack_max = 500 fs.nr_open = 500 HAProxy config: global     log /dev/

Re: Theoretical limits for a HAProxy instance

2022-12-13 Thread Willy Tarreau
Hi, On Tue, Dec 13, 2022 at 03:33:58PM +0100, Iago Alonso wrote: > Hi, > > We do hit our defined max ssl/conn rates, but given the spare > resources, we don't expect to suddenly return 5xx. What bothers me is that once this limit is reached there's no more connection accepted by haproxy so you s

Re: Theoretical limits for a HAProxy instance

2022-12-13 Thread Emerson Gomes
t this on discourse, and I got the suggestion to > post here. In said post, I've included screenshots of some of our > Prometheus metrics. > > https://discourse.haproxy.org/t/theoretical-limits-for-a-haproxy-instance/8168 > > Custom kernel parameters: > net.ipv4.ip_local_por

Re: Theoretical limits for a HAProxy instance

2022-12-13 Thread Iago Alonso
Hi, We do hit our defined max ssl/conn rates, but given the spare resources, we don't expect to suddenly return 5xx. Here's the output of `haproxy -vv` (I've also added it to the post on discourse): HAProxy version 2.6.6-274d1a4 2022/09/22 - https://haproxy.org/ Status: long-term supported branc

Re: Theoretical limits for a HAProxy instance

2022-12-12 Thread Jarno Huuskonen
Hi, On Mon, 2022-12-12 at 09:47 +0100, Iago Alonso wrote: > Can you share haproxy -vv output ? > HAProxy config: > global >     log /dev/log len 65535 local0 warning >     chroot /var/lib/haproxy >     stats socket /run/haproxy-admin.sock mode 660 level admin >     user haproxy >     group hapr

Theoretical limits for a HAProxy instance

2022-12-12 Thread Iago Alonso
Prometheus metrics. https://discourse.haproxy.org/t/theoretical-limits-for-a-haproxy-instance/8168 Custom kernel parameters: net.ipv4.ip_local_port_range = "1276860999" net.nf_conntrack_max = 500 fs.nr_open = 500 HAProxy config: global log /dev/log len 65535 local0 warning