I have an issue which I've don't know how to go ahead with.
So first I will describe my setup.
We have a pair of haproxy v2.0.5 that only takes HTTP traffic and relays
HTTPS traffic down to a pair of haproxys v1.7.11 doing SSL termination.
Here is a more "graphical" layout:
Internet =HTTP+HTTPS=> haproxy (v2.0.5, no-TLS)
haproxy (v2.0.5, no-TLS) =HTTP=> backend applications
haproxy (v2.0.5, no-TLS) =TCP/HTTPS=> haproxy (v1.7.11, TLS)
haproxy (v1.7.11, TLS) =HTTP=> backend applications

So the problem here is that this setup breaks when we try to upgrade the
haproxy (v1.7.11, TLS) to v2.0.5.
What happens is that it simply masses up established connections(`ss -s`)
until the internet facing haproxy runs out of resources and uses all cores
is 100% utilizied, gets too many orphaned connections and OOM killer starts
killing pids. So in terms of hardware usage this is going from ~60% CPU
utilization to 100% on 18 cores(HT turned off), 2 GB RAM usage to 32GB+swap.

So we do zero config changes, upgrade haproxy to 2.0.x + restart haproxy
and like a minute or so then it runs out of resources.
Each haproxy (v2.0.5, no-TLS) have an request rate of 55-90K/s.
Each haproxy (v1.7.11, TLS) have an request rate of 15-20/s.
Each haproxy (v2.0.5, no-TLS) have a connection rate of 7-12K/s.
Each haproxy (v1.7.11, TLS) has a connection rate of 6-7K/s.

I have no clue why a zero config change upgrade would break this setup.
Anyone that can help me go forward with troubleshooting this or explain
what might cause it to mass up established connections?

Any help appreciated,

Reply via email to