On 18 Nov 12:33, Maciej Zdeb wrote: > Hi again, > > So "# some headers manipulation, nothing different then on other clusters" > was the important factor in config. Under this comment I've hidden from you > one of our LUA scripts that is doing header manipulation like deleting all > headers from request when its name begins with "abc*". We're doing it on > all HAProxy servers, but only here it has such a big impact on the CPU, > because of huge RPS. > > If I understand correctly: > with nbproc = 20, lua interpreter worked on every process > with nbproc=1, nbthread=20, lua interpreter works on single process/thread > > I suspect that running lua on multiple threads is not a trivial task...
If you can share your lua script maybe we can see if this is doable more natively in haproxy > > > > > wt., 17 lis 2020 o 15:50 Maciej Zdeb <[email protected]> napisał(a): > > > Hi, > > > > We're in a process of migration from HAProxy[2.2.5] working on multiple > > processes to multiple threads. Additional motivation came from the > > announcement that the "nbproc" directive was marked as deprecated and will > > be killed in 2.5. > > > > Mostly the migration went smoothly but on one of our clusters the CPU > > usage went so high that we were forced to rollback to nbproc. There is > > nothing unusual in the config, but the traffic on this particular cluster > > is quite unusual. > > > > With nbproc set to 20 CPU idle drops at most to 70%, with nbthread = 20 > > after a couple of minutes at idle 50% it drops to 0%. HAProxy > > processes/threads are working on dedicated/isolated CPU cores. > > > > [image: image.png] > > > > I mentioned that traffic is quite unusual, because most of it are http > > requests with some payload in headers and very very small responses (like > > 200 OK). On multi-proc setup HAProxy handles about 20 to 30k of connections > > (on frontend and backend) and about 10-20k of http requests. Incoming > > traffic is just about 100-200Mbit/s and outgoing 40-100Mbit/s from frontend > > perspective. > > > > Did someone experience similar behavior of HAProxy? I'll try to collect > > more data and generate similar traffic with sample config to show a > > difference in performance between nbproc and nbthread. > > > > I'll greatly appreciate any hints on what I should focus. :) > > > > Current config is close to: > > frontend front > > mode http > > option http-keep-alive > > http-request add-header X-Forwarded-For %[src] > > > > # some headers manipulation, nothing different then on other clusters > > > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 1 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 2 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 3 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 4 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 5 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 6 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 7 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 8 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process 9 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 10 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 11 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 12 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 13 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 14 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 15 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 16 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 17 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 18 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 19 > > bind x.x.x.x:443 ssl crt /etc/cert/a.pem.pem alpn h2,http/1.1 process > > 20 > > default_backend back > > > > backend back > > option http-keep-alive > > mode http > > http-reuse always > > option httpchk GET /health HTTP/1.0\r\nHost:\ example.com > > http-check expect string OK > > > > server slot_0_checker 10.x.x.x:31180 check weight 54 > > server slot_1_checker 10.x.x.x:31146 check weight 33 > > server slot_2_checker 10.x.x.x:31313 check weight 55 > > server slot_3_checker 10.x.x.x:31281 check weight 33 disabled > > server slot_4_checker 10.x.x.x:31717 check weight 55 > > server slot_5_checker 10.x.x.x:31031 check weight 76 > > server slot_6_checker 10.x.x.x:31124 check weight 50 > > server slot_7_checker 10.x.x.x:31353 check weight 48 > > server slot_8_checker 10.x.x.x:31839 check weight 33 > > server slot_9_checker 10.x.x.x:31854 check weight 44 > > server slot_10_checker 10.x.x.x:31794 check weight 60 disabled > > server slot_11_checker 10.x.x.x:31561 check weight 56 > > server slot_12_checker 10.x.x.x:31814 check weight 57 > > server slot_13_checker 10.x.x.x:31535 check weight 44 disabled > > server slot_14_checker 10.x.x.x:31829 check weight 43 disabled > > server slot_15_checker 10.x.x.x:31655 check weight 40 disabled > > -- (o- Julien Pivotto //\ Open-Source Consultant V_/_ Inuits - https://www.inuits.eu
signature.asc
Description: PGP signature
