Sorry for spam. In the last message I said that the old process (after reload) is consuming cpu for lua processing and that's not true, it is processing other things also.
I'll take a break. ;) Then I'll verify if the issue exists on 2.3 and maybe 2.4 branch. For each version I need a week or two to be sure the issue does not occur. :( If 2.3 and 2.4 behaves the same way the 2.2 does, I'll try to confirm if there is any relation between infinite loops and custom configuration: - lua scripts (mainly used for header generation/manipulation), - spoe (used for sending metadata about each request to external service), - peers (we have a cluster of 12 HAProxy servers connected to each other). Kind regards, wt., 16 mar 2021 o 13:05 Maciej Zdeb <mac...@zdeb.pl> napisał(a): > Below is the output from perf top - it happens during reload (all threads > on an old process spike and use 100% cpu, processing lua) and after 15-30 > seconds old process exits. It is probably a different bug, because it > happens on an old process and I have no idea how it could affect the new > process. Maybe through peers protocol? Still both issues are related to lua > scripts (on my initial email in low_cpu_thread.txt it hung on lua script > responsible for removing http headers on hlua_http_get_headers function). > > Samples: 770K of event 'cycles:ppp', Event count (approx.): 336817908209 > Overhead Shared Object Symbol > 36.39% haproxy [.] > process_runnable_tasks > 34.82% haproxy [.] > run_tasks_from_lists > 10.04% haproxy [.] __task_wakeup > 5.63% libc-2.27.so [.] > __memset_avx2_erms > 1.87% haproxy [.] hlua_action > 0.79% haproxy [.] hlua_ctx_init > 0.77% haproxy [.] > stream_release_buffers > 0.58% libcrypto.so.1.1 [.] > rsaz_1024_sqr_avx2 > 0.56% haproxy [.] > stream_int_notify > 0.49% haproxy [.] > task_run_applet > 0.42% haproxy [.] > hlua_sample_fetch_wrapper > 0.38% haproxy [.] > spoe_handle_appctx > 0.38% [kernel] [k] do_syscall_64 > 0.37% haproxy [.] eb32sc_insert > 0.32% [vdso] [.] > __vdso_clock_gettime > 0.27% haproxy [.] > hlua_ctx_destroy > 0.22% haproxy [.] > stktable_get_entry > 0.20% haproxy [.] > process_stream > 0.19% [kernel] [k] > audit_filter_rules.constprop.13 > 0.17% haproxy [.] > si_applet_wake_cb > 0.14% libcrypto.so.1.1 [.] > rsaz_1024_mul_avx2 > > wt., 16 mar 2021 o 12:00 Maciej Zdeb <mac...@zdeb.pl> napisał(a): > >> Sure, patch from Christopher attached. :) >> >> wt., 16 mar 2021 o 10:58 Willy Tarreau <w...@1wt.eu> napisał(a): >> >>> Hi Maciej, >>> >>> On Tue, Mar 16, 2021 at 10:55:11AM +0100, Maciej Zdeb wrote: >>> > Hi, >>> > >>> > I'm returning with bad news, the patch did not help and the issue >>> occurred >>> > today (on patched 2.2.10). It is definitely related to reloads, >>> however it >>> > is very rare issue it worked flawlessly the whole week. >>> >>> OK. Just as a reminder (because I can't find it here), can you please >>> remind us what this patch was ? >>> >>> Thanks, >>> Willy >>> >>
