Hello everybody,
Hopefully somebody can point me in the right direction. I've got a use
case where I run multiple instances of haproxy in separate "ip netns"
namespaces that are plumbed using macvlan virtual interfaces to the
physical interfaces of a server.
I'm using a systemd template unit to start each instance, i.e.
---(cut)---
[Unit]
Description=HAProxy Load Balancer (instance %i)
Documentation=man:haproxy(1)
Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz
After=network-online.target rsyslog.service
Wants=network-online.target
[Service]
EnvironmentFile=-/etc/default/haproxy
EnvironmentFile=-/etc/sysconfig/haproxy
BindReadOnlyPaths=/dev/log:/var/lib/haproxy/dev/log
Environment="CONFIG=/etc/haproxy/haproxy@%i.cfg"
"PIDFILE=/run/haproxy@%i.pid" "EXTRAOPTS=-S /run/haproxy-master@%i.sock"
ExecStart=/usr/sbin/ip netns exec %i /usr/sbin/haproxy -Ws -f $CONFIG -p
$PIDFILE $EXTRAOPTS
ExecReload=/usr/sbin/ip netns exec %i /usr/sbin/haproxy -Ws -f $CONFIG
-c $EXTRAOPTS
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify
[Install]
WantedBy=multi-user.target
---(cut)---
This has been working fine on Debian 11, but not at all on Debian 13 -
the haproxy dies immediately on startup:
---(cut)---
Feb 09 21:37:09 test systemd[1]: Starting [email protected] -
HAProxy Load Balancer (instance test-ns)...
Feb 09 21:37:09 test ip[5429]: [NOTICE] (5429) : New worker (5431) forked
Feb 09 21:37:09 test ip[5429]: [NOTICE] (5429) : Loading success.
Feb 09 21:37:09 test systemd[1]: Started [email protected] -
HAProxy Load Balancer (instance test-ns).
Feb 09 21:37:09 test systemd[1]: [email protected]: Main process
exited, code=killed, status=11/SEGV
Feb 09 21:37:09 test systemd[1]: [email protected]: Killing
process 5431 (haproxy) with signal SIGKILL.
Feb 09 21:37:09 test systemd[1]: [email protected]: Killing
process 5432 (haproxy) with signal SIGKILL.
Feb 09 21:37:09 test kernel: haproxy[5433]: segfault at 28f8 ip
00007f6763573d33 sp 00007f6762bf25e0 error 4 in
libc.so.6[96d33,7f6763505000+164000] likely on CPU 9 (core 12, socket 0)
Feb 09 21:37:09 test kernel: Code: 1f 84 00 00 00 00 00 48 89 f2 31 f6
e9 36 f6 ff ff 66 0f 1f 44 00 00 41 56 41 55 41 54 55 53 48 89 fb 64 8b
2c 25 d0 02 00 00 <8b> 47 10 25 7f 01 00 00 83 f8 43 0f 87 ac 00 00 00
48 8d 1>
Feb 09 21:37:09 test systemd[1]: [email protected]: Failed with
result 'signal'.
---(cut)---
When I try the ExecStart command line in a plain shell, the haproxy
starts just fine.
I've attached a coredump and the haproxy config that I'm using in the
hopes that somebody can help me.
Thank you very much in advance!
Thomas
root@test:~# gdb haproxy
/var/lib/coredumps/core-haproxy-sig11-user0-group0-pid5502-time1770669534
GNU gdb (Debian 16.3-1) 16.3
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from haproxy...
(No debugging symbols found in haproxy)
[New LWP 5506]
[New LWP 5502]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/haproxy -Ws -f
/etc/haproxy/[email protected] -p /run/[email protected] -de -m 16'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 ___pthread_mutex_trylock (mutex=0x28e8) at ./nptl/pthread_mutex_trylock.c:33
warning: 33 ./nptl/pthread_mutex_trylock.c: No such file or directory
[Current thread is 1 (Thread 0x7fdbf89ff6c0 (LWP 5506))]
(gdb)
(gdb) t a a bt full
Thread 2 (Thread 0x7fdbf9c84140 (LWP 5502)):
#0 __syscall_cancel_arch () at
../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
No locals.
#1 0x00007fdbf92bd668 in __internal_syscall_cancel (a1=<optimized out>,
a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0,
a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49
result = <optimized out>
pd = <optimized out>
ch = <optimized out>
#2 0x00007fdbf92bdc8c in __futex_abstimed_wait_common64 (private=128,
futex_word=0x7fdbf89ff990, expected=<optimized out>, op=<optimized out>,
abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
No locals.
#3 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7fdbf89ff990,
expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x0,
private=private@entry=128, cancel=cancel@entry=true) at
./nptl/futex-internal.c:87
err = <optimized out>
clockbit = 256
op = <optimized out>
#4 0x00007fdbf92bdceb in __GI___futex_abstimed_wait_cancelable64
(futex_word=futex_word@entry=0x7fdbf89ff990, expected=<optimized out>,
clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128)
at ./nptl/futex-internal.c:139
No locals.
#5 0x00007fdbf92c2774 in __pthread_clockjoin_ex (threadid=140582745798336,
thread_return=0x0, clockid=0, abstime=0x0, block=<optimized out>) at
./nptl/pthread_join_common.c:108
ret = <optimized out>
_buffer = {__routine = 0x7fdbf92c25f0 <cleanup>, __arg =
0x7fdbf89ffce0, __canceltype = -2117861734, __prev = 0x0}
tid = <optimized out>
pd = 0x7fdbf89ff6c0
self = <optimized out>
result = 0
pd_result = <optimized out>
#6 0x0000560d81c8dc06 in ?? ()
No symbol table info available.
#7 0x0000560d81a07e76 in main ()
No symbol table info available.
Thread 1 (Thread 0x7fdbf89ff6c0 (LWP 5506)):
#0 ___pthread_mutex_trylock (mutex=0x28e8) at ./nptl/pthread_mutex_trylock.c:33
oldval = <optimized out>
id = 5506
again = <optimized out>
__PRETTY_FUNCTION__ = "___pthread_mutex_trylock"
#1 0x00007fdbf94cf375 in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2
No symbol table info available.
#2 0x00007fdbf94d046f in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2
No symbol table info available.
#3 0x00007fdbf94d06f0 in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2
No symbol table info available.
#4 0x00007fdbf94d236b in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2
No symbol table info available.
#5 0x00007fdbf9445842 in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2
No symbol table info available.
#6 0x00007fdbf9e96d06 in malloc (size=56) at ../include/rtld-malloc.h:56
No locals.
#7 _dl_map_object_deps (map=map@entry=0x7fdbf9cbeb30,
preloads=preloads@entry=0x0, npreloads=npreloads@entry=0,
trace_mode=trace_mode@entry=0, open_mode=open_mode@entry=-2147483648) at
./elf/dl-deps.c:463
known = <optimized out>
runp = <optimized out>
tail = <optimized out>
nlist = 3
i = <optimized out>
name = <optimized out>
errno_saved = <optimized out>
errno_reason = <optimized out>
exception = {objname = 0x0, errstring = 0x0, message_buffer = 0x0}
needed_space = {data = 0x7fdbf89f2a80, length = 1024, __space =
{__align = {__max_align_ll = 0, __max_align_ld = <invalid float value>}, __c =
"\000\000\000\000\000\000\000\000T\357\351\371\333\177\000\000\000\000\000\000\000\000\000\000%~=\371\333\177\000\000\370:\351\371\333\177\000\000t\304\351\371\333\177\000\000\000\000\000\000\000\000\000\000\002\000\000\220\002\000\000\000\220\n\316\371\333\177",
'\000' <repeats 706 times>...}}
__PRETTY_FUNCTION__ = "_dl_map_object_deps"
old_l_initfini = 0x0
l_initfini = <optimized out>
map_index = <optimized out>
l_reldeps = <optimized out>
#8 0x00007fdbf9ea0087 in dl_open_worker_begin (a=0x7fdbf89f31e0) at
./elf/dl-open.c:606
args = 0x7fdbf89f31e0
file = <optimized out>
mode = -2147483646
call_map = <optimized out>
dst = <optimized out>
new = 0x7fdbf9cbeb30
__PRETTY_FUNCTION__ = "dl_open_worker_begin"
reloc_mode = <optimized out>
first = <optimized out>
last = <optimized out>
j = <optimized out>
l = <optimized out>
relocation_in_progress = <optimized out>
r = <optimized out>
any_tls = <optimized out>
#9 0x00007fdbf9e95399 in __GI__dl_catch_exception
(exception=exception@entry=0x7fdbf89f3040, operate=operate@entry=0x7fdbf9e9ff80
<dl_open_worker_begin>, args=args@entry=0x7fdbf89f31e0) at ./elf/dl-catch.c:241
errcode = 0
c = {exception = 0x7fdbf89f3040, errcode = 0x7fdbf89f2f4c, env =
{{__jmpbuf = {140582745747936, 6250777515939346415, 140734807454248,
140582745747520, 140582755491865, 140582767598088, 6250780706392258543,
6250777496286541807}, __mask_was_saved = 0, __saved_mask = {__val = {0 <repeats
16 times>}}}}}
old = 0x7fdbf89f30c0
#10 0x00007fdbf9e9f7e8 in dl_open_worker (a=a@entry=0x7fdbf89f31e0) at
./elf/dl-open.c:761
ex = {objname = 0x0, errstring = 0x0, message_buffer = 0x0}
err = <optimized out>
args = 0x7fdbf89f31e0
mode = <optimized out>
new = <optimized out>
#11 0x00007fdbf9e95399 in __GI__dl_catch_exception
(exception=exception@entry=0x7fdbf89f31c0, operate=operate@entry=0x7fdbf9e9f7b0
<dl_open_worker>, args=args@entry=0x7fdbf89f31e0) at ./elf/dl-catch.c:241
errcode = 0
c = {exception = 0x7fdbf89f31c0, errcode = 0x7fdbf89f30bc, env =
{{__jmpbuf = {140582745747904, 6250777117444421615, 140734807454248, 9,
140582755491865, 140582767598088, 6250780705664546799, 6250777496286541807},
__mask_was_saved = 0, __saved_mask = {__val = {0 <repeats 16 times>}}}}}
old = 0x7fdbf89f32c0
#12 0x00007fdbf9e9fbe8 in _dl_open (file=0x7fdbf93d7e25 "libgcc_s.so.1",
mode=<optimized out>, caller_dlopen=0x7fdbf933e019
<__GI___libc_unwind_link_get+73>, nsid=<optimized out>, argc=9,
argv=0x7fff6034b228, env=0x7fff6034b278) at ./elf/dl-open.c:874
args = {file = 0x7fdbf93d7e25 "libgcc_s.so.1", mode = -2147483646,
caller_dlopen = 0x7fdbf933e019 <__GI___libc_unwind_link_get+73>, map =
0x7fdbf9cbeb30, nsid = 0, original_global_scope_pending_adds = 0,
libc_already_loaded = true, worker_continue = false, argc = 9, argv =
0x7fff6034b228, env = 0x7fff6034b278}
exception = {objname = 0x0, errstring = 0x0, message_buffer = 0x0}
errcode = <optimized out>
r_state = <optimized out>
__PRETTY_FUNCTION__ = "_dl_open"
#13 0x00007fdbf938dccd in do_dlopen (ptr=ptr@entry=0x7fdbf89f3430) at
./elf/dl-libc.c:95
args = 0x7fdbf89f3430
#14 0x00007fdbf9e95399 in __GI__dl_catch_exception
(exception=exception@entry=0x7fdbf89f33b0, operate=0x7fdbf938dc90 <do_dlopen>,
args=0x7fdbf89f3430) at ./elf/dl-catch.c:241
errcode = 32731
c = {exception = 0x7fdbf89f33b0, errcode = 0x7fdbf89f32bc, env =
{{__jmpbuf = {140582745748471, 6250780705549203439, 140582745748472, 0,
140734807452512, 140582737408000, 6250780705731655663, 6250777496286541807},
__mask_was_saved = 2, __saved_mask = {__val = {0 <repeats 16 times>}}}}}
old = 0x0
#15 0x00007fdbf9e954bf in _dl_catch_error (objname=0x7fdbf89f33f8,
errstring=0x7fdbf89f3400, mallocedp=0x7fdbf89f33f7, operate=<optimized out>,
args=<optimized out>) at ./elf/dl-catch.c:260
exception = {objname = 0x3000000020 <error: Cannot access memory at
address 0x3000000020>, errstring = 0x7fdbf89f3490 "\3204\237\370\333\177",
message_buffer = 0x7fdbf89f33d0 ""}
errorcode = <optimized out>
#16 0x00007fdbf938dc3f in dlerror_run (operate=operate@entry=0x7fdbf938dc90
<do_dlopen>, args=args@entry=0x7fdbf89f3430) at ./elf/dl-libc.c:45
objname = 0x7fff6034ac57 "\001\001\n1"
last_errstring = 0x0
malloced = false
result = <optimized out>
#17 0x00007fdbf938de4f in __libc_dlopen_mode (name=name@entry=0x7fdbf93d7e25
"libgcc_s.so.1", mode=mode@entry=-2147483646) at ./elf/dl-libc.c:162
args = {name = 0x7fdbf93d7e25 "libgcc_s.so.1", mode = -2147483646,
caller_dlopen = 0x7fdbf933e019 <__GI___libc_unwind_link_get+73>, map = 0x0}
#18 0x00007fdbf933e019 in __GI___libc_unwind_link_get () at
./misc/unwind-link.c:51
local_libgcc_handle = <optimized out>
local = <optimized out>
__PRETTY_FUNCTION__ = "__libc_unwind_link_get"
#19 0x00007fdbf92c1d29 in __GI___pthread_exit (value=0x0) at
./nptl/pthread_exit.c:28
unwind_link = <optimized out>
#20 0x0000560d81c8dc4f in ?? ()
No symbol table info available.
#21 0x00007fdbf92c0b7b in start_thread (arg=<optimized out>) at
./nptl/pthread_create.c:448
ret = <optimized out>
pd = <optimized out>
out = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {128, 6266535886278739984,
32, 0, 140734807452512, 140582737408000, 6250780705545009135,
6250776970899372015}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0},
data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#22 0x00007fdbf933e7b8 in __GI___clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
No locals.
# The global section deals with process-wide settings (security, resource usage)
global
# all file names are relative to the directory containing this config
# file by default
default-path config
# refuse to start if any warning is emitted at boot (keep configs clean)
zero-warning
# Security hardening: isolate and drop privileges
chroot /var/lib/haproxy
user haproxy
group haproxy
# daemonize
daemon
# do not keep old processes longer than that after a reload
hard-stop-after 5m
ssl-default-bind-ciphers
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
tune.ssl.default-dh-param 2048
# default settings common to all HTTP proxies below
defaults http
mode http
option httplog
log global
timeout client 1m
timeout server 1m
timeout connect 10s
timeout http-keep-alive 2m
timeout queue 15s
timeout tunnel 4h # for websocket
listen stats
bind 10.10.10.1:9000
mode http
stats enable
stats realm Haproxy\ Statistics
stats uri /stats
frontend https-panel
bind 1.1.1.1:8443 ssl crt-list /etc/haproxy/certificates
mode http
http-response set-header Strict-Transport-Security "max-age=16000000;
includeSubDomains; preload;"
default_backend back-https-panel
frontend https
bind 1.1.1.1:443 ssl crt-list /etc/haproxy/certificates
mode http
default_backend back-https
backend back-https-panel
http-request add-header X-Forwarded-Proto https
mode http
option forwardfor
server panel1 10.10.10.19:8443 check ssl verify none
server panel2 10.10.10.35:8443 check ssl verify none
server panel3 10.10.10.51:8443 check ssl verify none
backend back-https
http-request add-header X-Forwarded-Proto https
mode http
timeout connect 20s
timeout server 10m
balance source
option forwardfor
server server1 10.10.10.20:80 check
server server2 10.10.10.36:80 check
server server3 10.10.10.52:80 check
