Hello Thomas, On Tue, Feb 10, 2026 at 01:11:38PM +0100, Thomas Bätzler wrote: > Hello everybody, > > Hopefully somebody can point me in the right direction. I've got a use case > where I run multiple instances of haproxy in separate "ip netns" namespaces > that are plumbed using macvlan virtual interfaces to the physical interfaces > of a server. > > I'm using a systemd template unit to start each instance, i.e. > > ---(cut)--- > [Unit] > Description=HAProxy Load Balancer (instance %i) > Documentation=man:haproxy(1) > Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz > After=network-online.target rsyslog.service > Wants=network-online.target > > [Service] > EnvironmentFile=-/etc/default/haproxy > EnvironmentFile=-/etc/sysconfig/haproxy > BindReadOnlyPaths=/dev/log:/var/lib/haproxy/dev/log > Environment="CONFIG=/etc/haproxy/haproxy@%i.cfg" > "PIDFILE=/run/haproxy@%i.pid" "EXTRAOPTS=-S /run/haproxy-master@%i.sock" > ExecStart=/usr/sbin/ip netns exec %i /usr/sbin/haproxy -Ws -f $CONFIG -p > $PIDFILE $EXTRAOPTS > ExecReload=/usr/sbin/ip netns exec %i /usr/sbin/haproxy -Ws -f $CONFIG -c > $EXTRAOPTS > ExecReload=/bin/kill -USR2 $MAINPID > KillMode=mixed > Restart=always > SuccessExitStatus=143 > Type=notify > > [Install] > WantedBy=multi-user.target > ---(cut)--- > > > This has been working fine on Debian 11, but not at all on Debian 13 - the > haproxy dies immediately on startup: > > ---(cut)--- > Feb 09 21:37:09 test systemd[1]: Starting [email protected] - HAProxy > Load Balancer (instance test-ns)... > Feb 09 21:37:09 test ip[5429]: [NOTICE] (5429) : New worker (5431) forked > Feb 09 21:37:09 test ip[5429]: [NOTICE] (5429) : Loading success. > Feb 09 21:37:09 test systemd[1]: Started [email protected] - HAProxy > Load Balancer (instance test-ns). > Feb 09 21:37:09 test systemd[1]: [email protected]: Main process > exited, code=killed, status=11/SEGV > Feb 09 21:37:09 test systemd[1]: [email protected]: Killing process > 5431 (haproxy) with signal SIGKILL. > Feb 09 21:37:09 test systemd[1]: [email protected]: Killing process > 5432 (haproxy) with signal SIGKILL. > Feb 09 21:37:09 test kernel: haproxy[5433]: segfault at 28f8 ip > 00007f6763573d33 sp 00007f6762bf25e0 error 4 in > libc.so.6[96d33,7f6763505000+164000] likely on CPU 9 (core 12, socket 0) > Feb 09 21:37:09 test kernel: Code: 1f 84 00 00 00 00 00 48 89 f2 31 f6 e9 36 > f6 ff ff 66 0f 1f 44 00 00 41 56 41 55 41 54 55 53 48 89 fb 64 8b 2c 25 d0 > 02 00 00 <8b> 47 10 25 7f 01 00 00 83 f8 43 0f 87 ac 00 00 00 48 8d 1> > Feb 09 21:37:09 test systemd[1]: [email protected]: Failed with result > 'signal'. > ---(cut)--- > > When I try the ExecStart command line in a plain shell, the haproxy starts > just fine. > > I've attached a coredump and the haproxy config that I'm using in the hopes > that somebody can help me. > > Thank you very much in advance! > > Thomas
> root@test:~# gdb haproxy > /var/lib/coredumps/core-haproxy-sig11-user0-group0-pid5502-time1770669534 > GNU gdb (Debian 16.3-1) 16.3 > Copyright (C) 2024 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "x86_64-linux-gnu". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > <https://www.gnu.org/software/gdb/bugs/>. > Find the GDB manual and other documentation resources online at: > <http://www.gnu.org/software/gdb/documentation/>. > > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from haproxy... > (No debugging symbols found in haproxy) > [New LWP 5506] > [New LWP 5502] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Core was generated by `/usr/sbin/haproxy -Ws -f > /etc/haproxy/[email protected] -p /run/[email protected] -de -m 16'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 ___pthread_mutex_trylock (mutex=0x28e8) at > ./nptl/pthread_mutex_trylock.c:33 > > warning: 33 ./nptl/pthread_mutex_trylock.c: No such file or directory > [Current thread is 1 (Thread 0x7fdbf89ff6c0 (LWP 5506))] > (gdb) > (gdb) t a a bt full > > Thread 2 (Thread 0x7fdbf9c84140 (LWP 5502)): > #0 __syscall_cancel_arch () at > ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 > No locals. > #1 0x00007fdbf92bd668 in __internal_syscall_cancel (a1=<optimized out>, > a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, > a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 > result = <optimized out> > pd = <optimized out> > ch = <optimized out> > #2 0x00007fdbf92bdc8c in __futex_abstimed_wait_common64 (private=128, > futex_word=0x7fdbf89ff990, expected=<optimized out>, op=<optimized out>, > abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 > No locals. > #3 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7fdbf89ff990, > expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x0, > private=private@entry=128, cancel=cancel@entry=true) at > ./nptl/futex-internal.c:87 > err = <optimized out> > clockbit = 256 > op = <optimized out> > #4 0x00007fdbf92bdceb in __GI___futex_abstimed_wait_cancelable64 > (futex_word=futex_word@entry=0x7fdbf89ff990, expected=<optimized out>, > clockid=clockid@entry=0, abstime=abstime@entry=0x0, > private=private@entry=128) at ./nptl/futex-internal.c:139 > No locals. > #5 0x00007fdbf92c2774 in __pthread_clockjoin_ex (threadid=140582745798336, > thread_return=0x0, clockid=0, abstime=0x0, block=<optimized out>) at > ./nptl/pthread_join_common.c:108 > ret = <optimized out> > _buffer = {__routine = 0x7fdbf92c25f0 <cleanup>, __arg = > 0x7fdbf89ffce0, __canceltype = -2117861734, __prev = 0x0} > tid = <optimized out> > pd = 0x7fdbf89ff6c0 > self = <optimized out> > result = 0 > pd_result = <optimized out> > #6 0x0000560d81c8dc06 in ?? () > No symbol table info available. > #7 0x0000560d81a07e76 in main () > No symbol table info available. > > Thread 1 (Thread 0x7fdbf89ff6c0 (LWP 5506)): > #0 ___pthread_mutex_trylock (mutex=0x28e8) at > ./nptl/pthread_mutex_trylock.c:33 > oldval = <optimized out> > id = 5506 > again = <optimized out> > __PRETTY_FUNCTION__ = "___pthread_mutex_trylock" > #1 0x00007fdbf94cf375 in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2 > No symbol table info available. > #2 0x00007fdbf94d046f in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2 > No symbol table info available. > #3 0x00007fdbf94d06f0 in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2 > No symbol table info available. > #4 0x00007fdbf94d236b in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2 > No symbol table info available. > #5 0x00007fdbf9445842 in ?? () from /lib/x86_64-linux-gnu/libjemalloc.so.2 > No symbol table info available. > #6 0x00007fdbf9e96d06 in malloc (size=56) at ../include/rtld-malloc.h:56 > No locals. > #7 _dl_map_object_deps (map=map@entry=0x7fdbf9cbeb30, > preloads=preloads@entry=0x0, npreloads=npreloads@entry=0, > trace_mode=trace_mode@entry=0, open_mode=open_mode@entry=-2147483648) at > ./elf/dl-deps.c:463 > known = <optimized out> > runp = <optimized out> > tail = <optimized out> > nlist = 3 > i = <optimized out> > name = <optimized out> > errno_saved = <optimized out> > errno_reason = <optimized out> > exception = {objname = 0x0, errstring = 0x0, message_buffer = 0x0} > needed_space = {data = 0x7fdbf89f2a80, length = 1024, __space = > {__align = {__max_align_ll = 0, __max_align_ld = <invalid float value>}, __c > = > "\000\000\000\000\000\000\000\000T\357\351\371\333\177\000\000\000\000\000\000\000\000\000\000%~=\371\333\177\000\000\370:\351\371\333\177\000\000t\304\351\371\333\177\000\000\000\000\000\000\000\000\000\000\002\000\000\220\002\000\000\000\220\n\316\371\333\177", > '\000' <repeats 706 times>...}} > __PRETTY_FUNCTION__ = "_dl_map_object_deps" > old_l_initfini = 0x0 > l_initfini = <optimized out> > map_index = <optimized out> > l_reldeps = <optimized out> > #8 0x00007fdbf9ea0087 in dl_open_worker_begin (a=0x7fdbf89f31e0) at > ./elf/dl-open.c:606 > args = 0x7fdbf89f31e0 > file = <optimized out> > mode = -2147483646 > call_map = <optimized out> > dst = <optimized out> > new = 0x7fdbf9cbeb30 > __PRETTY_FUNCTION__ = "dl_open_worker_begin" > reloc_mode = <optimized out> > first = <optimized out> > last = <optimized out> > j = <optimized out> > l = <optimized out> > relocation_in_progress = <optimized out> > r = <optimized out> > any_tls = <optimized out> > #9 0x00007fdbf9e95399 in __GI__dl_catch_exception > (exception=exception@entry=0x7fdbf89f3040, > operate=operate@entry=0x7fdbf9e9ff80 <dl_open_worker_begin>, > args=args@entry=0x7fdbf89f31e0) at ./elf/dl-catch.c:241 > errcode = 0 > c = {exception = 0x7fdbf89f3040, errcode = 0x7fdbf89f2f4c, env = > {{__jmpbuf = {140582745747936, 6250777515939346415, 140734807454248, > 140582745747520, 140582755491865, 140582767598088, 6250780706392258543, > 6250777496286541807}, __mask_was_saved = 0, __saved_mask = {__val = {0 > <repeats 16 times>}}}}} > old = 0x7fdbf89f30c0 > #10 0x00007fdbf9e9f7e8 in dl_open_worker (a=a@entry=0x7fdbf89f31e0) at > ./elf/dl-open.c:761 > ex = {objname = 0x0, errstring = 0x0, message_buffer = 0x0} > err = <optimized out> > args = 0x7fdbf89f31e0 > mode = <optimized out> > new = <optimized out> > #11 0x00007fdbf9e95399 in __GI__dl_catch_exception > (exception=exception@entry=0x7fdbf89f31c0, > operate=operate@entry=0x7fdbf9e9f7b0 <dl_open_worker>, > args=args@entry=0x7fdbf89f31e0) at ./elf/dl-catch.c:241 > errcode = 0 > c = {exception = 0x7fdbf89f31c0, errcode = 0x7fdbf89f30bc, env = > {{__jmpbuf = {140582745747904, 6250777117444421615, 140734807454248, 9, > 140582755491865, 140582767598088, 6250780705664546799, 6250777496286541807}, > __mask_was_saved = 0, __saved_mask = {__val = {0 <repeats 16 times>}}}}} > old = 0x7fdbf89f32c0 > #12 0x00007fdbf9e9fbe8 in _dl_open (file=0x7fdbf93d7e25 "libgcc_s.so.1", > mode=<optimized out>, caller_dlopen=0x7fdbf933e019 > <__GI___libc_unwind_link_get+73>, nsid=<optimized out>, argc=9, > argv=0x7fff6034b228, env=0x7fff6034b278) at ./elf/dl-open.c:874 > args = {file = 0x7fdbf93d7e25 "libgcc_s.so.1", mode = -2147483646, > caller_dlopen = 0x7fdbf933e019 <__GI___libc_unwind_link_get+73>, map = > 0x7fdbf9cbeb30, nsid = 0, original_global_scope_pending_adds = 0, > libc_already_loaded = true, worker_continue = false, argc = 9, argv = > 0x7fff6034b228, env = 0x7fff6034b278} > exception = {objname = 0x0, errstring = 0x0, message_buffer = 0x0} > errcode = <optimized out> > r_state = <optimized out> > __PRETTY_FUNCTION__ = "_dl_open" > #13 0x00007fdbf938dccd in do_dlopen (ptr=ptr@entry=0x7fdbf89f3430) at > ./elf/dl-libc.c:95 > args = 0x7fdbf89f3430 > #14 0x00007fdbf9e95399 in __GI__dl_catch_exception > (exception=exception@entry=0x7fdbf89f33b0, operate=0x7fdbf938dc90 > <do_dlopen>, args=0x7fdbf89f3430) at ./elf/dl-catch.c:241 > errcode = 32731 > c = {exception = 0x7fdbf89f33b0, errcode = 0x7fdbf89f32bc, env = > {{__jmpbuf = {140582745748471, 6250780705549203439, 140582745748472, 0, > 140734807452512, 140582737408000, 6250780705731655663, 6250777496286541807}, > __mask_was_saved = 2, __saved_mask = {__val = {0 <repeats 16 times>}}}}} > old = 0x0 > #15 0x00007fdbf9e954bf in _dl_catch_error (objname=0x7fdbf89f33f8, > errstring=0x7fdbf89f3400, mallocedp=0x7fdbf89f33f7, operate=<optimized out>, > args=<optimized out>) at ./elf/dl-catch.c:260 > exception = {objname = 0x3000000020 <error: Cannot access memory at > address 0x3000000020>, errstring = 0x7fdbf89f3490 "\3204\237\370\333\177", > message_buffer = 0x7fdbf89f33d0 ""} > errorcode = <optimized out> > #16 0x00007fdbf938dc3f in dlerror_run (operate=operate@entry=0x7fdbf938dc90 > <do_dlopen>, args=args@entry=0x7fdbf89f3430) at ./elf/dl-libc.c:45 > objname = 0x7fff6034ac57 "\001\001\n1" > last_errstring = 0x0 > malloced = false > result = <optimized out> > #17 0x00007fdbf938de4f in __libc_dlopen_mode (name=name@entry=0x7fdbf93d7e25 > "libgcc_s.so.1", mode=mode@entry=-2147483646) at ./elf/dl-libc.c:162 > args = {name = 0x7fdbf93d7e25 "libgcc_s.so.1", mode = -2147483646, > caller_dlopen = 0x7fdbf933e019 <__GI___libc_unwind_link_get+73>, map = 0x0} > #18 0x00007fdbf933e019 in __GI___libc_unwind_link_get () at > ./misc/unwind-link.c:51 > local_libgcc_handle = <optimized out> > local = <optimized out> > __PRETTY_FUNCTION__ = "__libc_unwind_link_get" > #19 0x00007fdbf92c1d29 in __GI___pthread_exit (value=0x0) at > ./nptl/pthread_exit.c:28 > unwind_link = <optimized out> > #20 0x0000560d81c8dc4f in ?? () > No symbol table info available. > #21 0x00007fdbf92c0b7b in start_thread (arg=<optimized out>) at > ./nptl/pthread_create.c:448 > ret = <optimized out> > pd = <optimized out> > out = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {128, 6266535886278739984, > 32, 0, 140734807452512, 140582737408000, 6250780705545009135, > 6250776970899372015}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, > 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > #22 0x00007fdbf933e7b8 in __GI___clone3 () at > ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 > No locals. This trace is very strange, it doesn't even crash in haproxy code, it seems to crash in the dynamic linker while loading dynamic libraries, either libgcc_s or jemalloc, it's uncertain to me. It's not uncommon to face chicken-and-egg issues in program initialization in general since every step expects to be started once the other ones are ready, but it's not necessarily granted that a memory allocator is properly initialized at this point for example. IMHO at this point the config was not even processed yet, and you could equally use /dev/null as the config path and it would do the same. It might even be possible that it happens due to the link ordering, and that changing the order of -l on the LDFLAGS could fix it. I have no idea what role the namespaces can play in this by the way. I suspect that you could get out of that problem by just dropping jemalloc from your build options since it crashes there (it seems to be using a mutex on a not-yet allocated variable). But even if that would solve the problem, it wouldn't necessarily tell us what the exact cause is. Willy
