--- Begin Message ---
Package: bareos-filedaemon
Version: 16.2.6-3
Severity: important
Dear Maintainer,
my backup system is running Debian-Stretch, but I also backup my
Debian-Sid development system. There bareos-fd crashes regularly:
> # ls -gGtr /var/lib/bareos
> insgesamt 5929744
> drwxrwxr-x 2 4096 Apr 22 2016 storage
> -rw-r----- 1 152255384 Okt 30 19:29 bareos-fd.core.1810
> -rw-r----- 1 713 Okt 30 19:29 bareos.1810.traceback
> -rw-r----- 1 183 Okt 30 19:29 scout-fd.1810.bactrace
> -rw-r----- 1 152255384 Okt 30 19:59 bareos-fd.core.3553
> -rw-r----- 1 713 Okt 30 19:59 bareos.3553.traceback
> -rw-r----- 1 183 Okt 30 19:59 scout-fd.3553.bactrace
> -rw-r----- 1 152255384 Okt 30 20:29 bareos-fd.core.3838
> -rw-r----- 1 711 Okt 30 20:29 bareos.3838.traceback
> -rw-r----- 1 183 Okt 30 20:29 scout-fd.3838.bactrace
> -rw-r----- 1 152255384 Okt 30 20:59 bareos-fd.core.4057
> -rw-r----- 1 711 Okt 30 20:59 bareos.4057.traceback
> -rw-r----- 1 152255384 Okt 30 21:30 bareos-fd.core.4243
> -rw-r----- 1 715 Okt 30 21:30 bareos.4243.traceback
> -rw-r----- 1 152255384 Okt 30 22:00 bareos-fd.core.4466
> -rw-r----- 1 715 Okt 30 22:00 bareos.4466.traceback
> -rw-r----- 1 183 Okt 30 22:00 scout-fd.4466.bactrace
> -rw-r----- 1 152255384 Okt 30 22:30 bareos-fd.core.4675
> -rw-r----- 1 711 Okt 30 22:30 bareos.4675.traceback
> -rw-r----- 1 152255384 Okt 31 13:32 bareos-fd.core.1677
> -rw-r----- 1 715 Okt 31 13:32 bareos.1677.traceback
> -rw-r----- 1 183 Okt 31 13:32 scout-fd.1677.bactrace
> -rw-r----- 1 152255384 Okt 31 14:02 bareos-fd.core.3422
> -rw-r----- 1 716 Okt 31 14:02 bareos.3422.traceback
> -rw-r----- 1 152255384 Okt 31 14:32 bareos-fd.core.17961
> -rw-r----- 1 717 Okt 31 14:32 bareos.17961.traceback
> -rw-r----- 1 154385712 Okt 31 15:02 bareos-fd.core.19198
> -rw-r----- 1 854 Okt 31 15:02 bareos.19198.traceback
> -rw-r----- 1 152255384 Okt 31 15:32 bareos-fd.core.6969
> -rw-r----- 1 713 Okt 31 15:32 bareos.6969.traceback
> -rw-r----- 1 183 Okt 31 15:32 scout-fd.6969.bactrace
> -rw-r----- 1 152255384 Okt 31 16:02 bareos-fd.core.6432
> -rw-r----- 1 715 Okt 31 16:02 bareos.6432.traceback
> -rw-r----- 1 152255384 Okt 31 16:32 bareos-fd.core.6619
> -rw-r----- 1 711 Okt 31 16:32 bareos.6619.traceback
> -rw-r----- 1 183 Okt 31 16:32 scout-fd.6619.bactrace
> -rw-r----- 1 152255384 Okt 31 17:02 bareos-fd.core.7072
> -rw-r----- 1 713 Okt 31 17:02 bareos.7072.traceback
> -rw-r----- 1 183 Okt 31 17:02 scout-fd.7072.bactrace
> -rw-r----- 1 152255384 Okt 31 17:32 bareos-fd.core.7323
> -rw-r----- 1 713 Okt 31 17:32 bareos.7323.traceback
> -rw-r----- 1 152255384 Okt 31 18:02 bareos-fd.core.7515
> -rw-r----- 1 713 Okt 31 18:02 bareos.7515.traceback
> -rw-r----- 1 183 Okt 31 18:02 scout-fd.7515.bactrace
> -rw-r----- 1 152255384 Okt 31 18:32 bareos-fd.core.7699
> -rw-r----- 1 713 Okt 31 18:32 bareos.7699.traceback
> -rw-r----- 1 152255384 Okt 31 19:02 bareos-fd.core.7966
> -rw-r----- 1 713 Okt 31 19:02 bareos.7966.traceback
> -rw-r----- 1 152255384 Okt 31 19:32 bareos-fd.core.8130
> -rw-r----- 1 713 Okt 31 19:32 bareos.8130.traceback
> -rw-r----- 1 183 Okt 31 19:32 scout-fd.8130.bactrace
> -rw-r----- 1 152255384 Okt 31 20:02 bareos-fd.core.8339
> -rw-r----- 1 713 Okt 31 20:02 bareos.8339.traceback
> -rw-r----- 1 152255384 Okt 31 20:32 bareos-fd.core.8515
> -rw-r----- 1 713 Okt 31 20:32 bareos.8515.traceback
> -rw-r----- 1 152255384 Okt 31 21:02 bareos-fd.core.8702
> -rw-r----- 1 711 Okt 31 21:02 bareos.8702.traceback
> -rw-r----- 1 152255384 Okt 31 21:32 bareos-fd.core.8931
> -rw-r----- 1 715 Okt 31 21:32 bareos.8931.traceback
> -rw-r----- 1 152255384 Okt 31 22:02 bareos-fd.core.9135
> -rw-r----- 1 715 Okt 31 22:02 bareos.9135.traceback
> -rw-r----- 1 152255384 Okt 31 22:32 bareos-fd.core.9311
> -rw-r----- 1 711 Okt 31 22:32 bareos.9311.traceback
> -rw-r----- 1 152255384 Nov 4 12:20 bareos-fd.core.1761
> -rw-r----- 1 711 Nov 4 12:20 bareos.1761.traceback
> -rw-r----- 1 183 Nov 4 12:20 scout-fd.1761.bactrace
> -rw-r----- 1 152255384 Nov 8 22:10 bareos-fd.core.1694
> -rw-r----- 1 713 Nov 8 22:10 bareos.1694.traceback
> -rw-r----- 1 152255384 Nov 15 20:41 bareos-fd.core.1720
> -rw-r----- 1 713 Nov 15 20:41 bareos.1720.traceback
> -rw-r----- 1 183 Nov 15 20:41 scout-fd.1720.bactrace
> -rw-r----- 1 152255384 Nov 18 07:59 bareos-fd.core.1691
> -rw-r----- 1 715 Nov 18 07:59 bareos.1691.traceback
> -rw-r----- 1 183 Nov 18 07:59 scout-fd.1691.bactrace
> -rw-r----- 1 152259480 Nov 19 18:18 bareos-fd.core.1704
> -rw-r----- 1 715 Nov 19 18:18 bareos.1704.traceback
> -rw-r----- 1 183 Nov 19 18:18 scout-fd.1704.bactrace
> -rw-r----- 1 152259480 Nov 24 07:12 bareos-fd.core.1772
> -rw-r----- 1 711 Nov 24 07:12 bareos.1772.traceback
> -rw-r----- 1 152259480 Dez 6 19:39 bareos-fd.core.1756
> -rw-r----- 1 713 Dez 6 19:39 bareos.1756.traceback
> -rw-r----- 1 183 Dez 6 19:40 scout-fd.1756.bactrace
> -rw-r----- 1 152259480 Dez 6 21:38 bareos-fd.core.1751
> -rw-r----- 1 715 Dez 6 21:38 bareos.1751.traceback
> -rw-r----- 1 152259480 Dez 8 20:14 bareos-fd.core.1798
> -rw-r----- 1 712 Dez 8 20:14 bareos.1798.traceback
> -rw-r----- 1 183 Dez 8 20:14 scout-fd.1798.bactrace
> -rw-r----- 1 152259480 Dez 9 08:12 bareos-fd.core.1855
> -rw-r----- 1 715 Dez 9 08:12 bareos.1855.traceback
> -rw-r----- 1 183 Dez 9 08:12 scout-fd.1855.bactrace
> -rw-r----- 1 152259480 Dez 9 08:42 bareos-fd.core.4644
> -rw-r----- 1 711 Dez 9 08:42 bareos.4644.traceback
> -rw-r----- 1 183 Dez 9 08:42 scout-fd.4644.bactrace
> -rw-r----- 1 152259480 Dez 9 09:12 bareos-fd.core.5298
> -rw-r----- 1 713 Dez 9 09:12 bareos.5298.traceback
> -rw-r----- 1 152259480 Dez 9 09:42 bareos-fd.core.5620
> -rw-r----- 1 711 Dez 9 09:42 bareos.5620.traceback
> -rw-r----- 1 183 Dez 9 09:42 scout-fd.5620.bactrace
> -rw-r----- 1 133484544 Dez 9 10:13 bareos-fd.core.5885
> -rw-r----- 1 196 Dez 10 00:17 bareos-fd.9102.state
> -rw-r----- 1 5 Dez 10 10:44 bareos-fd.9102.pid
That's ~6 GiB in total.
There should be a limit on the number of core files and on the maximum space to
use for core files.
The latest bareos.1810.traceback contains this:
> reated /var/lib/bareos/bareos-fd.core.1810 for doing postmortem debugging
> [New LWP 1810]
> [New LWP 1811]
> [New LWP 1838]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/sbin/bareos-fd'.
> #0 0x00007f091dd2cb3a in __waitpid (pid=3524, stat_loc=0x7ffc0b55582c,
> options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
> 29 ../sysdeps/unix/sysv/linux/waitpid.c: Datei oder Verzeichnis nicht
> gefunden.
> [Current thread is 1 (Thread 0x7f091f77e740 (LWP 1810))]
> $1 = 1970234227
> $2 = -973383064
> $3 = -973383000
> /usr/lib/bareos/scripts/btraceback.gdb:4: Error in sourced command file:
> No symbol "catalog_db" in current context.
The bttraceback.gdb contains this:
> print my_name
> print exename
> print exepath
> print catalog_db
^^^^^^^^^^ fails
> print version
> print host_os
> print distname
> print distver
> print host_name
> print dist_name
> show env TestName
> bt
> thread apply all bt
> f 0
> info locals
> f 1
> info locals
> f 2
> info locals
> f 3
> info locals
> f 4
> info locals
> f 5
> info locals
> f 6
> info locals
> f 7
> info locals
> detach
> quit
After fixing it here's the gdb output:
> # gdb /usr/sbin/bareos-fd bareos-fd.core.5620 --batch -x
> /usr/lib/bareos/scripts/btraceback.gdb
> [New LWP 5620]
> [New LWP 5621]
> [New LWP 5622]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/sbin/bareos-fd'.
> #0 0x00007fa576604cfa in __waitpid (pid=5860, stat_loc=0x7ffc0bec336c,
> options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
> 29 ../sysdeps/unix/sysv/linux/waitpid.c: Datei oder Verzeichnis nicht
> gefunden.
> [Current thread is 1 (Thread 0x7fa578057d00 (LWP 5620))]
> $1 = 1970234227
> $2 = 389401192
> $3 = 389401256
> $4 = 2003004648
> $5 = 2003004628
> $6 = 2003004621
> $7 = 2003005896
> $8 = 1970234227
> $9 = 2003005936
> Environment variable "TestName" not defined.
> #0 0x00007fa576604cfa in __waitpid (pid=5860, stat_loc=0x7ffc0bec336c,
> options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
> #1 0x00007fa5776272d4 in signal_handler () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #2 <signal handler called>
> #3 0x00007fa57761a9fe in e_msg(char const*, int, int, int, char const*, ...)
> () from /usr/lib/bareos/libbareos-16.2.6.so
> #4 0x00007fa5775fa6ce in bnet_thread_server_tcp(dlist*, int, alist*,
> workq_tag*, bool, void* (*)(void*)) () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #5 0x00005621171033cf in ?? ()
> #6 0x00007fa5755b9561 in __libc_start_main (main=0x5621171028e0, argc=1,
> argv=0x7ffc0bec4e58, init=<optimized out>, fini=<optimized out>,
> rtld_fini=<optimized out>, stack_end=0x7ffc0bec4e48) at
> ../csu/libc-start.c:297
> #7 0x00005621171035ba in ?? ()
>
> Thread 3 (Thread 0x7fa573162700 (LWP 5622)):
> #0 0x00007fa576601076 in futex_abstimed_wait_cancelable (private=<optimized
> out>, abstime=0x7fa573161df0, expected=0, futex_word=0x7fa57784c048) at
> ../sysdeps/unix/sysv/linux/futex-internal.h:205
> #1 __pthread_cond_wait_common (abstime=0x7fa573161df0, mutex=0x7fa57784c060,
> cond=0x7fa57784c020) at pthread_cond_wait.c:539
> #2 __pthread_cond_timedwait (cond=0x7fa57784c020, mutex=0x7fa57784c060,
> abstime=0x7fa573161df0) at pthread_cond_wait.c:667
> #3 0x00007fa5776155ac in bthread_cond_timedwait_p(pthread_cond_t*,
> pthread_mutex_t*, timespec const*, char const*, int) () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #4 0x00007fa57763090f in watchdog_thread () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #5 0x00007fa5776150ff in lmgr_thread_launcher () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #6 0x00007fa5765fa519 in start_thread (arg=0x7fa573162700) at
> pthread_create.c:456
> #7 0x00007fa575685a5f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
>
> Thread 2 (Thread 0x7fa573963700 (LWP 5621)):
> #0 0x00007fa57660499d in nanosleep () at
> ../sysdeps/unix/syscall-template.S:84
> #1 0x00007fa577605dd3 in bmicrosleep(int, int) () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #2 0x00007fa57761503c in check_deadlock(void*) () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #3 0x00007fa5765fa519 in start_thread (arg=0x7fa573963700) at
> pthread_create.c:456
> #4 0x00007fa575685a5f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
>
> Thread 1 (Thread 0x7fa578057d00 (LWP 5620)):
> #0 0x00007fa576604cfa in __waitpid (pid=5860, stat_loc=0x7ffc0bec336c,
> options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
> #1 0x00007fa5776272d4 in signal_handler () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #2 <signal handler called>
> #3 0x00007fa57761a9fe in e_msg(char const*, int, int, int, char const*, ...)
> () from /usr/lib/bareos/libbareos-16.2.6.so
> #4 0x00007fa5775fa6ce in bnet_thread_server_tcp(dlist*, int, alist*,
> workq_tag*, bool, void* (*)(void*)) () from
> /usr/lib/bareos/libbareos-16.2.6.so
> #5 0x00005621171033cf in ?? ()
> #6 0x00007fa5755b9561 in __libc_start_main (main=0x5621171028e0, argc=1,
> argv=0x7ffc0bec4e58, init=<optimized out>, fini=<optimized out>,
> rtld_fini=<optimized out>, stack_end=0x7ffc0bec4e48) at
> ../csu/libc-start.c:297
> #7 0x00005621171035ba in ?? ()
> #0 0x00007fa576604cfa in __waitpid (pid=5860, stat_loc=0x7ffc0bec336c,
> options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
> 29 in ../sysdeps/unix/sysv/linux/waitpid.c
> resultvar = 18446744073709551104
> sc_cancel_oldtype = 0
> #1 0x00007fa5776272d4 in signal_handler () from
> /usr/lib/bareos/libbareos-16.2.6.so
> No symbol table info available.
> #2 <signal handler called>
> No locals.
> #3 0x00007fa57761a9fe in e_msg(char const*, int, int, int, char const*, ...)
> () from /usr/lib/bareos/libbareos-16.2.6.so
> No symbol table info available.
> #4 0x00007fa5775fa6ce in bnet_thread_server_tcp(dlist*, int, alist*,
> workq_tag*, bool, void* (*)(void*)) () from
> /usr/lib/bareos/libbareos-16.2.6.so
> No symbol table info available.
> #5 0x00005621171033cf in ?? ()
> No symbol table info available.
> #6 0x00007fa5755b9561 in __libc_start_main (main=0x5621171028e0, argc=1,
> argv=0x7ffc0bec4e58, init=<optimized out>, fini=<optimized out>,
> rtld_fini=<optimized out>, stack_end=0x7ffc0bec4e48) at
> ../csu/libc-start.c:297
> 297 ../csu/libc-start.c: Datei oder Verzeichnis nicht gefunden.
> result = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -5235568947671730364,
> 94700120847760, 140720508522064, 0, 0, -1950696135560136892,
> -1990800644230351036}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0,
> 0x7ffc0bec4e68, 0x7fa5780a7150}, data = {prev = 0x0, cleanup = 0x0,
> canceltype = 200035944}}}
> not_first_call = <optimized out>
> #7 0x00005621171035ba in ?? ()
> No symbol table info available.
If you need more info, please ask. Maybe I can find some time to debug that
myself.
Thanks for maintaining Bareos anyway.
-- System Information:
Debian Release: buster/sid
APT prefers unstable
APT policy: (990, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.14.3 (SMP w/2 CPU cores; PREEMPT)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8),
LANGUAGE=de:en_US (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages bareos-filedaemon depends on:
ii adduser 3.116
ii bareos-common 16.2.6-3
ii debconf [debconf-2.0] 1.5.65
ii init-system-helpers 1.51
ii libacl1 2.2.52-3+b1
ii libc6 2.25-3
ii libcap2 1:2.25-1.2
ii libgcc1 1:7.2.0-17
ii libgnutls30 3.5.16-1
ii libjansson4 2.10-1
ii liblzo2-2 2.08-1.2+b2
ii libstdc++6 7.2.0-17
ii libwrap0 7.6.q-27
ii lsb-base 9.20170808
ii lsof 4.89+dfsg-0.1
ii zlib1g 1:1.2.8.dfsg-5
bareos-filedaemon recommends no packages.
bareos-filedaemon suggests no packages.
-- no debconf information
--- End Message ---
