In the last couple of days since upgrading bareos to the latest in the "current" repository I've found the director getting stuck talking to the backend database.
I'll execute "update slots" or "run" and then the console hangs and I get an error like this: *run Could not open catalog database "8^". The database name is some random set of characters. This kind of sounds like a memory bounds error. Here's the version of bareos I'm running. Surprised that a "pre-release" is in "current" rather than "next" or "experimental". *version bareos-dir Version: 23.1.1~pre3.47aeb30a1 (21 November 2024) Red Hat Enterprise Linux release 9.5 (Plow) redhat Red Hat Enterprise Linux release 9.5 (Plow) Watching /var/log/messages I see this when the job fails: Nov 27 15:22:50 bareos bareos-dir[2021875]: Connect failure: ERR=error:0A0000C7:SSL routines::peer did not return a certificate Nov 27 15:22:50 bareos bareos-dir[2021875]: lib/bnet.cc:125 TLS Negotiation failed. I do have all of my daemons setup to use TLS certificates and it's been working fine for a few months on this installation. Disconnecting from bconsole and reconnecting and I see these errors: 27-Nov 15:18 bareos-dir JobId 0: Fatal error: cats/postgresql.cc:232 Unable to connect to PostgreSQL server. Database=8^ User= Possible causes: SQL server not running; password incorrect; max_connections exceeded. (connection pointer is NULL ) 27-Nov 15:22 bareos-dir JobId 0: Fatal error: cats/postgresql.cc:232 Unable to connect to PostgreSQL server. Database=8^ User= Possible causes: SQL server not running; password incorrect; max_connections exceeded. (connection pointer is NULL ) The postgresql server is local and only used by bareos. After restarting the console application things appear to be happier. I had left the console running for quite some time. This morning when it happened bareos-dir crashed and gave me a traceback, shown below Reading symbols from /usr/sbin/bareos-dir... Reading symbols from .gnu_debugdata for /usr/sbin/bareos-dir... (No debugging symbols found in .gnu_debugdata for /usr/sbin/bareos-dir) Attaching to program: /usr/sbin/bareos-dir, process 1772827 [New LWP 1772830] [New LWP 1772831] [New LWP 1780188] [New LWP 1799021] [New LWP 1799033] [New LWP 1803537] [New LWP 1803538] [New LWP 1803539] [New LWP 1803540] [New LWP 1803541] [New LWP 1803542] [New LWP 1803543] [New LWP 1803544] [New LWP 2018381] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install bareos -director-23.1.1~pre3.47aeb30a1-149.el9.x86_64 (gdb) 'my_name' has unknown type; cast it to its declared type (gdb) 'exename' has unknown type; cast it to its declared type (gdb) 'exepath' has unknown type; cast it to its declared type (gdb) No symbol "catalog_db" in current context. (gdb) 'kBareosVersionStrings' has unknown type; cast it to its declared type (gdb) 'kBareosVersionStrings' has unknown type; cast it to its declared type (gdb) 'host_name' has unknown type; cast it to its declared type (gdb) Environment variable "TestName" not defined. (gdb) #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x000055f9bacfb9fb in main () (gdb) Thread 15 (Thread 0x7f06a2ffd640 (LWP 2018381) "bareos-dir"): #0 0x00007f06e5301aff in poll () from /lib64/libc.so.6 #1 0x00007f06e596cdd9 in WaitForReadableFd(int, int, bool) () from /usr/lib64/bareos/libbareos.so.23 #2 0x00007f06e597c889 in TlsOpenSslPrivate::OpensslBsockReadwrite(BareosSocket*, char*, int, bool) () from /usr/lib64/bareos/libbareos.so.23 #3 0x00007f06e5902294 in BareosSocketTCP::recv() () from /usr/lib64/bareos /libbareos.so.23 #4 0x000055f9badaf070 in directordaemon::BgetDirmsg(BareosSocket*, bool) () #5 0x000055f9bad43f1f in msg_thread () #6 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #7 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 14 (Thread 0x7f06cbfff640 (LWP 1803544) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 13 (Thread 0x7f06e19f9640 (LWP 1803543) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 12 (Thread 0x7f06e29fb640 (LWP 1803542) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 11 (Thread 0x7f06e31fc640 (LWP 1803541) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 10 (Thread 0x7f06e09f7640 (LWP 1803540) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 9 (Thread 0x7f06c97fa640 (LWP 1803539) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 8 (Thread 0x7f06e11f8640 (LWP 1803538) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 7 (Thread 0x7f06caffd640 (LWP 1803537) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 6 (Thread 0x7f06ca7fc640 (LWP 1799033) "bareos-dir"): #0 0x00007f06e52868ba in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f06e5289692 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6 #2 0x000055f9bad4a3e3 in directordaemon::WaitForStorageDaemonTermination(JobControlRecord*) () #3 0x000055f9bad9f2dd in directordaemon::DoNativeVbackup(JobControlRecord*) () #4 0x000055f9bad3ae7d in directordaemon::job_thread(void*) [clone .lto_priv.0] () #5 0x000055f9bad41e90 in jobq_server () #6 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #7 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 5 (Thread 0x7f06e21fa640 (LWP 1799021) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos /libbareos.so.23 #3 0x000055f9bad42803 in jobq_server () #4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 4 (Thread 0x7f06e39fd640 (LWP 1780188) "bareos-dir"): #0 0x00007f06e52d8b3f in wait4 () from /lib64/libc.so.6 #1 0x00007f06e597566b in SignalHandler () from /usr/lib64/bareos /libbareos.so.23 #2 <signal handler called> #3 0x00007f06e5370c64 in __strnlen_evex () from /lib64/libc.so.6 #4 0x00007f06e58fdc52 in Bvsnprintf(char*, int, char const*, __va_list_tag*) () from /usr/lib64/bareos/libbareos.so.23 #5 0x00007f06e595beb3 in PoolMem::Bvsprintf(char const*, __va_list_tag*) () from /usr/lib64/bareos/libbareos.so.23 #6 0x000055f9bad77638 in directordaemon::UaContext::vSendMsg(int, char const*, char const*, __va_list_tag*) () #7 0x000055f9bad77e5a in directordaemon::UaContext::ErrorMsg(char const*, ...) () #8 0x000055f9bad6a6aa in directordaemon::OpenDb(directordaemon::UaContext*, bool) [clone .part.0] () #9 0x000055f9bad98846 in directordaemon::UpdateCmd(directordaemon::UaContext*, char const*) () #10 0x000055f9bad625df in directordaemon::Do_a_command(directordaemon::UaContext*) () #11 0x000055f9bad50ce2 in directordaemon::HandleConnectionRequest(ConfigurationParser*, void*) () #12 0x00007f06e59735a2 in WorkerThread(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>) () from /usr/lib64/bareos /libbareos.so.23 #13 0x00007f06e596c515 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> > > >::_M_run() () from /usr/lib64/bareos/libbareos.so.23 #14 0x00007f06e56dbad4 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #15 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #16 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 3 (Thread 0x7f06e41fe640 (LWP 1772831) "bareos-dir"): #0 0x00007f06e5301aff in poll () from /lib64/libc.so.6 #1 0x00007f06e58fa598 in BnetThreadServerTcp(dlist<IPADDR>*, alist<s_sockfd*>*, ThreadList&, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, std::atomic<BnetServerState>*, std::function<void* (void*)>, std::function<void ()>) () from /usr/lib64/ bareos/libbareos.so.23 #2 0x000055f9bad51fdf in connect_thread () #3 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #4 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 2 (Thread 0x7f06e49ff640 (LWP 1772830) "bareos-dir"): #0 0x00007f06e52868ba in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f06e52893c4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f06e59862c0 in watchdog_thread () from /usr/lib64/bareos /libbareos.so.23 #3 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6 #4 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6 Thread 1 (Thread 0x7f06e4e69880 (LWP 1772827) "bareos-dir"): #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 #2 0x000055f9bacfb9fb in main () (gdb) #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6 (gdb) No symbol table info available. (gdb) #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6 (gdb) No symbol table info available. (gdb) #2 0x000055f9bacfb9fb in main () (gdb) No symbol table info available. (gdb) No frame at level 3. (gdb) No symbol table info available. (gdb) No frame at level 4. (gdb) No symbol table info available. (gdb) No frame at level 5. (gdb) No symbol table info available. (gdb) No frame at level 6. (gdb) No symbol table info available. (gdb) No frame at level 7. (gdb) No symbol table info available. (gdb) Detaching from program: /usr/sbin/bareos-dir, process 1772827 [Inferior 1 (process 1772827) detached] (gdb) -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/bareos-users/59f8f0fd-fc28-4b6c-be50-112cae49b0b9n%40googlegroups.com.
