On 2020-07-02 14:10, Martin Simmons wrote: >>>>>> On Wed, 1 Jul 2020 15:54:27 -0400, Phil Stracchino said: >> >> On 2020-06-26 17:58, Phil Stracchino wrote: >>> Oh, another detail I learned this morning: >>> >>> Cancelling a SINGLE hung job does not crash the Director. It appears it >>> is only attempting to cancel a SECOND hung job that causes the Director >>> to crash. >> >> >> After four days of normal operation I got another hung job this morning, >> still with HAproxy not in use. So it is *much less* prevalent without >> HAproxy but still occurring (one failure every 3-4 days instead of 2-3 >> failed jobs per day). This is the same running Director process as >> handled the failure mentioned above, and the Director was frozen and >> unresponsive for nearly ten minutes attempting to cancel the second job >> before it finally marked it as cancelled. (But the job still shows as >> running and with a fatal error.) > > I'm confused how it can be the same Director process. Didn't that Director > crash when you attempted to cancel the second job?
No, I had a single hung job which was successfully cancelled (though it took a long time), and then no more failures until yesterday when there was again a single hung job (which also took a long time to cancel but did eventually successfully cancel). It seems as long as I don't try to cancel two hung jobs at once, there is no crash. > > >> ==== >> *cancel >> Select Job(s): >> 1: JobId=25114 Job=Narn_Backup.2020-07-01_04.30.00_37 >> 2: JobId=25115 Job=MySQL_Backup_New.2020-07-01_04.55.00_39 >> Choose Job list to cancel (1-2): 1 >> JobId=25114 Job=Narn_Backup.2020-07-01_04.30.00_37 >> Confirm cancel of 1 Job (yes/no): yes >> Failed to connect to File daemon. > > This failure to connect is strange. > > >> Martin, you're still convinced this has to do with the SD? > > I think the hanging jobs are caused by a problem with how Bacula recovers from > an SQL error while inserting attributes into the database. This is not a > problem with the SD itself, but the hanging does involve the SD so that's why > it would be useful to get simultaneous backtraces from all 3 daemons while > they are hanging. > > >> I'm still >> working on trying to figure out how I can get dbx installed so that I >> can backtrace the SD. pstack isn't very informative: >> >> asgard:root:~:1 # pstack 10971 >> 10971: /opt/bacula/sbin/bacula-sd -v -c /opt/bacula/etc/bacula-sd.conf >> ------------ lwp# 1 / thread# 1 --------------- >> ffff80ffbf580daa pollsys (ffff80ffbfffcd10, 1, 0, 0) >> ffff80ffbf525755 pselect () + 181 >> ffff80ffbf525bd4 select () + 68 >> ffff80ffb5211023 __1cSbnet_thread_server6FpnFdlist_ipnJworkq_tag_pFpv_4_v_ >> () + 963 >> 0000000000419c84 main () + 724 >> 3f763a7574735070 ???????? () >> ------------ lwp# 3 / thread# 3 --------------- >> ffff80ffbf577e97 lwp_park (0, ffff80ffbe5bbe30, 0) >> ffff80ffbf5716fa cond_wait_queue () + 62 >> ffff80ffbf571b38 cond_wait_common () + 1dc >> ffff80ffbf571dc7 __cond_timedwait () + a7 >> ffff80ffbf571e11 cond_timedwait () + 29 >> ffff80ffbf571e45 pthread_cond_timedwait () + 9 >> ffff80ffb526a41a watchdog_thread () + 57a >> ------------ lwp# 70 / thread# 70 --------------- >> ffff80ffbf580e7a read (10, ffff80ffbd3c495c, 4) >> ffff80ffb524fdbe __1cJBSOCKCORELread_nbytes6Mpci_i_ () + 4e > > So it just stops there? That's annoying. > > Which compiler and options were used to compile on Solaris? What is the > setting for CFLAGS in the generated bacula/src/stored/Makefile? Sun Developer Studio 12.6 Configure invocation: ./configure --prefix=/opt/bacula --with-dump-email=r...@caerllewys.net --with-job-email=r...@caerllewys.net --with-smtp-host=smtp.caerllewys.net --with-subsys-dir=/opt/bacula/var --with-working-dir=/opt/bacula/var --enable-build-stored --disable-build-dird --enable-smartalloc --with-mysql=/opt/mysql/mysql CC=/opt/suncc/bin/CC CFLAGS='-fast -xarch=generic -xtarget=generic -xcache=generic -m64 -g' CPPFLAGS='-fast -xarch=generic -xtarget=generic -xcache=generic -m64 -g' CXX=/opt/suncc/bin/CC CXXFLAGS='-march=native -mfpmath=sse -pipe -m64 -g' LDFLAGS="-m64" Which, as expected, yields: asgard:root:/netstore/src/bacula-9.6.5:11 # grep CFLAGS src/stored/Makefile CFLAGS = -fast -xarch=generic -xtarget=generic -xcache=generic -m64 -g -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 _______________________________________________ Bacula-devel mailing list Bacula-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-devel
