On 2020-06-24 07:19, kern wrote:
> OK.
OK, this morning after rebuilding the SD with smartalloc, and switching
the Director's connection back via HAproxy after increasing HAproxy's
client and server timeouts to one hour, five of six backup jobs are
hung. This time, client asgard's OS backup succeeded without errors
(writing 153 files totalling 71.63MB), but the 'Netstore backup' job
(which backs up the NAS storage, and ALSO runs on asgard, concurrently
with asgard's OS backup) hung. I mention this only because it's the
first time I've seen the two jobs on that client split that way with one
succeeded and one failed.
I also note that BAT's 'Jobs Run' list says that all of the jobs are
still running, but Status Dir in bconsole says they all have fatal
errors. Jobs Run does not reflect this. Status of the clients ALSO
does not report an error; as far as the clients are concerned, the job
is still running.
Running Jobs:
Console connected at 24-Jun-20 07:47
Console connected at 24-Jun-20 07:52
JobId Type Level Files Bytes Name Status
======================================================================
25053 Back Incr 0 0 Netstore Backup has a fatal error
25054 Back Incr 0 0 Babylon5 Backup has a fatal error
25055 Back Incr 0 0 Fisherprice Backup has a fatal error
25056 Back Incr 0 0 Minbar Backup has a fatal error
25057 Back Incr 0 0 Narn Backup has a fatal error
25058 Back Incr 0 0 MySQL Backup New is waiting for
higher priority jobs to finish
====
Client status says each client is stuck processing some random file:
*status client=minbar
Connecting to Client minbar at minbar.caerllewys.net:9102
minbar-fd Version: 9.6.5 (11 June 2020) x86_64-pc-linux-gnu gentoo
Daemon started 20-Jun-20 14:39. Jobs: run=6 running=1.
Heap: heap=18,446,744,073,709,547,520 smbytes=938,731
max_bytes=1,194,379 bufs=213 max_bufs=278
Sizes: boffset_t=8 size_t=8 debug=0 trace=0 mode=0,0 bwlimit=0kB/s
Plugin: bpipe-fd.so
Running Jobs:
JobId 25056 Job Minbar_Backup.2020-06-24_04.30.00_06 is running.
Incremental Backup Job started: 24-Jun-20 04:30
Files=62 Bytes=20,215,020 AveBytes/sec=1,595 LastBytes/sec=1,595
Errors=0
Bwlimit=0 ReadBytes=20,280,556
Files: Examined=3,550 Backed up=62
Processing file: /home/grimes/recurse.log
SDReadSeqNo=6 fd=6 SDtls=0
Director connected at: 24-Jun-20 08:01
====
*status client=babylon5
Connecting to Client babylon5 at babylon5.caerllewys.net:9102
babylon5-fd Version: 9.6.5 (11 June 2020) x86_64-pc-linux-gnu gentoo
Daemon started 23-Jun-20 20:07. Jobs: run=0 running=1.
Heap: heap=102,400 smbytes=429,390 max_bytes=441,829 bufs=254 max_bufs=296
Sizes: boffset_t=8 size_t=8 debug=0 trace=0 mode=0,0 bwlimit=0kB/s
Plugin: bpipe-fd.so
Running Jobs:
JobId 25054 Job Babylon5_Backup.2020-06-24_04.30.00_04 is running.
Incremental Backup Job started: 24-Jun-20 04:30
Files=1,033 Bytes=186,607,064 AveBytes/sec=14,693
LastBytes/sec=14,693 Errors=0
Bwlimit=0 ReadBytes=186,672,557
Files: Examined=73,024 Backed up=1,033
Processing file: /home/alaric/.moonchild productions/pale
moon/alaric/adblockplus/patterns-backup5.ini
SDReadSeqNo=6 fd=6 SDtls=0
Director connected at: 24-Jun-20 08:01
====
*status client=narn
Connecting to Client narn at narn.caerllewys.net:9102
narn-fd Version: 9.6.5 (11 June 2020) x86_64-pc-linux-gnu gentoo
Daemon started 22-Jun-20 17:15. Jobs: run=1 running=1.
Heap: heap=102,400 smbytes=552,487 max_bytes=552,504 bufs=253 max_bufs=263
Sizes: boffset_t=8 size_t=8 debug=0 trace=0 mode=0,0 bwlimit=0kB/s
Plugin: bpipe-fd.so
Running Jobs:
JobId 25057 Job Narn_Backup.2020-06-24_04.30.00_07 is running.
Incremental Backup Job started: 24-Jun-20 04:30
Files=2,436 Bytes=16,360,625 AveBytes/sec=1,287 LastBytes/sec=1,287
Errors=0
Bwlimit=0 ReadBytes=16,360,625
Files: Examined=279,692 Backed up=2,436
Processing file:
/usr/src/linux-5.7.5-gentoo/tools/power/cpupower/debug/i386/Makefile
SDReadSeqNo=6 fd=6 SDtls=0
Director connected at: 24-Jun-20 08:01
====
*status client=asgard
Connecting to Client asgard at asgard.caerllewys.net:9102
asgard-fd Version: 9.6.5 (11 June 2020) i386-pc-solaris2.11 solaris 5.11
Daemon started 23-Jun-20 14:06. Jobs: run=1 running=1.
Heap: heap=1,310,720 smbytes=452,226 max_bytes=556,610 bufs=232
max_bufs=380
Sizes: boffset_t=8 size_t=8 debug=0 trace=0 mode=0,0 bwlimit=0kB/s
Plugin: bpipe-fd.so
Running Jobs:
JobId 25053 Job Netstore_Backup.2020-06-24_04.30.00_03 is running.
Incremental Backup Job started: 24-Jun-20 04:30
Files=37 Bytes=34,679,690 AveBytes/sec=2,728 LastBytes/sec=2,728
Errors=0
Bwlimit=0 ReadBytes=34,745,226
Files: Examined=155,075 Backed up=37
Processing file: /netstore/src/bacula-9.6.5/src/dird/.libs/bacula-dir
SDReadSeqNo=6 fd=13 SDtls=0
Director connected at: 24-Jun-20 08:01
====
*status client=fisherprice
Connecting to Client fisherprice at fisherprice.caerllewys.net:9102
bacula-fd Version: 9.4.4 (28 May 2019) x86_64-redhat-linux-gnu redhat
Daemon started 15-Jun-20 14:06. Jobs: run=13 running=0.
Heap: heap=12,288 smbytes=412,333 max_bytes=1,486,386 bufs=183
max_bufs=2,887
Sizes: boffset_t=8 size_t=8 debug=200 trace=1 mode=0,0 bwlimit=0kB/s
Plugin: bpipe-fd.so(1)
Running Jobs:
JobId 25055 Job Fisherprice_Backup.2020-06-24_04.30.00_05 is running.
Incremental Backup Job started: 24-Jun-20 04:30
Files=24 Bytes=82,215,960 AveBytes/sec=6,417 LastBytes/sec=6,417
Errors=0
Bwlimit=0 ReadBytes=82,281,496
Files: Examined=5,394 Backed up=24
Processing file: /var/cache/dnf/updates-filenames.solvx
SDReadSeqNo=6 fd=10 SDtls=0
Director connected at: 24-Jun-20 08:03
====
I have yet to see the hang occur on the same file twice on a client, so
it appears to have nothing to do with the files themselves. Often a
re-run of the job on the same client will succeed even with HAproxy.
There is a strong random factor in the failure.
I don't have gdb on the Solaris server that hosts the SD. I'm going to
have to install it, which may take me a little while to figure out how
to correctly compile it on Solaris 11.3. (I'll also have to rebuild
Bacula on asgard with debugging enabled.) But I'd say the problem here
is clearly tied to the combination of Bacula Director 9.6.5
(specifically) + HAproxy. It does not occur if the Director is 9.6.3
and connecting to the DB via HAproxy, or if the Director is 9.6.5 but
connecting directly to a single node of the DB cluster. It doesn't seem
to matter what version the clients are, and it doesn't seem to matter
whether the SD is also on 9.6.5. And it doesn't appear to be a HAproxy
timeout issue.
I'd already been considering migrating from HAproxy to ProxySQL, so
along with everything else I'll go ahead and get ProxySQL set up and see
whether I can also reproduce the problem using ProxySQL.
Tracebacks:
Director:
Thread 11 (Thread 0x7f01deaa0700 (LWP 26244)):
#0 0x00007f01e1596dfc in read () from /lib64/libpthread.so.0
#1 0x00007f01e15eaed3 in BSOCKCORE::socketRead (this=0x7f01d0009c18,
len=4, buf=0x7f01dea9fdf4, fd=<optimized out>) at ../lib/bsockcore.h:202
#2 BSOCKCORE::read_nbytes (nbytes=<optimized out>, ptr=0x7f01dea9fdf4
"\001\177", this=<optimized out>) at bsockcore.c:1144
#3 BSOCKCORE::read_nbytes (this=0x7f01d0009c18, ptr=<optimized out>,
nbytes=4) at bsockcore.c:1130
#4 0x00007f01e15c43bd in BSOCK::recv (this=this@entry=0x7f01d0009c18)
at bsock.c:441
#5 0x00005591f0edd6d6 in handle_UA_client_request (arg=0x7f01d0009c18)
at ua_server.c:144
#6 0x00007f01e15f95b5 in workq_server (arg=0x5591f0f209c0 <ua_workq>)
at workq.c:372
#7 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f01c67fc700 (LWP 24613)):
#0 0x00007f01e1596dfc in read () from /lib64/libpthread.so.0
#1 0x00007f01e15eaed3 in BSOCKCORE::socketRead (this=0x7f01d0004b78,
len=4, buf=0x7f01c67fbdf4, fd=<optimized out>) at ../lib/bsockcore.h:202
#2 BSOCKCORE::read_nbytes (nbytes=<optimized out>, ptr=0x7f01c67fbdf4
"\001\177", this=<optimized out>) at bsockcore.c:1144
#3 BSOCKCORE::read_nbytes (this=0x7f01d0004b78, ptr=<optimized out>,
nbytes=4) at bsockcore.c:1130
#4 0x00007f01e15c43bd in BSOCK::recv (this=this@entry=0x7f01d0004b78)
at bsock.c:441
#5 0x00005591f0edd6d6 in handle_UA_client_request (arg=0x7f01d0004b78)
at ua_server.c:144
#6 0x00007f01e15f95b5 in workq_server (arg=0x5591f0f209c0 <ua_workq>)
at workq.c:372
#7 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f01de29f700 (LWP 8895)):
#0 0x00007f01e15974c5 in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f01e15c04e6 in bmicrosleep (sec=sec@entry=2,
usec=usec@entry=0) at bsys.c:192
#2 0x00005591f0ea8d92 in jobq_server (arg=0x5591f0f206a0 <job_queue>)
at jobq.c:616
#3 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f01c6ffd700 (LWP 1625)):
#0 0x00007f01e1596dfc in read () from /lib64/libpthread.so.0
#1 0x00007f01e15eaed3 in BSOCKCORE::socketRead (this=0x7f01bc006668,
len=4, buf=0x7f01c6ffc9b4, fd=<optimized out>) at ../lib/bsockcore.h:202
#2 BSOCKCORE::read_nbytes (nbytes=<optimized out>, ptr=0x7f01c6ffc9b4
"\372\377\377\377", this=<optimized out>) at bsockcore.c:1144
#3 BSOCKCORE::read_nbytes (this=0x7f01bc006668, ptr=<optimized out>,
nbytes=4) at bsockcore.c:1130
#4 0x00007f01e15c43bd in BSOCK::recv (this=this@entry=0x7f01bc006668)
at bsock.c:441
#5 0x00005591f0e9f6e7 in bget_dirmsg (bs=bs@entry=0x7f01bc006668) at
getmsg.c:150
#6 0x00005591f0e8dd78 in wait_for_job_termination
(jcr=jcr@entry=0x5591f18b59c8, timeout=timeout@entry=0) at backup.c:685
#7 0x00005591f0e90009 in do_backup (jcr=jcr@entry=0x5591f18b59c8) at
backup.c:633
#8 0x00005591f0ea2318 in job_thread (arg=0x5591f18b59c8) at job.c:453
#9 0x00005591f0ea87fb in jobq_server (arg=0x5591f0f206a0 <job_queue>)
at jobq.c:468
#10 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f01c77fe700 (LWP 1624)):
#0 0x00007f01e1596dfc in read () from /lib64/libpthread.so.0
#1 0x00007f01e15eaed3 in BSOCKCORE::socketRead (this=0x7f01b8006678,
len=4, buf=0x7f01c77fd9b4, fd=<optimized out>) at ../lib/bsockcore.h:202
#2 BSOCKCORE::read_nbytes (nbytes=<optimized out>, ptr=0x7f01c77fd9b4
"\372\377\377\377", this=<optimized out>) at bsockcore.c:1144
#3 BSOCKCORE::read_nbytes (this=0x7f01b8006678, ptr=<optimized out>,
nbytes=4) at bsockcore.c:1130
#4 0x00007f01e15c43bd in BSOCK::recv (this=this@entry=0x7f01b8006678)
at bsock.c:441
#5 0x00005591f0e9f6e7 in bget_dirmsg (bs=bs@entry=0x7f01b8006678) at
getmsg.c:150
#6 0x00005591f0e8dd78 in wait_for_job_termination
(jcr=jcr@entry=0x5591f18aa678, timeout=timeout@entry=0) at backup.c:685
#7 0x00005591f0e90009 in do_backup (jcr=jcr@entry=0x5591f18aa678) at
backup.c:633
#8 0x00005591f0ea2318 in job_thread (arg=0x5591f18aa678) at job.c:453
#9 0x00005591f0ea87fb in jobq_server (arg=0x5591f0f206a0 <job_queue>)
at jobq.c:468
#10 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f01dca9c700 (LWP 1620)):
#0 0x00007f01e1596dfc in read () from /lib64/libpthread.so.0
#1 0x00007f01e15eaed3 in BSOCKCORE::socketRead (this=0x7f01cc0067b8,
len=4, buf=0x7f01dca9b9b4, fd=<optimized out>) at ../lib/bsockcore.h:202
#2 BSOCKCORE::read_nbytes (nbytes=<optimized out>, ptr=0x7f01dca9b9b4
"\372\377\377\377", this=<optimized out>) at bsockcore.c:1144
#3 BSOCKCORE::read_nbytes (this=0x7f01cc0067b8, ptr=<optimized out>,
nbytes=4) at bsockcore.c:1130
#4 0x00007f01e15c43bd in BSOCK::recv (this=this@entry=0x7f01cc0067b8)
at bsock.c:441
#5 0x00005591f0e9f6e7 in bget_dirmsg (bs=bs@entry=0x7f01cc0067b8) at
getmsg.c:150
#6 0x00005591f0e8dd78 in wait_for_job_termination
(jcr=jcr@entry=0x5591f189f308, timeout=timeout@entry=0) at backup.c:685
#7 0x00005591f0e90009 in do_backup (jcr=jcr@entry=0x5591f189f308) at
backup.c:633
#8 0x00005591f0ea2318 in job_thread (arg=0x5591f189f308) at job.c:453
#9 0x00005591f0ea87fb in jobq_server (arg=0x5591f0f206a0 <job_queue>)
at jobq.c:468
#10 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--c
Thread 5 (Thread 0x7f01dd29d700 (LWP 1619)):
#0 0x00007f01e1596dfc in read () from /lib64/libpthread.so.0
#1 0x00007f01e15eaed3 in BSOCKCORE::socketRead (this=0x7f01c80067a8,
len=4, buf=0x7f01dd29c9b4, fd=<optimized out>) at ../lib/bsockcore.h:202
#2 BSOCKCORE::read_nbytes (nbytes=<optimized out>, ptr=0x7f01dd29c9b4
"\372\377\377\377", this=<optimized out>) at bsockcore.c:1144
#3 BSOCKCORE::read_nbytes (this=0x7f01c80067a8, ptr=<optimized out>,
nbytes=4) at bsockcore.c:1130
#4 0x00007f01e15c43bd in BSOCK::recv (this=this@entry=0x7f01c80067a8)
at bsock.c:441
#5 0x00005591f0e9f6e7 in bget_dirmsg (bs=bs@entry=0x7f01c80067a8) at
getmsg.c:150
#6 0x00005591f0e8dd78 in wait_for_job_termination
(jcr=jcr@entry=0x5591f1894038, timeout=timeout@entry=0) at backup.c:685
#7 0x00005591f0e90009 in do_backup (jcr=jcr@entry=0x5591f1894038) at
backup.c:633
#8 0x00005591f0ea2318 in job_thread (arg=0x5591f1894038) at job.c:453
#9 0x00005591f0ea87fb in jobq_server (arg=0x5591f0f206a0 <job_queue>)
at jobq.c:468
#10 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f01dda9e700 (LWP 1618)):
#0 0x00007f01e1596dfc in read () from /lib64/libpthread.so.0
#1 0x00007f01e15eaed3 in BSOCKCORE::socketRead (this=0x7f01d4003958,
len=4, buf=0x7f01dda9d9b4, fd=<optimized out>) at ../lib/bsockcore.h:202
#2 BSOCKCORE::read_nbytes (nbytes=<optimized out>, ptr=0x7f01dda9d9b4
"\372\377\377\377", this=<optimized out>) at bsockcore.c:1144
#3 BSOCKCORE::read_nbytes (this=0x7f01d4003958, ptr=<optimized out>,
nbytes=4) at bsockcore.c:1130
#4 0x00007f01e15c43bd in BSOCK::recv (this=this@entry=0x7f01d4003958)
at bsock.c:441
#5 0x00005591f0e9f6e7 in bget_dirmsg (bs=bs@entry=0x7f01d4003958) at
getmsg.c:150
#6 0x00005591f0e8dd78 in wait_for_job_termination
(jcr=jcr@entry=0x5591f18c0e98, timeout=timeout@entry=0) at backup.c:685
#7 0x00005591f0e90009 in do_backup (jcr=jcr@entry=0x5591f18c0e98) at
backup.c:633
#8 0x00005591f0ea2318 in job_thread (arg=0x5591f18c0e98) at job.c:453
#9 0x00005591f0ea87fb in jobq_server (arg=0x5591f0f206a0 <job_queue>)
at jobq.c:468
#10 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f01df2a1700 (LWP 14773)):
#0 0x00007f01e1593878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007f01e15f8be9 in watchdog_thread (arg=<optimized out>) at
watchdog.c:299
#2 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f01dfaa2700 (LWP 14772)):
#0 0x00007f01e12439d3 in select () from /lib64/libc.so.6
#1 0x00007f01e15c3a38 in bnet_thread_server
(addrs=addrs@entry=0x5591f183d718, max_clients=20,
client_wq=client_wq@entry=0x5591f0f209c0 <ua_workq>,
handle_client_request=handle_client_request@entry=0x5591f0edd650
<handle_UA_client_request(void*)>) at bnet_server.c:166
#2 0x00005591f0edd296 in connect_thread (arg=0x5591f183d718) at
ua_server.c:85
#3 0x00007f01e158cea7 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f01e124bc6f in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f01e05650c0 (LWP 14765)):
#0 0x00007f01e15974c5 in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f01e15c04e6 in bmicrosleep (sec=sec@entry=60,
usec=usec@entry=0) at bsys.c:192
#2 0x00005591f0eb50d4 in wait_for_next_job
(one_shot_job_to_run=<optimized out>) at scheduler.c:121
#3 0x00005591f0e875f5 in main (argc=<optimized out>, argv=<optimized
out>) at dird.c:387
Director's FD:
Thread 4 (Thread 0x7f5d76f33700 (LWP 1638)):
#0 0x00007f5d78fbb9d3 in select () from /lib64/libc.so.6
#1 0x00007f5d7935e10a in fd_wait_data (fd=6, mode=mode@entry=WAIT_READ,
sec=sec@entry=5, msec=msec@entry=0) at bsys.c:1206
#2 0x00007f5d793866cb in BSOCKCORE::wait_data_intr
(this=this@entry=0x7f5d68000b88, sec=sec@entry=5, msec=msec@entry=0) at
bsockcore.c:875
#3 0x00005587c5a02118 in sd_heartbeat_thread (arg=0x7f5d6c000b88) at
heartbeat.c:69
#4 0x00007f5d79328ea7 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f5d78fc3c6f in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f5d7671a700 (LWP 1636)):
#0 0x00007f5d79332d5f in write () from /lib64/libpthread.so.0
#1 0x00007f5d79386daf in BSOCKCORE::socketWrite (this=0x7f5d6c003b58,
len=637, buf=0x7f5d6c0ad232, fd=<optimized out>) at ../lib/bsockcore.h:203
#2 BSOCKCORE::write_nbytes (nbytes=<optimized out>, ptr=0x7f5d6c0ad232
"\033\001\025?920\322\024)\tg", this=<optimized out>) at bsockcore.c:1079
#3 BSOCKCORE::write_nbytes (this=this@entry=0x7f5d6c003b58,
ptr=<optimized out>, nbytes=2419) at bsockcore.c:1064
#4 0x00007f5d79362578 in BSOCK::write_nbytes (this=0x7f5d6c003b58,
ptr=<optimized out>, nbytes=2419) at bsock.c:831
#5 0x00007f5d7936151b in BSOCK::send (aflags=0, this=0x7f5d6c003b58) at
bsock.c:368
#6 BSOCK::send (this=this@entry=0x7f5d6c003b58, aflags=aflags@entry=0)
at bsock.c:249
#7 0x00005587c59f708b in BSOCK::send (this=0x7f5d6c003b58) at
../lib/bsock.h:75
#8 process_and_send_data (bctx=...) at backup.c:845
#9 0x00005587c59f9310 in send_data (stream=<optimized out>, bctx=...)
at backup.c:655
#10 save_file (jcr=0x7f5d6c000b88, ff_pkt=0x7f5d6c001898,
top_level=<optimized out>) at backup.c:502
#11 0x00007f5d793d418d in find_one_file (jcr=0x7f5d6c000b88,
ff_pkt=0x7f5d6c001898, handle_file=0x7f5d793d2ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=2050,
top_level=<optimized out>) at find_one.c:542
#12 0x00007f5d793d4d3c in find_one_file (jcr=0x7f5d6c000b88,
ff_pkt=0x7f5d6c001898, handle_file=0x7f5d793d2ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#13 0x00007f5d793d4d3c in find_one_file (jcr=0x7f5d6c000b88,
ff_pkt=0x7f5d6c001898, handle_file=0x7f5d793d2ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#14 0x00007f5d793d4d3c in find_one_file (jcr=jcr@entry=0x7f5d6c000b88,
ff_pkt=ff_pkt@entry=0x7f5d6c001898,
handle_file=handle_file@entry=0x7f5d793d2ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=fname@entry=0x7f5d6c002c48 "/",
parent_device=parent_device@entry=18446744073709551615,
top_level=top_level@entry=true) at find_one.c:768
#15 0x00007f5d793d1d4f in find_files (jcr=jcr@entry=0x7f5d6c000b88,
ff=0x7f5d6c001898, file_save=file_save@entry=0x5587c59f8530
<save_file(JCR*, FF_PKT*, bool)>, plugin_save=0x5587c59fd4b0
<plugin_save(JCR*, FF_PKT*, bool)>) at find.c:186
#16 0x00005587c59f6bc6 in blast_data_to_storage_daemon
(jcr=jcr@entry=0x7f5d6c000b88, addr=addr@entry=0x0) at backup.c:166
#17 0x00005587c5a075d1 in backup_cmd (jcr=0x7f5d6c000b88) at job.c:2517
#18 0x00005587c5a08836 in handle_director_request (dir=0x5587c7671ff8)
at job.c:344
#19 handle_connection_request (caller=0x5587c7671ff8) at job.c:504
#20 0x00007f5d793955b5 in workq_server (arg=0x5587c5a2ebe0 <dir_workq>)
at workq.c:372
#21 0x00007f5d79328ea7 in start_thread () from /lib64/libpthread.so.0
#22 0x00007f5d78fc3c6f in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f5d77f35700 (LWP 5144)):
#0 0x00007f5d7932f878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007f5d79394be9 in watchdog_thread (arg=<optimized out>) at
watchdog.c:299
#2 0x00007f5d79328ea7 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f5d78fc3c6f in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f5d78a03740 (LWP 5131)):
#0 0x00007f5d78fbb9d3 in select () from /lib64/libc.so.6
#1 0x00007f5d7935fa38 in bnet_thread_server (addrs=0x5587c766e4b8,
max_clients=20, client_wq=0x5587c5a2ebe0 <dir_workq>,
handle_client_request=0x5587c5a07dd0 <handle_connection_request(void*)>)
at bnet_server.c:166
#2 0x00005587c59f5760 in main (argc=<optimized out>, argv=<optimized
out>) at filed.c:277
Client babylon5:
Thread 4 (Thread 0x7fa2f5484700 (LWP 11770)):
#0 0x00007fa2f7d44123 in select () from /lib64/libc.so.6
#1 0x00007fa2f80f3242 in fd_wait_data (fd=6, mode=<optimized out>,
mode@entry=WAIT_READ, sec=sec@entry=5, msec=msec@entry=0) at bsys.c:1206
#2 0x00007fa2f811cb5b in BSOCKCORE::wait_data_intr
(this=this@entry=0x7fa2ec000b88, sec=sec@entry=5, msec=msec@entry=0) at
bsockcore.c:875
#3 0x00005604f5d31e98 in sd_heartbeat_thread (arg=0x7fa2e8000b88) at
heartbeat.c:69
#4 0x00007fa2f80bd057 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fa2f7d4c6cf in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fa2f5ccc700 (LWP 11769)):
#0 0x00007fa2f80c74ef in write () from /lib64/libpthread.so.0
#1 0x00007fa2f811d29f in BSOCKCORE::socketWrite (this=0x7fa2e80038b8,
len=29826, buf=0x7fa2e808d78c, fd=<optimized out>) at ../lib/bsockcore.h:203
#2 BSOCKCORE::write_nbytes (nbytes=<optimized out>, ptr=0x7fa2e808d78c
"@", this=<optimized out>) at bsockcore.c:1079
#3 BSOCKCORE::write_nbytes (this=this@entry=0x7fa2e80038b8,
ptr=<optimized out>, nbytes=29826) at bsockcore.c:1064
#4 0x00007fa2f80f7878 in BSOCK::write_nbytes (this=0x7fa2e80038b8,
ptr=<optimized out>, nbytes=29826) at bsock.c:831
#5 0x00007fa2f80f677a in BSOCK::send (aflags=0, this=0x7fa2e80038b8) at
bsock.c:368
#6 BSOCK::send (this=this@entry=0x7fa2e80038b8, aflags=aflags@entry=0)
at bsock.c:249
#7 0x00005604f5d270d1 in BSOCK::send (this=0x7fa2e80038b8) at
../lib/bsock.h:75
#8 process_and_send_data (bctx=...) at backup.c:845
#9 0x00005604f5d292d0 in send_data (stream=<optimized out>, bctx=...)
at backup.c:655
#10 save_file (jcr=0x7fa2e8000b88, ff_pkt=0x7fa2e80015f8,
top_level=<optimized out>) at backup.c:502
#11 0x00007fa2f816b1e8 in find_one_file (jcr=<optimized out>,
ff_pkt=0x7fa2e80015f8, handle_file=0x7fa2f8169b00 <our_callback(JCR*,
FF_PKT*, bool)>, fname=0x7fa2e8005be8 "/home/alaric/.moonchild
productions/pale moon/alaric/adblockplus/patterns-backup5.ini",
parent_device=2304, top_level=<optimized out>) at find_one.c:542
#12 0x00007fa2f816bdf3 in find_one_file (jcr=<optimized out>,
ff_pkt=0x7fa2e80015f8, handle_file=<optimized out>, fname=0x7fa2e80b61c8
"/home/alaric/.moonchild productions/pale moon/alaric/adblockplus",
parent_device=<optimized out>, top_level=<optimized out>) at find_one.c:768
#13 0x00007fa2f816bdf3 in find_one_file (jcr=<optimized out>,
ff_pkt=0x7fa2e80015f8, handle_file=<optimized out>, fname=0x7fa2e80056a8
"/home/alaric/.moonchild productions/pale moon/alaric",
parent_device=<optimized out>, top_level=<optimized out>) at find_one.c:768
#14 0x00007fa2f816bdf3 in find_one_file (jcr=<optimized out>,
ff_pkt=0x7fa2e80015f8, handle_file=<optimized out>, fname=0x7fa2e808d088
"/home/alaric/.moonchild productions/pale moon",
parent_device=<optimized out>, top_level=<optimized out>) at find_one.c:768
#15 0x00007fa2f816bdf3 in find_one_file (jcr=<optimized out>,
ff_pkt=0x7fa2e80015f8, handle_file=<optimized out>, fname=0x7fa2e807bfc8
"/home/alaric/.moonchild productions", parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#16 0x00007fa2f816bdf3 in find_one_file (jcr=<optimized out>,
ff_pkt=0x7fa2e80015f8, handle_file=<optimized out>, fname=0x7fa2e8073748
"/home/alaric", parent_device=<optimized out>, top_level=<optimized
out>) at find_one.c:768
#17 0x00007fa2f816bdf3 in find_one_file (jcr=<optimized out>,
ff_pkt=0x7fa2e80015f8, handle_file=<optimized out>, fname=0x7fa2e806af08
"/home", parent_device=<optimized out>, top_level=<optimized out>) at
find_one.c:768
#18 0x00007fa2f816bdf3 in find_one_file (jcr=jcr@entry=0x7fa2e8000b88,
ff_pkt=ff_pkt@entry=0x7fa2e80015f8,
handle_file=handle_file@entry=0x7fa2f8169b00 <our_callback(JCR*,
FF_PKT*, bool)>, fname=fname@entry=0x7fa2e80029a8 "/",
parent_device=parent_device@entry=18446744073709551615,
top_level=top_level@entry=true) at find_one.c:768
#19 0x00007fa2f8168db7 in find_files (jcr=jcr@entry=0x7fa2e8000b88,
ff=0x7fa2e80015f8, file_save=file_save@entry=0x5604f5d28500
<save_file(JCR*, FF_PKT*, bool)>, plugin_save=0x5604f5d2d440
<plugin_save(JCR*, FF_PKT*, bool)>) at find.c:186
#20 0x00005604f5d26c06 in blast_data_to_storage_daemon
(jcr=jcr@entry=0x7fa2e8000b88, addr=addr@entry=0x0) at backup.c:166
#21 0x00005604f5d3741f in backup_cmd (jcr=0x7fa2e8000b88) at job.c:2517
#22 0x00005604f5d38680 in handle_director_request (dir=0x5604f68e7538)
at job.c:344
#23 handle_connection_request (caller=0x5604f68e7538) at job.c:504
#24 0x00007fa2f812c12c in workq_server (arg=0x5604f5d5eca0 <dir_workq>)
at workq.c:372
#25 0x00007fa2f80bd057 in start_thread () from /lib64/libpthread.so.0
#26 0x00007fa2f7d4c6cf in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fa2f6cce700 (LWP 4446)):
#0 0x00007fa2f80c3d08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fa2f812b750 in watchdog_thread (arg=<optimized out>) at
watchdog.c:299
#2 0x00007fa2f80bd057 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fa2f7d4c6cf in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa2f7777740 (LWP 4438)):
#0 0x00007fa2f7d44123 in select () from /lib64/libc.so.6
#1 0x00007fa2f80f4c60 in bnet_thread_server (addrs=0x5604f68de5b8,
max_clients=20, client_wq=0x5604f5d5eca0 <dir_workq>,
handle_client_request=0x5604f5d37c20 <handle_connection_request(void*)>)
at bnet_server.c:166
#2 0x00005604f5d25777 in main (argc=<optimized out>, argv=<optimized
out>) at filed.c:277
Client narn:
Thread 4 (Thread 0x7f3ff6ccb700 (LWP 21335)):
#0 0x00007f3ff85509d3 in select () from /lib64/libc.so.6
#1 0x00007f3ff88f310a in fd_wait_data (fd=6, mode=mode@entry=WAIT_READ,
sec=sec@entry=5, msec=msec@entry=0) at bsys.c:1206
#2 0x00007f3ff891b6cb in BSOCKCORE::wait_data_intr
(this=this@entry=0x7f3ff0002168, sec=sec@entry=5, msec=msec@entry=0) at
bsockcore.c:875
#3 0x00005622dc098118 in sd_heartbeat_thread (arg=0x7f3fec0053a8) at
heartbeat.c:69
#4 0x00007f3ff88bdea7 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f3ff8558c6f in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f3ff5c88700 (LWP 21333)):
#0 0x00007f3ff88c7d5f in write () from /lib64/libpthread.so.0
#1 0x00007f3ff891bdaf in BSOCKCORE::socketWrite (this=0x7f3fec0073d8,
len=142, buf=0x7f3fec007a8c, fd=<optimized out>) at ../lib/bsockcore.h:203
#2 BSOCKCORE::write_nbytes (nbytes=<optimized out>, ptr=0x7f3fec007a8c
"", this=<optimized out>) at bsockcore.c:1079
#3 BSOCKCORE::write_nbytes (this=this@entry=0x7f3fec0073d8,
ptr=<optimized out>, nbytes=142) at bsockcore.c:1064
#4 0x00007f3ff88f7578 in BSOCK::write_nbytes (this=0x7f3fec0073d8,
ptr=<optimized out>, nbytes=142) at bsock.c:831
#5 0x00007f3ff88f651b in BSOCK::send (aflags=<optimized out>,
this=0x7f3fec0073d8) at bsock.c:368
#6 BSOCK::send (this=0x7f3fec0073d8, aflags=<optimized out>) at bsock.c:249
#7 0x00007f3ff891a470 in BSOCKCORE::fsend (this=0x7f3fec0073d8,
fmt=0x5622dc0b4721 "%ld %d %s%c%s%c%c%s%c%d%c") at bsockcore.c:584
#8 0x00005622dc08e021 in encode_and_send_attributes (bctx=...) at
backup.c:1020
#9 0x00005622dc08e73b in save_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, top_level=<optimized out>) at backup.c:432
#10 0x00007f3ff896918d in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=2050,
top_level=<optimized out>) at find_one.c:542
#11 0x00007f3ff8969d3c in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#12 0x00007f3ff8969d3c in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#13 0x00007f3ff8969d3c in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#14 0x00007f3ff8969d3c in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#15 0x00007f3ff8969d3c in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#16 0x00007f3ff8969d3c in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#17 0x00007f3ff8969d3c in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#18 0x00007f3ff8969d3c in find_one_file (jcr=0x7f3fec0053a8,
ff_pkt=0x7f3fec005a28, handle_file=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=<optimized out>, parent_device=<optimized out>,
top_level=<optimized out>) at find_one.c:768
#19 0x00007f3ff8969d3c in find_one_file (jcr=jcr@entry=0x7f3fec0053a8,
ff_pkt=ff_pkt@entry=0x7f3fec005a28,
handle_file=handle_file@entry=0x7f3ff8967ab0 <our_callback(JCR*,
FF_PKT*, bool)>, fname=fname@entry=0x7f3fec0064c8 "/",
parent_device=parent_device@entry=18446744073709551615,
top_level=top_level@entry=true) at find_one.c:768
#20 0x00007f3ff8966d4f in find_files (jcr=jcr@entry=0x7f3fec0053a8,
ff=0x7f3fec005a28, file_save=file_save@entry=0x5622dc08e530
<save_file(JCR*, FF_PKT*, bool)>, plugin_save=0x5622dc0934b0
<plugin_save(JCR*, FF_PKT*, bool)>) at find.c:186
#21 0x00005622dc08cbc6 in blast_data_to_storage_daemon
(jcr=jcr@entry=0x7f3fec0053a8, addr=addr@entry=0x0) at backup.c:166
#22 0x00005622dc09d5d1 in backup_cmd (jcr=0x7f3fec0053a8) at job.c:2517
#23 0x00005622dc09e836 in handle_director_request (dir=0x5622dd9b93f8)
at job.c:344
#24 handle_connection_request (caller=0x5622dd9b93f8) at job.c:504
#25 0x00007f3ff892a5b5 in workq_server (arg=0x5622dc0c4be0 <dir_workq>)
at workq.c:372
#26 0x00007f3ff88bdea7 in start_thread () from /lib64/libpthread.so.0
#27 0x00007f3ff8558c6f in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f3ff74cc700 (LWP 1473)):
#0 0x00007f3ff88c4878 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007f3ff8929be9 in watchdog_thread (arg=<optimized out>) at
watchdog.c:299
#2 0x00007f3ff88bdea7 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f3ff8558c6f in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f3ff7f98740 (LWP 1466)):
#0 0x00007f3ff85509d3 in select () from /lib64/libc.so.6
#1 0x00007f3ff88f4a38 in bnet_thread_server (addrs=0x5622dd9b54b8,
max_clients=20, client_wq=0x5622dc0c4be0 <dir_workq>,
handle_client_request=0x5622dc09ddd0 <handle_connection_request(void*)>)
at bnet_server.c:166
#2 0x00005622dc08b760 in main (argc=<optimized out>, argv=<optimized
out>) at filed.c:277
--
Phil Stracchino
Babylon Communications
ph...@caerllewys.net
p...@co.ordinate.org
Landline: +1.603.293.8485
Mobile: +1.603.998.6958
_______________________________________________
Bacula-devel mailing list
Bacula-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-devel
