First note that regardless of the timing for any particular DLE, *every* single DLE for this one
host is failing, while all other hosts are getting fully backed up without any trouble.
On the host that is failing, there are no senbackup debug files since it
started failing.
On that same host, the sendsize debug file from last night includes (this is
just a segment):
"/tmp/amanda/client/daily/sendsize.20150120233005.debug" 1165 lines, 145734
characters
Tue Jan 20 23:30:05 2015: thd-32a58: sendsize: pid 11633 ruid 555 euid 555 version 3.3.2: start at
Tue Jan 20 23:30:05 2015
Tue Jan 20 23:30:05 2015: thd-32a58: sendsize: version 3.3.2
Tue Jan 20 23:30:05 2015: thd-32a58: sendsize: pid 11633 ruid 555 euid 555 version 3.3.2: rename at
Tue Jan 20 23:30:05 2015
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: waiting for any estimate child:
2 running
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: calculating for amname /,
dirname /, spindle 100 DUMP
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: getting size via dump for /
level 0
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: calculating for amname /export/baja, dirname
/export/baja, spindle 45010 GNUTAR
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: getting size via gnutar for
/export/baja level 0
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: calculating for device
/dev/rdsk/c1t0d0s0 with ufs
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: running "/usr/local/etc/amanda/tools/ufsdump 0Ssf
1048576 - /dev/rdsk/c1t0d0s0"
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: running
/usr/local/libexec/amanda/killpgrp
Tue Jan 20 23:30:08 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /export/baja --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_export_baja_0.new --sparse --ignore-failed-read
--totals ." in pipeline
Tue Jan 20 23:30:19 2015: thd-32a58: sendsize: 13847108608
Tue Jan 20 23:30:19 2015: thd-32a58: sendsize: .....
Tue Jan 20 23:30:19 2015: thd-32a58: sendsize: estimate time for / level 0:
11.540
Tue Jan 20 23:30:19 2015: thd-32a58: sendsize: estimate size for / level 0:
13522567 KB
Tue Jan 20 23:30:19 2015: thd-32a58: sendsize: asking killpgrp to terminate
Tue Jan 20 23:30:20 2015: thd-32a58: sendsize: getting size via dump for /
level 1
Tue Jan 20 23:30:20 2015: thd-32a58: sendsize: calculating for device
/dev/rdsk/c1t0d0s0 with ufs
Tue Jan 20 23:30:20 2015: thd-32a58: sendsize: running "/usr/local/etc/amanda/tools/ufsdump 1Ssf
1048576 - /dev/rdsk/c1t0d0s0"
Tue Jan 20 23:30:20 2015: thd-32a58: sendsize: running
/usr/local/libexec/amanda/killpgrp
Tue Jan 20 23:32:33 2015: thd-32a58: sendsize: 1461860352
Tue Jan 20 23:32:33 2015: thd-32a58: sendsize: .....
Tue Jan 20 23:32:33 2015: thd-32a58: sendsize: estimate time for / level 1:
133.065
Tue Jan 20 23:32:33 2015: thd-32a58: sendsize: estimate size for / level 1:
1427598 KB
Tue Jan 20 23:32:33 2015: thd-32a58: sendsize: asking killpgrp to terminate
Tue Jan 20 23:32:34 2015: thd-32a58: sendsize: done with amname / dirname /
spindle 100
Tue Jan 20 23:32:34 2015: thd-32a58: sendsize: waiting for any estimate child:
2 running
Tue Jan 20 23:32:34 2015: thd-32a58: sendsize: calculating for amname /archive, dirname /archive,
spindle 100 GNUTAR
Tue Jan 20 23:32:34 2015: thd-32a58: sendsize: getting size via gnutar for
/archive level 0
Tue Jan 20 23:32:34 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /archive --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_archive_0.new --sparse --ignore-failed-read
--totals
." in pipeline
Tue Jan 20 23:32:58 2015: thd-32a58: sendsize: Total bytes written: 10917795840
(11GiB, 63MiB/s)
Tue Jan 20 23:32:58 2015: thd-32a58: sendsize: .....
Tue Jan 20 23:32:58 2015: thd-32a58: sendsize: estimate time for /export/baja
level 0: 169.954
Tue Jan 20 23:32:58 2015: thd-32a58: sendsize: estimate size for /export/baja
level 0: 10661910 KB
Tue Jan 20 23:32:58 2015: thd-32a58: sendsize: waiting for runtar
"/export/baja" child
Tue Jan 20 23:32:58 2015: thd-32a58: sendsize: after runtar /export/baja wait
Tue Jan 20 23:32:58 2015: thd-32a58: sendsize: getting size via gnutar for
/export/baja level 1
Tue Jan 20 23:32:58 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /export/baja --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_export_baja_1.new --sparse --ignore-failed-read
--totals ." in pipeline
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: Total bytes written: 133120
(130KiB, 39KiB/s)
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: .....
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: estimate time for /export/baja
level 1: 3.525
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: estimate size for /export/baja
level 1: 130 KB
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: waiting for runtar
"/export/baja" child
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: after runtar /export/baja wait
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: done with amname /export/baja dirname /export/baja
spindle 45010
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: waiting for any estimate child:
2 running
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: calculating for amname /export/barbados, dirname
/export/barbados, spindle 45010 GNUTAR
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: getting size via gnutar for
/export/barbados level 0
Tue Jan 20 23:33:01 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /export/barbados --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_export_barbados_0.new --sparse --ignore-fail
ed-read --totals ." in pipeline
Tue Jan 20 23:36:44 2015: thd-32a58: sendsize: Total bytes written: 11691386880
(11GiB, 51MiB/s)
Tue Jan 20 23:36:44 2015: thd-32a58: sendsize: .....
Tue Jan 20 23:36:44 2015: thd-32a58: sendsize: estimate time for
/export/barbados level 0: 222.591
Tue Jan 20 23:36:44 2015: thd-32a58: sendsize: estimate size for
/export/barbados level 0: 11417370 KB
Tue Jan 20 23:36:44 2015: thd-32a58: sendsize: waiting for runtar
"/export/barbados" child
Tue Jan 20 23:36:44 2015: thd-32a58: sendsize: after runtar /export/barbados
wait
Tue Jan 20 23:36:44 2015: thd-32a58: sendsize: getting size via gnutar for
/export/barbados level 1
Tue Jan 20 23:36:44 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /export/barbados --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_export_barbados_1.new --sparse --ignore-fail
ed-read --totals ." in pipeline
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: Total bytes written: 153600
(150KiB, 125KiB/s)
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: .....
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: estimate time for
/export/barbados level 1: 1.378
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: estimate size for
/export/barbados level 1: 150 KB
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: waiting for runtar
"/export/barbados" child
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: after runtar /export/barbados
wait
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: done with amname /export/barbados dirname
/export/barbados spindle 45010
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: waiting for any estimate child:
2 running
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: calculating for amname /export/bermuda, dirname
/export/bermuda, spindle 45010 GNUTAR
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: getting size via gnutar for
/export/bermuda level 0
Tue Jan 20 23:36:45 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /export/bermuda --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_export_bermuda_0.new --sparse --ignore-failed
-read --totals ." in pipeline
SKIPPING A THOUSAND OR SO LINES . . . .
Wed Jan 21 04:40:50 2015: thd-32a58: sendsize: .....
Wed Jan 21 04:40:50 2015: thd-32a58: sendsize: estimate time for
/u1/home/micro/./k level 0: 3245.840
Wed Jan 21 04:40:50 2015: thd-32a58: sendsize: estimate size for
/u1/home/micro/./k level 0: 45555590 KB
Wed Jan 21 04:40:50 2015: thd-32a58: sendsize: waiting for runtar
"/u1/home/micro/./k" child
Wed Jan 21 04:40:50 2015: thd-32a58: sendsize: after runtar /u1/home/micro/./k
wait
Wed Jan 21 04:40:50 2015: thd-32a58: sendsize: getting size via gnutar for
/u1/home/micro/./k level 1
Wed Jan 21 04:40:51 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /u1/home/micro --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_u1_home_micro_._k_1.new --sparse --ignore-fail
ed-read --totals --files-from /tmp/amanda/sendsize._u1_home_micro_._k.20150121044050.include" in
pipeline
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: Total bytes written: 6410967040
(6.0GiB, 9.6MiB/s)
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: .....
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: estimate time for
/u1/home/micro/./k level 1: 640.501
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: estimate size for
/u1/home/micro/./k level 1: 6260710 KB
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: waiting for runtar
"/u1/home/micro/./k" child
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: after runtar /u1/home/micro/./k
wait
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: done with amname /u1/home/micro/./k dirname
/u1/home/micro spindle 506
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: waiting for any estimate child:
1 running
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: calculating for amname /u1/home/micro/./l-z, dirname
/u1/home/micro, spindle 506 GNUTAR
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: getting size via gnutar for
/u1/home/micro/./l-z level 0
Wed Jan 21 04:51:31 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /u1/home/micro --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_u1_home_micro_._l-z_0.new --sparse --ignore-fa
iled-read --totals --files-from /tmp/amanda/sendsize._u1_home_micro_._l-z.20150121045131.include" in
pipeline
Wed Jan 21 05:16:06 2015: thd-32a58: sendsize: Total bytes written: 18660341760
(18GiB, 13MiB/s)
Wed Jan 21 05:16:06 2015: thd-32a58: sendsize: .....
Wed Jan 21 05:16:06 2015: thd-32a58: sendsize: estimate time for
/u1/home/micro/./l-z level 0: 1475.354
Wed Jan 21 05:16:06 2015: thd-32a58: sendsize: estimate size for /u1/home/micro/./l-z level 0:
18222990 KB
Wed Jan 21 05:16:06 2015: thd-32a58: sendsize: waiting for runtar
"/u1/home/micro/./l-z" child
Wed Jan 21 05:16:06 2015: thd-32a58: sendsize: after runtar
/u1/home/micro/./l-z wait
Wed Jan 21 05:16:06 2015: thd-32a58: sendsize: getting size via gnutar for
/u1/home/micro/./l-z level 1
Wed Jan 21 05:16:06 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /u1/home/micro --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_u1_home_micro_._l-z_1.new --sparse --ignore-fa
iled-read --totals --files-from /tmp/amanda/sendsize._u1_home_micro_._l-z.20150121051606.include" in
pipeline
Wed Jan 21 05:29:09 2015: thd-32a58: sendsize: Total bytes written: 8945582080
(8.4GiB, 11MiB/s)
Wed Jan 21 05:29:09 2015: thd-32a58: sendsize: .....
Wed Jan 21 05:29:09 2015: thd-32a58: sendsize: estimate time for
/u1/home/micro/./l-z level 1: 782.213
Wed Jan 21 05:29:09 2015: thd-32a58: sendsize: estimate size for /u1/home/micro/./l-z level 1:
8735920 KB
Wed Jan 21 05:29:09 2015: thd-32a58: sendsize: waiting for runtar
"/u1/home/micro/./l-z" child
Wed Jan 21 05:29:09 2015: thd-32a58: sendsize: after runtar
/u1/home/micro/./l-z wait
Wed Jan 21 05:29:09 2015: thd-32a58: sendsize: getting size via gnutar for
/u1/home/micro/./l-z level 2
Wed Jan 21 05:29:09 2015: thd-32a58: sendsize: Spawning "/usr/local/libexec/amanda/runtar runtar
daily /usr/local/etc/amanda/tools/gtar --create --file /dev/null --numeric-
owner --directory /u1/home/micro --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/marlin.bio.mor.nsm_u1_home_micro_._l-z_2.new --sparse --ignore-fa
iled-read --totals --files-from /tmp/amanda/sendsize._u1_home_micro_._l-z.20150121052909.include" in
pipeline
Wed Jan 21 05:42:15 2015: thd-32a58: sendsize: Total bytes written: 8937635840
(8.4GiB, 11MiB/s)
Wed Jan 21 05:42:15 2015: thd-32a58: sendsize: .....
Wed Jan 21 05:42:15 2015: thd-32a58: sendsize: estimate time for
/u1/home/micro/./l-z level 2: 786.054
Wed Jan 21 05:42:15 2015: thd-32a58: sendsize: estimate size for /u1/home/micro/./l-z level 2:
8728160 KB
Wed Jan 21 05:42:15 2015: thd-32a58: sendsize: waiting for runtar
"/u1/home/micro/./l-z" child
Wed Jan 21 05:42:15 2015: thd-32a58: sendsize: after runtar
/u1/home/micro/./l-z wait
Wed Jan 21 05:42:15 2015: thd-32a58: sendsize: done with amname /u1/home/micro/./l-z dirname
/u1/home/micro spindle 506
AND THAT IS THE END OF THE SENDSIZE DEBUG FILE
On 1/21/15 3:02 PM, Jean-Louis Martineau wrote:
You get error at estimate or at backup time?
Look at the time in the sendsize and sendbackup debug file to find which one is
slow.
On 01/21/2015 02:40 PM, Chris Hoogendyk wrote:
Folks,
I have an Ubuntu 14.04 LTS system running Amanda 3.3.6 server backing up a Solaris 10 system with
Amanda 3.3.2.
I had it working and I was getting backups of that particular Solaris system. Then I suddenly
started getting the "timeout on reply pipe" on every single dle on that system, but not on any
other systems. There is also another virtually identical Solaris system (except with Amanda
2.5.1p3) that has continued getting backed up as well as a number of Ubuntu systems with various
versions of Ubuntu (10.04LTS, 12.04LTS, or 14.04LTS) and Amanda (either 2.5.1p3, 3.3.2, or 3.3.6).
If I run `amcheck -c daily`, I get 0 problems.
How do I troubleshoot this? Why would it have suddenly come up (last Friday) and then been
consistently non functional? (Whereas before it was consistently functional). I've poked through
the /tmp/amanda debug logs, but haven't been able to identify any errors that would tell me what
was wrong.
I should note that most of these servers are in the same two adjacent racks and have GigE
connections to the same switch.
The server that is not getting backed up at present is our main departmental server that is
running mail services, web, file shares, printing, anonymous ftp, mysql, etc. for a fairly active
department.
--
---------------
Chris Hoogendyk
-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 347 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst
<[email protected]>
---------------
Erdös 4
