Hello again,
I was watching the /tmp/amanda/client log file of the dump
of a single one of the problem partitions and I am including
the log file here.
These are the last 5 lines of the log, full log follows lower down.
1268937862.061246: sendbackup: Spawning "/usr/local/libexec/amanda/runtar
runtar NOCONFIG /usr/sfw/bin/gtar --create --file - --directory
/export/zones/sc1beaz1 --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/dorldom1_export_zones_sc1beaz1_0.new
--sparse --ignore-failed-read --totals --exclude-from
/tmp/amanda/sendbackup._export_zones_sc1beaz1.20100318144422.exclude ." in
pipeline
1268937862.062958: sendbackup: gnutar: /usr/local/libexec/amanda/runtar: pid
19380
1268937862.063654: sendbackup: Started backup
1268941918.662429: sendbackup: 47: size(|): Total bytes written:
21471805440 (20GiB, 5.1MiB/s)
1268941923.786954: sendbackup: Parsed backup messages
1268941923.787212: sendbackup: pid 19377 finish time Thu Mar 18 15:52:03 2010
I am uncertain as to why the tar completed, it wasn't really.
Or at least the server didn't see a proper handoff/rundown from
the client.
The server things the client dumped 92.31% of the partition
and was waiting.
It would have waited I believe dtimeout seconds ?
I killed the job as it had become clear it wasn't going to
finish and I wanted to make sure we had a tape loaded for
this evenings run.
I re-checked amanda.conf, we had ample work area, we had
chunksize set to 1gig on each work area partition, I do not
know why we failed to finish dumping, but it seems that the
client quit prematurely.
thank you,
Brian
# cat sendbackup.20100318144421.debug
1268937861.999648: sendbackup: pid 19377 ruid 110 euid 110 version 2.6.1p1:
start at Thu Mar 18 14:44:21 2010
1268937861.999948: sendbackup: rlimit_nofile: 65536 65536
1268937862.000023: sendbackup: rlimit_nofile: 65536 65536
1268937862.000064: sendbackup: version 2.6.1p1
1268937862.000094: sendbackup: build: VERSION="Amanda-2.6.1p1"
1268937862.000124: sendbackup: BUILT_DATE="Wed Jul 22 15:53:06 EDT
2009"
1268937862.000153: sendbackup: BUILT_MACH="sparc-sun-solaris2.10"
BUILT_REV="2059"
1268937862.000187: sendbackup: BUILT_BRANCH="amanda-261" CC="cc"
1268937862.000219: sendbackup: paths: bindir="/usr/local/bin"
sbindir="/usr/local/sbin"
1268937862.000252: sendbackup: libexecdir="/usr/local/libexec"
1268937862.000286: sendbackup:
amlibexecdir="/usr/local/libexec/amanda"
1268937862.000320: sendbackup: mandir="/usr/local/share/man"
AMANDA_TMPDIR="/tmp/amanda"
1268937862.000351: sendbackup: AMANDA_DBGDIR="/tmp/amanda"
1268937862.000380: sendbackup: CONFIG_DIR="/usr/local/etc/amanda"
DEV_PREFIX="/dev/dsk/"
1268937862.000409: sendbackup: RDEV_PREFIX="/dev/rdsk/"
DUMP="/usr/sbin/ufsdump"
1268937862.000438: sendbackup: RESTORE="/usr/sbin/ufsrestore"
VDUMP=UNDEF VRESTORE=UNDEF
1268937862.000467: sendbackup: XFSDUMP=UNDEF XFSRESTORE=UNDEF
VXDUMP=UNDEF VXRESTORE=UNDEF
1268937862.000496: sendbackup: SAMBA_CLIENT="/usr/sfw/bin/smbclient"
1268937862.000525: sendbackup: GNUTAR="/usr/sfw/bin/gtar"
COMPRESS_PATH="/bin/gzip"
1268937862.000554: sendbackup: UNCOMPRESS_PATH="/bin/gzip"
LPRCMD="/bin/lpr"
1268937862.000583: sendbackup: MAILER=UNDEF
1268937862.000611: sendbackup:
listed_incr_dir="/usr/local/var/amanda/gnutar-lists"
1268937862.000639: sendbackup: defs: DEFAULT_SERVER="lyra"
DEFAULT_CONFIG="DailySet1"
1268937862.000668: sendbackup: DEFAULT_TAPE_SERVER="lyra"
DEFAULT_TAPE_DEVICE=""
1268937862.000697: sendbackup: HAVE_MMAP NEED_STRSTR HAVE_SYSVSHM
AMFLOCK_POSIX AMFLOCK_LOCKF
1268937862.000726: sendbackup: AMFLOCK_LNLOCK SETPGRP_VOID
AMANDA_DEBUG_DAYS=4 BSD_SECURITY
1268937862.000755: sendbackup: USE_AMANDAHOSTS CLIENT_LOGIN="amanda"
CHECK_USERID HAVE_GZIP
1268937862.000783: sendbackup: COMPRESS_SUFFIX=".gz"
COMPRESS_FAST_OPT="--fast"
1268937862.000812: sendbackup: COMPRESS_BEST_OPT="--best"
UNCOMPRESS_OPT="-dc"
1268937862.000910: sendbackup: Version 2.6.1p1
1268937862.025706: sendbackup: sendbackup req: <GNUTAR /export/zones/sc1beaz1
0 1970:1:1:0:0:0 OPTIONS
|;auth=bsd;compress-fast;exclude-list=/export/home/amanda/exclude-list;>
1268937862.026102: sendbackup: Parsed request as: program `GNUTAR'
1268937862.026139: sendbackup: disk
`/export/zones/sc1beaz1'
1268937862.026170: sendbackup: device
`/export/zones/sc1beaz1'
1268937862.026202: sendbackup: level 0
1268937862.026231: sendbackup: since 1970:1:1:0:0:0
1268937862.026260: sendbackup: options
`|;auth=bsd;compress-fast;exclude-list=/export/home/amanda/exclude-list;'
1268937862.026967: sendbackup: start: dorldom1:/export/zones/sc1beaz1 lev 0
1268937862.027175: sendbackup: pipespawnv: stdoutfd is 50
1268937862.027302: sendbackup: Spawning "/bin/gzip /bin/gzip --fast" in pipeline
1268937862.029562: sendbackup: gnutar: pid 19379: /bin/gzip1268937862.030112:
sendbackup: pid 19379: /bin/gzip --fast
1268937862.054751: sendbackup: doing level 0 dump as listed-incremental to
'/usr/local/var/amanda/gnutar-lists/dorldom1_export_zones_sc1beaz1_0.new'
1268937862.060543: sendbackup: pipespawnv: stdoutfd is 6
1268937862.061246: sendbackup: Spawning "/usr/local/libexec/amanda/runtar
runtar NOCONFIG /usr/sfw/bin/gtar --create --file - --directory
/export/zones/sc1beaz1 --one-file-system --listed-incremental
/usr/local/var/amanda/gnutar-lists/dorldom1_export_zones_sc1beaz1_0.new
--sparse --ignore-failed-read --totals --exclude-from
/tmp/amanda/sendbackup._export_zones_sc1beaz1.20100318144422.exclude ." in
pipeline
1268937862.062958: sendbackup: gnutar: /usr/local/libexec/amanda/runtar: pid
19380
1268937862.063654: sendbackup: Started backup
1268941918.662429: sendbackup: 47: size(|): Total bytes written:
21471805440 (20GiB, 5.1MiB/s)
1268941923.786954: sendbackup: Parsed backup messages
1268941923.787212: sendbackup: pid 19377 finish time Thu Mar 18 15:52:03 2010
On Thu, Mar 18, 2010 at 02:22:50PM -0400, Brian Cuttler wrote:
>
> Hi Amanda users,
>
> I have gotten rid of most of the file changes message by
> using exclude lists, I'm a little baffled that /devices
> or "." change but this may be a side effect of running an
> amanda client in an "ldom" (Logical Domain) and then running
> zones of top of it (Solaris 10x86 base system).
>
> I am seeing "Connection timed out" messages... before this
> most recent run I increased "dtimeout" up to 3600 (default
> is 1200, I had previously set it to 2400).
>
> The sc1beaz1 structure was hit and miss, the dorldom1z1 was
> recently added back into the disklist (zone for oracle that
> was in testing for an extended period, we are moving towards
> production).
>
> Happy and able to include log files but needed to know what
> was needed, client or server side, of if there was another
> config switch I should try first.
>
> Server is a FW system running Solaris 9 and amanda 2.4.4
> this client is Solaris 10x86 with amanda 2.6.1p1.
>
> The dumps certainly start, they just done finish...
>
> thank you,
>
> Brian
>
> ----- Forwarded message from Amanda on Gat0 <[email protected]> -----
>
> These dumps were to tape MIMOSA18.
> The next tape Amanda expects to use is: MIMOSA19.
>
> FAILURE AND STRANGE DUMP SUMMARY:
> dorldom1 / lev 0 STRANGE
> dorldom1 /export/zones/sc1beaz1 lev 0 FAILED [mesg read: Connection timed
> out]
> dorldom1 /export/zones/dorldom1z1 lev 0 FAILED [mesg read: Connection
> timed out]
>
>
> STATISTICS:
> Total Full Daily
> -------- -------- --------
> Estimate Time (hrs:min) 1:18
> Run Time (hrs:min) 4:26
> Dump Time (hrs:min) 1:01 0:57 0:04
> Output Size (meg) 5532.8 5528.9 3.9
> Original Size (meg) 10895.3 10809.6 85.7
> Avg Compressed Size (%) 50.8 51.1 4.5 (level:#disks ...)
> Filesystems Dumped 5 1 4 (1:4)
> Avg Dump Rate (k/s) 1545.9 1660.2 15.5
>
> Tape Time (hrs:min) 0:26 0:26 0:00
> Tape Size (meg) 5532.9 5528.9 4.0
> Tape Used (%) 5.1 5.0 0.0 (level:#disks ...)
> Filesystems Taped 5 1 4 (1:4)
> Avg Tp Write Rate (k/s) 3606.6 3615.9 789.3
>
> USAGE BY TAPE:
> Label Time Size % Nb
> MIMOSA18 0:26 5532.9 5.1 5
>
>
> FAILED AND STRANGE DUMP DETAILS:
>
> /-- dorldom1 / lev 0 STRANGE
> sendbackup: start [dorldom1:/ level 0]
> sendbackup: info BACKUP=/usr/sfw/bin/gtar
> sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sfw/bin/gtar -xpGf - ...
> sendbackup: info COMPRESS_SUFFIX=.gz
> sendbackup: info end
> ? /usr/sfw/bin/gtar: .: file changed as we read it
> ? /usr/sfw/bin/gtar:
> ./usr/local/var/amanda/gnutar-lists/dorldom1_export_zones_sc1beaz3_1.new:
> Warning: Cannot stat: No such file or directory
> | Total bytes written: 11334727680 (11GiB, 3.2MiB/s)
> sendbackup: size 11069070
> sendbackup: end
> \--------
>
> /-- dorldom1 /export/zones/sc1beaz1 lev 0 FAILED [mesg read: Connection
> timed out]
> sendbackup: start [dorldom1:/export/zones/sc1beaz1 level 0]
> sendbackup: info BACKUP=/usr/sfw/bin/gtar
> sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sfw/bin/gtar -xpGf - ...
> sendbackup: info COMPRESS_SUFFIX=.gz
> sendbackup: info end
> \--------
>
> /-- dorldom1 /export/zones/dorldom1z1 lev 0 FAILED [mesg read: Connection
> timed out]
> sendbackup: start [dorldom1:/export/zones/dorldom1z1 level 0]
> sendbackup: info BACKUP=/usr/sfw/bin/gtar
> sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sfw/bin/gtar -xpGf - ...
> sendbackup: info COMPRESS_SUFFIX=.gz
> sendbackup: info end
> \--------
>
>
> NOTES:
> planner: Forcing full dump of dorldom1:/export/zones/dorldom1z1 as directed.
> driver: WARNING: /amanda/work: 57344000 KB requested, but only 57092022 KB
> available.
> taper: tape MIMOSA18 kb 5665824 fm 5 [OK]
>
>
> DUMP SUMMARY:
> DUMPER STATS TAPER STATS
> HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS
> KB/s
> ------------------------ --------------------------------------- -------------
> dorldom / 0 11069070 5661600 51.1 56:50 1660.2 26:06 3615.9
> dorldom /export/home 1 10 32 320.0 0:05 0.0 0:01 25.5
> dorldom -es/dorldom1z1 0 FAILED ----------------------------------------------
> dorldom -ones/sc1beaz1 0 FAILED ----------------------------------------------
> dorldom -ones/sc1beaz2 1 30340 1728 5.7 1:37 17.5 0:01 1227.0
> dorldom -ones/sc1beaz3 1 57390 2272 4.0 2:27 15.3 0:02 1464.7
> dorldom /rpool 1 10 32 320.0 0:05 0.0 0:01 34.1
>
> (brought to you by Amanda version 2.4.4)
>
> ----- End forwarded message -----
> ---
> Brian R Cuttler [email protected]
> Computer Systems Support (v) 518 486-1697
> Wadsworth Center (f) 518 473-6384
> NYS Department of Health Help Desk 518 473-0773
>
>
>
> IMPORTANT NOTICE: This e-mail and any attachments may contain
> confidential or sensitive information which is, or may be, legally
> privileged or otherwise protected by law from further disclosure. It
> is intended only for the addressee. If you received this in error or
> from someone who was not authorized to send it to you, please do not
> distribute, copy or use it or any attachments. Please notify the
> sender immediately by reply e-mail and delete this from your
> system. Thank you for your cooperation.
>
>
---
Brian R Cuttler [email protected]
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773
IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.
