Continuation of previous initiated thread. Server, Solaris 10/x86 amanda 2.6.1p1, I believe -20090227 Client, also 2.6.1p1, Solaris 10/Sparc.
Have reduced maxdumps to 3, client still hanging... [lyra] ~ 117> ps -ef | grep amanda amanda 9087 7436 0 - ? 0:00 <defunct> amanda 7644 7436 0 - ? 0:00 <defunct> amanda 8487 7436 0 - ? 0:13 <defunct> amanda 8304 7436 0 - ? 0:00 <defunct> amanda 7437 7436 0 - ? 0:04 <defunct> amanda 7438 7436 0 - ? 0:00 <defunct> amanda 8099 7436 0 - ? 0:00 <defunct> amanda 9086 7436 0 - ? 0:02 <defunct> amanda 7465 7436 0 - ? 0:00 <defunct> amanda 7436 15684 0 20:00:01 ? 16:56 /usr/local/libexec/amanda/amandad amanda 9521 7436 0 - ? 0:00 <defunct> brian 12593 12573 0 09:39:24 pts/3 0:00 grep amanda amanda 8488 7436 0 - ? 0:00 <defunct> amanda 11972 7436 0 - ? 0:00 <defunct> amanda 8303 7436 0 - ? 0:13 <defunct> amanda 10597 7436 0 - ? 0:00 <defunct> amanda 9520 7436 0 - ? 0:04 <defunct> Not sure what to try next... We are also seeing some partial dumps, is there a work around for that ? Is there a switch or something I can set in the dumptype to cause dump to continue, rather than failing when it doesn't get input ? Interestingly, frustraitingly, we are seeing a lot of open files, dispite using snapshots. I'd contended with my peers here that using snapshots doesn't quiess the partition, it mearly seems to capture open files in an open state, it does not in and of itself produce a more reliable backup in terms of backuping up static data. I'd really like to run down the application in order to get a good snapshot - or am I misinterpreting what I see ? The dump of curie:/ always seems to fail, and there is nothing special about it, ufsdump of the root of the server itself. We always seem to have the same broken pipe... As always, happy to attach logs, but the initial post was already long and the logs are unlikely to differ from ones I send a week or so ago. ----- Forwarded message from Amanda/Curie <[email protected]> ----- Hostname: curie Org : Curie Config : curie Date : July 15, 2009 These dumps were to tape Curie10. The next 2 tapes Amanda expects to use are: Curie11, Curie12. FAILURE DUMP SUMMARY: lyra / lev 1 FAILED [spawn /bin/gzip: dup2 out: Bad file number] lyra / lev 1 FAILED [spawn /bin/gzip: dup2 out: Bad file number] lyra /db3 lev 0 FAILED [spawn /bin/gzip: dup2 err: Bad file number] lyra /db3 lev 0 FAILED [too many dumper retry: "[request failed: timeout waiting for REP]"] curie / lev 0 FAILED [spawn /usr/bin/gzip: dup2 out: Bad file number] curie / lev 0 FAILED [spawn /usr/bin/gzip: dup2 out: Bad file number] mailserv /usr1 lev 1 FAILED [missing size line from sendbackup] mailserv /usr1 lev 1 FAILED [missing size line from sendbackup] mailserv /usr1 lev 1: partial taper: STRANGE DUMP SUMMARY: squidzone1 /sqcache1/var lev 1 STRANGE (see below) squidzone2 /sqcache2/var lev 1 STRANGE (see below) squidzone2 /sqcache2/squidguard lev 0 STRANGE (see below) STATISTICS: Total Full Incr. -------- -------- -------- Estimate Time (hrs:min) 1:38 Run Time (hrs:min) 10:03 Dump Time (hrs:min) 22:20 13:41 8:39 Output Size (meg) 345339.0 151458.9 193880.1 Original Size (meg) 447849.3 209115.9 238733.5 Avg Compressed Size (%) 70.1 58.7 78.0 (level:#disks ...) Filesystems Dumped 45 15 30 (1:30) Avg Dump Rate (k/s) 4399.0 3149.5 6374.6 Tape Time (hrs:min) 8:09 3:59 4:09 Tape Size (meg) 330212.9 151458.9 178754.0 Tape Used (%) 41.1 18.9 22.3 (level:#disks ...) Filesystems Taped 46 15 31 (1:31) (level:#chunks ...) Chunks Taped 46 15 31 (1:31) Avg Tp Write Rate (k/s) 11526.5 10793.1 12230.6 USAGE BY TAPE: Label Time Size % Nb Nc Curie10 8:09 330213M 41.1 46 46 FAILED DUMP DETAILS: /-- lyra / lev 1 FAILED [spawn /bin/gzip: dup2 out: Bad file number] sendbackup: start [lyra:/ level 1] sendbackup: info BACKUP=/usr/sbin/ufsdump sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sbin/ufsrestore -xpGf - ... sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end sendbackup: error [spawn /bin/gzip: dup2 out: Bad file number] | DUMP: Date of this level 1 dump: Wed Jul 15 20:20:39 2009 | DUMP: Date of last level 0 dump: Fri Jul 10 20:02:27 2009 | DUMP: Dumping /dev/rdsk/c0t0d0s4 (lyra:/) to standard output. | DUMP: Mapping (Pass I) [regular files] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Writing 32 Kilobyte records | DUMP: Estimated 3508630 blocks (1713.20MB) on 0.03 tapes. ? sendbackup: index tee cannot write [Broken pipe] ? DUMP: Write error 0 blocks into volume 1 ? DUMP: Write error on standard output | DUMP: Cannot recover | DUMP: The ENTIRE dump is aborted. ? dump (8307) /usr/sbin/ufsdump returned 3 ? index index returned 1 ? compress (8305) compress returned 1 sendbackup: error [dump (8307) /usr/sbin/ufsdump returned 3, compress (8305) compress returned 1] \-------- /-- lyra / lev 1 FAILED [spawn /bin/gzip: dup2 out: Bad file number] sendbackup: start [lyra:/ level 1] sendbackup: error [spawn /bin/gzip: dup2 out: Bad file number] sendbackup: info BACKUP=/usr/sbin/ufsdump sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sbin/ufsrestore -xpGf - ... sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end | DUMP: Date of this level 1 dump: Wed Jul 15 20:23:01 2009 | DUMP: Date of last level 0 dump: Fri Jul 10 20:02:27 2009 | DUMP: Dumping /dev/rdsk/c0t0d0s4 (lyra:/) to standard output. | DUMP: Mapping (Pass I) [regular files] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Writing 32 Kilobyte records | DUMP: Estimated 3508658 blocks (1713.21MB) on 0.03 tapes. ? sendbackup: index tee cannot write [Broken pipe] ? DUMP: Write error 0 blocks into volume 1 ? DUMP: Write error on standard output | DUMP: Cannot recover | DUMP: The ENTIRE dump is aborted. ? dump (8493) /usr/sbin/ufsdump returned 3 ? index index returned 1 ? compress (8491) compress returned 1 sendbackup: error [dump (8493) /usr/sbin/ufsdump returned 3, compress (8491) compress returned 1] \-------- /-- lyra /db3 lev 0 FAILED [spawn /bin/gzip: dup2 err: Bad file number] sendbackup: error [spawn /bin/gzip: dup2 err: Bad file number] \-------- /-- curie / lev 0 FAILED [spawn /usr/bin/gzip: dup2 out: Bad file number] sendbackup: start [curie:/ level 0] sendbackup: error [spawn /usr/bin/gzip: dup2 out: Bad file number] sendbackup: info BACKUP=/usr/sbin/ufsdump sendbackup: info RECOVER_CMD=/usr/bin/gzip -dc |/usr/sbin/ufsrestore -xpGf - ... sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end | DUMP: Date of this level 0 dump: Wed Jul 15 21:13:09 2009 | DUMP: Date of last level 0 dump: the epoch | DUMP: Dumping /dev/md/rdsk/d10 (curie:/) to standard output. | DUMP: Mapping (Pass I) [regular files] | DUMP: Mapping (Pass II) [directories] | DUMP: Writing 32 Kilobyte records | DUMP: Estimated 15915568 blocks (7771.27MB) on 0.12 tapes. ? sendbackup: index tee cannot write [Broken pipe] ? DUMP: Write error 0 blocks into volume 1 ? DUMP: Write error on standard output | DUMP: Cannot recover | DUMP: The ENTIRE dump is aborted. ? dump (5453) /usr/sbin/ufsdump returned 3 ? index index returned 1 ? compress (5451) compress returned 1 sendbackup: error [dump (5453) /usr/sbin/ufsdump returned 3, compress (5451) compress returned 1] \-------- /-- curie / lev 0 FAILED [spawn /usr/bin/gzip: dup2 out: Bad file number] sendbackup: start [curie:/ level 0] sendbackup: error [spawn /usr/bin/gzip: dup2 out: Bad file number] sendbackup: info BACKUP=/usr/sbin/ufsdump sendbackup: info RECOVER_CMD=/usr/bin/gzip -dc |/usr/sbin/ufsrestore -xpGf - ... sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end | DUMP: Date of this level 0 dump: Wed Jul 15 21:13:38 2009 | DUMP: Date of last level 0 dump: the epoch | DUMP: Dumping /dev/md/rdsk/d10 (curie:/) to standard output. | DUMP: Mapping (Pass I) [regular files] | DUMP: Mapping (Pass II) [directories] | DUMP: Writing 32 Kilobyte records | DUMP: Estimated 15915574 blocks (7771.28MB) on 0.12 tapes. ? sendbackup: index tee cannot write [Broken pipe] ? DUMP: Write error 0 blocks into volume 1 ? DUMP: Write error on standard output | DUMP: Cannot recover | DUMP: The ENTIRE dump is aborted. ? index index returned 1 ? compress (5468) compress returned 1 ? dump (5471) /usr/sbin/ufsdump returned 3 sendbackup: error [compress (5468) compress returned 1, dump (5471) /usr/sbin/ufsdump returned 3] \-------- /-- mailserv /usr1 lev 1 FAILED [missing size line from sendbackup] sendbackup: start [mailserv:/usr1 level 1] sendbackup: info BACKUP=/usr/sbin/ufsdump sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sbin/ufsrestore -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end | DUMP: Date of this level 1 dump: Wed Jul 15 20:07:47 2009 | DUMP: Date of last level 0 dump: Tue Apr 28 18:48:36 2009 | DUMP: Dumping /dev/rdsk/c3t0d0s6 (mailserv:/usr1) to standard output. | DUMP: Mapping (Pass I) [regular files] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Writing 32 Kilobyte records | DUMP: Estimated 58390164 blocks (28510.82MB) on 0.42 tapes. | DUMP: Dumping (Pass III) [directories] | DUMP: Dumping (Pass IV) [regular files] | DUMP: 7.70% done, finished in 1:59 | DUMP: 16.06% done, finished in 1:44 | DUMP: 25.73% done, finished in 1:26 | DUMP: 35.33% done, finished in 1:13 | DUMP: 44.51% done, finished in 1:02 | DUMP: 53.46% done, finished in 0:52 | DUMP: 64.81% done, finished in 0:38 | DUMP: 75.73% done, finished in 0:25 | DUMP: 87.42% done, finished in 0:12 | DUMP: Warning - block 1957714610 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 2363809498 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 3704413812 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 3436370122 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 1520493790 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 3268468446 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 2593299082 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 3368739956 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 1957087964 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 1956299474 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 3872579292 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 1521273572 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 3899951306 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 1519964902 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 3637697742 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 2833151128 is beyond the end of `/dev/rdsk/c3t0d0s6' <--- many lines removed ---> | DUMP: Warning - block 2694866122 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 2934736084 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 3672140910 is beyond the end of `/dev/rdsk/c3t0d0s6' ? DUMP: More than 32 block read errors from dump device `/dev/rdsk/c3t0d0s6' | DUMP: NEEDS ATTENTION: Do you want to attempt to continue? ("yes" or "no") DUMP: The ENTIRE dump is aborted. ??error [/usr/sbin/ufsdump returned 3] ? dumper: strange [missing size line from sendbackup] \-------- /-- mailserv /usr1 lev 1 FAILED [missing size line from sendbackup] sendbackup: start [mailserv:/usr1 level 1] sendbackup: info BACKUP=/usr/sbin/ufsdump sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sbin/ufsrestore -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end | DUMP: Date of this level 1 dump: Wed Jul 15 21:48:01 2009 | DUMP: Date of last level 0 dump: Tue Apr 28 18:48:36 2009 | DUMP: Dumping /dev/rdsk/c3t0d0s6 (mailserv:/usr1) to standard output. | DUMP: Mapping (Pass I) [regular files] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Mapping (Pass II) [directories] | DUMP: Writing 32 Kilobyte records | DUMP: Estimated 58437252 blocks (28533.81MB) on 0.42 tapes. | DUMP: Dumping (Pass III) [directories] | DUMP: Dumping (Pass IV) [regular files] | DUMP: 13.16% done, finished in 1:06 | DUMP: 24.46% done, finished in 1:01 | DUMP: 35.23% done, finished in 0:55 | DUMP: 44.94% done, finished in 0:49 | DUMP: 55.60% done, finished in 0:39 | DUMP: 66.37% done, finished in 0:30 | DUMP: 78.36% done, finished in 0:19 | DUMP: 88.62% done, finished in 0:10 | DUMP: Warning - block 2453935036 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 2453882400 is beyond the end of `/dev/rdsk/c3t0d0s6' | DUMP: Warning - block 2454043076 is beyond the end of `/dev/rdsk/c3t0d0s6' <--- many lines removed ---> | DUMP: Warning - block 3806913748 is beyond the end of `/dev/rdsk/c3t0d0s6' ? DUMP: More than 32 block read errors from dump device `/dev/rdsk/c3t0d0s6' | DUMP: NEEDS ATTENTION: Do you want to attempt to continue? ("yes" or "no") DUMP: The ENTIRE dump is aborted. ??error [/usr/sbin/ufsdump returned 3] ? dumper: strange [missing size line from sendbackup] \-------- STRANGE DUMP DETAILS: /-- squidzone1 /sqcache1/var lev 1 STRANGE sendbackup: start [squidzone1:/sqcache1/var level 1] sendbackup: info BACKUP=/usr/local/bin/gtar sendbackup: info RECOVER_CMD=/usr/local/bin/gtar -xpGf - ... sendbackup: info end ? gtar: ./logs/access.log: file changed as we read it | Total bytes written: 523622400 (500MiB, 9.7MiB/s) sendbackup: size 511350 sendbackup: end \-------- /-- squidzone2 /sqcache2/var lev 1 STRANGE sendbackup: start [squidzone2:/sqcache2/var level 1] sendbackup: info BACKUP=/usr/local/bin/gtar sendbackup: info RECOVER_CMD=/usr/local/bin/gtar -xpGf - ... sendbackup: info end ? gtar: ./logs/access.log: file changed as we read it | Total bytes written: 742635520 (709MiB, 7.4MiB/s) sendbackup: size 725230 sendbackup: end \-------- /-- squidzone2 /sqcache2/squidguard lev 0 STRANGE sendbackup: start [squidzone2:/sqcache2/squidguard level 0] sendbackup: info BACKUP=/usr/local/bin/gtar sendbackup: info RECOVER_CMD=/usr/local/bin/gtar -xpGf - ... sendbackup: info end ? gtar: ./log/squidGuard.log: file changed as we read it | Total bytes written: 3191879680 (3.0GiB, 9.0MiB/s) sendbackup: size 3117070 sendbackup: end \-------- NOTES: planner: disk mailserv:/usr1, estimate of level 0 failed. planner: Full dump of squidzone2:/sqcache2/squidguard promoted from 6 days ahead. planner: Full dump of squidzone1:/sqcache1/squidguard promoted from 6 days ahead. planner: Full dump of trel:/Users promoted from 1 day ahead. planner: Full dump of muninn:/var promoted from 2 days ahead. planner: Full dump of h220:/ promoted from 2 days ahead. planner: Full dump of squidtwo:/ promoted from 2 days ahead. planner: Full dump of pavlov:/ promoted from 2 days ahead. planner: Full dump of gatem:/usr1 promoted from 2 days ahead. planner: Full dump of c110:/ promoted from 2 days ahead. planner: Full dump of nlascar:/ promoted from 2 days ahead. taper: tape Curie10 kb 338138026 fm 46 [OK] big estimate: nlascar / 0 est: 6408M out 4101M big estimate: nlascar /var 0 est: 327M out 164M small estimate: finsen / 1 est: 57M out 199M DUMP SUMMARY: DUMPER STATS TAPER STATS HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s -------------------------- ------------------------------------- ------------- c110 / 0 1165 619 53.2 10:30 1006.2 0:50 12646.7 c110 /opt 1 0 0 1.6 0:41 0.1 0:00 797.6 curie / 0 FAILED -------------------------------------------- curie /export 1 96060 74970 78.0 123:31 10358.8 102:53 12436.7 curie /thump/flar 1 0 0 -- 0:01 0.8 0:00 102.5 curie -ump/source 1 36474 29879 81.9 57:15 8905.9 39:57 12761.9 curie -p/vmfs-bak 1 0 0 -- 0:01 0.7 0:00 72.2 dnix /dev/sda1 1 1841 1841 -- 18:54 1662.2 3:26 9150.0 everest /images3 1 0 0 -- 0:01 1.2 0:00 183.6 finsen / 1 547 199 36.3 2:53 1178.2 0:55 3663.5 finsen /export 1 33204 18265 55.0 105:25 2957.2 24:10 12901.2 gatem / 1 629 629 -- 7:37 1409.6 1:49 5885.2 gatem /usr1 0 63830 63830 -- 298:25 3650.6 84:18 12923.4 h220 / 0 4678 3130 66.9 49:56 1069.9 4:14 12593.6 h220 /opt 1 591 43 7.3 10:50 68.1 0:03 12696.7 huginn / 1 30479 30479 -- 73:26 7083.5 40:58 12695.2 ldap1 / 0 7784 2524 32.4 18:36 2315.9 3:25 12581.8 ldap1 -xport/home 1 0 0 1.0 0:05 1.0 0:00 1740.6 ldap1 /usr1 1 627 51 8.2 1:05 805.2 0:04 12750.4 lyra / 1 FAILED -------------------------------------------- lyra /3rdparty 0 40359 35645 88.3 215:49 2818.9 47:08 12904.5 lyra /db1 1 14077 2303 16.4 31:52 1233.7 3:06 12652.9 lyra /db2 1 18560 2078 11.2 37:03 957.3 2:46 12794.0 lyra /db3 0 FAILED -------------------------------------------- lyra /db4 0 25112 3516 14.0 59:40 1005.6 4:42 12764.7 lyra -port/home0 1 2 0 8.3 0:23 5.9 0:00 10717.5 lyra /ndevelop 0 9474 7037 74.3 58:25 2055.7 9:31 12614.7 lyra /space 1 104 28 26.4 6:37 71.1 0:03 8438.3 mailserv / 1 2618 499 19.1 7:01 1215.7 0:42 12145.9 mailserv /usr1 1 15100 -- PARTIAL 22:10 11629.3 PARTIAL muninn / 1 554 165 29.8 17:55 157.5 0:14 12531.6 muninn /var 0 3822 617 16.2 15:08 696.1 1:04 9897.2 ngato / 1 0 0 5.8 0:49 0.5 0:00 8948.1 nlascar / 0 3102 4101 132.2 13:30 5186.2 5:45 12172.8 nlascar /boot 1 0 0 -- 0:03 0.8 0:00 454.4 nlascar /data 1 0 0 0.9 0:36 0.1 0:00 1526.5 nlascar /var 0 292 164 56.1 1:39 1688.8 0:32 5233.7 panther / 1 1 0 7.8 2:33 0.7 0:00 10244.1 panther /data 1 0 0 -- 0:01 0.7 0:00 188.1 pavlov / 0 15079 9726 64.5 24:31 6770.9 40:30 4098.5 squidone / 1 159 19 11.9 5:24 59.9 0:02 12613.5 squidtwo / 0 25568 12758 49.9 36:15 6006.7 25:07 8667.4 squidzone1 -squidguard 0 2549 2549 -- 4:44 9185.5 3:27 12598.9 squidzone1 -cache1/var 1 499 499 -- 0:52 9875.7 0:41 12603.1 squidzone2 -squidguard 0 3044 3044 -- 5:41 9149.8 5:01 10369.9 squidzone2 -cache2/var 1 708 708 -- 1:38 7420.4 1:18 9337.0 trel /Users 0 3258 2201 67.5 7:57 4724.8 3:54 9612.1 trel /trelAQ 1 995 995 -- 4:26 3830.6 4:08 4113.7 trel /trelRZ 1 2 2 -- 0:08 219.3 0:00 12825.2 (brought to you by Amanda version 2.6.1p1) ----- End forwarded message ----- --- Brian R Cuttler [email protected] Computer Systems Support (v) 518 486-1697 Wadsworth Center (f) 518 473-6384 NYS Department of Health Help Desk 518 473-0773 IMPORTANT NOTICE: This e-mail and any attachments may contain confidential or sensitive information which is, or may be, legally privileged or otherwise protected by law from further disclosure. It is intended only for the addressee. If you received this in error or from someone who was not authorized to send it to you, please do not distribute, copy or use it or any attachments. Please notify the sender immediately by reply e-mail and delete this from your system. Thank you for your cooperation.
