I presume you are looking in (your flavor's equivalent to) /var/log/messages to see if there are systems error messages about the CRC messages?
Deb Baddorf > On Jan 22, 2021, at 5:21 AM, Gene Heskett <[email protected]> wrote: > > from the email: > > FAILURE DUMP SUMMARY: > coyote /GenesAmandaHelper-0.61/config-bak lev 0 partial taper: source > server crc (6158b8f5:29861110032) and input server crc > (4f171223:29861110032) differ) > coyote /GenesAmandaHelper-0.61/config-bak lev 0 FAILED [data timeout] > coyote /GenesAmandaHelper-0.61/config-bak lev 0 partial taper: > successfully taped a partial dump > coyote /opt lev 0 partial taper: source server crc > (1f484cf4:15602100421) and input server crc (a7bc8db8:15602100421) > differ) > coyote /opt lev 0 FAILED [data timeout] > coyote /opt lev 0 FAILED [failed to set shm-ring] > coyote /usr/movies lev 0 partial taper: source server crc > (55338628:13370009600) and input server crc (feecb660:13370009600) > differ) > coyote /usr/movies lev 0 was successfully retried > > And: > > FAILED DUMP DETAILS: > /-- coyote /GenesAmandaHelper-0.61/config-bak lev 0 FAILED [data > timeout] > sendbackup: info BACKUP=APPLICATION > sendbackup: info APPLICATION=amgtar > sendbackup: info > RECOVER_CMD=/bin/gzip -dc |/usr/local/libexec/amanda/application/amgtar > restore [./file-to-restore]+ > sendbackup: info COMPRESS_SUFFIX=.gz > sendbackup: info end > \-------- > /-- coyote /opt lev 0 FAILED [data timeout] > sendbackup: info BACKUP=APPLICATION > sendbackup: info APPLICATION=amgtar > sendbackup: info > RECOVER_CMD=/bin/gzip -dc |/usr/local/libexec/amanda/application/amgtar > restore [./file-to-restore]+ > sendbackup: info COMPRESS_SUFFIX=.gz > sendbackup: info end > \-------- > > > NOTES: > planner: Incremental of coyote:/usr/local bumped to level 2. > driver: coyote /GenesAmandaHelper-0.61/config-bak 20210122020104 0 > [Will retry dump because of holding disk error: source server crc > (6158b8f5:29861110032) and input server crc (4f171223:29861110032) > differ)] > driver: coyote /opt 20210122020104 0 [Will retry dump because of > holding disk error: source server crc (1f484cf4:15602100421) and input > server crc (a7bc8db8:15602100421) differ)] > driver: coyote /usr/movies 20210122020104 0 [Will retry dump because of > holding disk error: source server crc (55338628:13370009600) and input > server crc (feecb660:13370009600) differ)] > taper: tape Dailys-2 kb 73355394 fm 79 [OK] > big estimate: coyote /GenesAmandaHelper-0.61/config-bak 0 > est: 28000M out 0M > > left in the holding disk: > > root@coyote:data$ ls -l /sdb/dumps/20210122020104/ > total 57456372 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:31 > coyote._GenesAmandaHelper-0.61_config-bak.0 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:08 > coyote._GenesAmandaHelper-0.61_config-bak.0.1 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:16 > coyote._GenesAmandaHelper-0.61_config-bak.0.10 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:17 > coyote._GenesAmandaHelper-0.61_config-bak.0.11 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:18 > coyote._GenesAmandaHelper-0.61_config-bak.0.12 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:19 > coyote._GenesAmandaHelper-0.61_config-bak.0.13 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:20 > coyote._GenesAmandaHelper-0.61_config-bak.0.14 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:21 > coyote._GenesAmandaHelper-0.61_config-bak.0.15 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:22 > coyote._GenesAmandaHelper-0.61_config-bak.0.16 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:23 > coyote._GenesAmandaHelper-0.61_config-bak.0.17 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:23 > coyote._GenesAmandaHelper-0.61_config-bak.0.18 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:24 > coyote._GenesAmandaHelper-0.61_config-bak.0.19 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:09 > coyote._GenesAmandaHelper-0.61_config-bak.0.2 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:25 > coyote._GenesAmandaHelper-0.61_config-bak.0.20 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:26 > coyote._GenesAmandaHelper-0.61_config-bak.0.21 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:27 > coyote._GenesAmandaHelper-0.61_config-bak.0.22 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:28 > coyote._GenesAmandaHelper-0.61_config-bak.0.23 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:28 > coyote._GenesAmandaHelper-0.61_config-bak.0.24 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:29 > coyote._GenesAmandaHelper-0.61_config-bak.0.25 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:30 > coyote._GenesAmandaHelper-0.61_config-bak.0.26 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:31 > coyote._GenesAmandaHelper-0.61_config-bak.0.27 > -rw------- 1 amanda amanda 501932304 Jan 22 02:31 > coyote._GenesAmandaHelper-0.61_config-bak.0.28 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:10 > coyote._GenesAmandaHelper-0.61_config-bak.0.3 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:11 > coyote._GenesAmandaHelper-0.61_config-bak.0.4 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:12 > coyote._GenesAmandaHelper-0.61_config-bak.0.5 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:13 > coyote._GenesAmandaHelper-0.61_config-bak.0.6 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:14 > coyote._GenesAmandaHelper-0.61_config-bak.0.7 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:15 > coyote._GenesAmandaHelper-0.61_config-bak.0.8 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:16 > coyote._GenesAmandaHelper-0.61_config-bak.0.9 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:54 coyote._opt.0 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:33 coyote._opt.0.1 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:49 coyote._opt.0.10 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:49 coyote._opt.0.11 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:50 coyote._opt.0.12 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:51 coyote._opt.0.13 > -rw------- 1 amanda amanda 922527941 Jan 22 02:54 coyote._opt.0.14 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:34 coyote._opt.0.2 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:35 coyote._opt.0.3 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:38 coyote._opt.0.4 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:41 coyote._opt.0.5 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:43 coyote._opt.0.6 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:45 coyote._opt.0.7 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:46 coyote._opt.0.8 > -rw------- 1 amanda amanda 1048576000 Jan 22 02:47 coyote._opt.0.9 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:59 coyote._usr_movies.0 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:55 coyote._usr_movies.0.1 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:58 > coyote._usr_movies.0.10 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:58 > coyote._usr_movies.0.11 > -rw------- 1 amanda amanda 787523584 Jan 22 03:59 > coyote._usr_movies.0.12 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:55 coyote._usr_movies.0.2 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.3 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.4 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.5 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.6 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.7 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.8 > -rw------- 1 amanda amanda 1048576000 Jan 22 03:57 coyote._usr_movies.0.9 > root@coyote:data$ > > And I was asked to show the amstatus output when it failed: > > root@coyote:data$ cat /home/amanda/log/amstat.d/amstat-210122-0507 > Using: /usr/local/var/amanda/Daily/amdump.1 > > Thats it, normally its about 10k of stuff > > > One of the failed ones header: > > root@coyote:data$ dd if=00046.coyote._opt.0 bs=32k count=1 > AMANDA: SPLIT_FILE 20210122020104 coyote /opt part 1/-1 lev 0 comp .gz > program APPLICATION > APPLICATION=amgtar > ORIGSIZE=24681320 > NATIVE-CRC=b5751dc0:25273671680 > CLIENT-CRC=a7bc8db8:15602100421 > SERVER-CRC=a7bc8db8:15602100421 > DLE=<<ENDDLE > <dle> > <program>APPLICATION</program> > <disk>/opt</disk> > <level>0</level> > <auth>bsdtcp</auth> > <compress>BEST</compress> > <record>YES</record> > <index>YES</index> > <datapath>AMANDA</datapath> > <exclude> > <list>/GenesAmandaHelper-0.61/excludes</list> > </exclude> > <backup-program> > <plugin>amgtar</plugin> > <property> > <name>ignore</name> > <value encoding="raw" > raw="OiBzb2NrZXQgaWdub3JlZCQ=">:_socket_ignored$</value> <value > encoding="raw" > raw="ZmlsZSBjaGFuZ2VkIGFzIHdlIHJlYWQgaXQk">file_changed_as_we_read_it$</value> > </property> > <property> > <name>one-file-system</name> > <value>yes</value> > </property> > <property> > <name>check-device</name> > <value>no</value> > </property> > </backup-program> > </dle> > ENDDLE > To restore, position tape at start of file and run: > dd if=<tape> bs=32k > skip=1 | /bin/gzip -dc | /usr/local/libexec/amanda/application/amgtar > restore [./file-to-restore]+ > > > 1+0 records in > 1+0 records out > 32768 bytes (33 kB, 32 KiB) copied, 0.0179666 s, 1.8 MB/s > > If I hunt down a succesfully retried dle and dump its header, there will > not be any crc reports in it. > > Conclusions/clues: > > 1. it only blows up on a big, but randomly selected level 0 > that may be from any of the 5 machines being backed up. > > 2. its nearly always because of a CRC error in the holding disk. > > 3. the holding disk has been swapped out twice now. Original was spinning > rust, 2 replacements are SSD's and are much faster. > > 4. amstatus always fails, IMO the failure that starts all this. > > 5. the error messages are not the least illuminating IMNSHO. the dle > for /GenesAmandelper/config-bak is 40Gigs but its the last 60 copys of > the configs that made these backup, and a copy of amanda's own database > for the last 60 backups, and the last 60 reports generated by amanda's > activities. > > Obviously I need help. Many thanks to those who try. > > Cheers, Gene Heskett > -- > "There are four boxes to be used in defense of liberty: > soap, ballot, jury, and ammo. Please use in that order." > -Ed Howdershelt (Author) > If we desire respect for the law, we must first make the law respectable. > - Louis D. Brandeis > Genes Web page > <https://urldefense.proofpoint.com/v2/url?u=http-3A__geneslinuxbox.net-3A6309_gene&d=DwICaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=HMrKaRiCv4jddln9fLPIOw&m=yQM2ya6OAM4AxYcZ_Q7XtA9d-kai2Ck8TU7WUkY0gJc&s=x2ZmAHmmYZyPAW8QIuCM7YE2IGmELKrVhLpCJKgQpEE&e= > >
