Re: *Slow* amrestore
On Fri, 12 Mar 2004 at 11:08am, Gerhard den Hollander wrote > * Joshua Baker-LePain <[EMAIL PROTECTED]> (Thu, Mar 11, 2004 at 04:29:57PM -0500) > > > Anyways, it looks to be hardware, and not amanda or kernel related. So, > > thanks all, and sorry for the noise. Is it Friday yet? > > It is now. Yeah, and we started the day here with power outages and their lasting effects on equipment I don't control but which affects my stuff. Woo hoo. But at least it's ACC tourney day (college basketball, for those who have no idea what I'm talking about)... > What you describe above sounds like either: > - bad termination > (like an active terminator at the end of the chain, and the drive > set to terminate) Would that affect both drives on the chain? As I mentioned, I get good read speeds off the other drive in the library, on the same chain. > - cable length problem > there is a 3M maximum chainlength, the rul of thumb is (or at least > used to be) half a meter for each device. If you have 2 1-m scsi > cables connecting the 2 devices to your machine, that will end up > being exactly 3m , add some extra for the internal cabling in the > machine and you crossed it. But these are Ultra160 LVD drives. Most references I've seen say that such a chain can be 12m, not that I'm anywhere near that. I've got about a 1.5m cable going from the host to the library, and little .3m or so jumper between the two drives (they're in the same library). And again, wouldn't an overly long cable kill both drives' performance? Thanks again. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: *Slow* amrestore
On Thu, 11 Mar 2004 at 4:08pm, Jon LaBadie wrote > On Thu, Mar 11, 2004 at 03:29:23PM -0500, Joshua Baker-LePain wrote: > > > > But 4M works. ?? And to add insult to injury, that's going at about > > 70K/s. > > What about our old oft occuring observations on scsi devices, But isn't 1 goat/week enough? > Writing through that cable uses different wires than reading. So, here's where it gets fun. The tape drive I've been using is one of two in my library. It happens to be the 2nd one -- both in terms of position on the chain and SCSI ID. On a lark, I moved the tape to /dev/nst0 (note: same chain). 'amrestore -r' chugged right along at what looked to be full speed for the tape drive. The drives are the same model (SDX-700C), although I only recently added the second (slow) one, so it has less usage and a newer firmware revision. I cleaned the "slow" drive once, and that made an improvement, but only to 900K/s or so. Although, if I watch xosview, the transfer appears very bursty, whereas on the "fast" drive it's pretty consistent. I'll run the cleaning cycle again, and maybe upgrade the firmware on it (there is a newer firmware revision available). As an aside, Sony actually provides a Linux tool to do this, which almost shocked me out my boots. If all that doesn't work, I may have to look at the cable between nst0 and nst1. Anyways, it looks to be hardware, and not amanda or kernel related. So, thanks all, and sorry for the noise. Is it Friday yet? -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: *Slow* amrestore
On Thu, Mar 11, 2004 at 03:29:23PM -0500, Joshua Baker-LePain wrote: > > But 4M works. ?? And to add insult to injury, that's going at about > 70K/s. > What about our old oft occuring observations on scsi devices, Writing through that cable uses different wires than reading. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: *Slow* amrestore
Have you taken a look around in /proc/scsi? /proc/scsi/scsi should give you some basic information, and the subdir for your driver should give more details, such as what transfer rate the drive is negotatiated at, for example /proc/scsi/aic7xxx/0 for an Adaptec 2940 series. Perhaps there was a bus problem and the device hooked up at 10 MHz async or something. Reboot and get into the SCSI bios config page and make sure everything in there is kosher. I have even seen a few terminators go bad, so such things are not impossible, even though the drive was writing fine before. Since "dd" is having just as much problems, I would suspect that amanda is not the problem per se. Could be kernel, could be hardware, could be the tape cartridge, could be some stupid, proprietary Sony thing, dip switch settings on the drive, hardware compression settings. In some cases, I have taken tape drives and hooked them up to a Windows box so I could run proprietary diagnostics, take a look at the tape drive's internal logs, update the drive firmware. I think the weirdest stuff was using a 9600 bps null modem serial terminal connection to connect to the embedded console on an Exabyte 10h library, very strange set up. I have also pulled out SCSI cards and put them on a Windows box to check out the SCSI BIOS and load new firmware. Run a cleaning tape through just in case--with some drives, I had the problem of drives actually needing cleaning more frequently than the drives thought they needed it. Exabyte Eliant drives were notorious for that--I would run into these problems where I had to run a cleaning tape through 5-6 times in a row (even though the drive said it didn't need cleaning) and then finally it was fine, but in those cases, I was also seeing read and/or write errors. The Eliants just based cleaning on hours of read/write time, which turned out not to be often enough, so crud would accumulate over time--no doubt because amanda was streaming so efficiently that the drive was actually processing more length of tape than ordinarily anticipated :-) --jonathan
Re: *Slow* amrestore
On Thu, 11 Mar 2004 at 2:52pm, Jonathan Dill wrote > Hmm. Check also "mt status" to make sure the drive thinks that the > blocksize is "0" if not change it with "mt blksize". The files will be > perfectly fine with bs=16M. gzip and/or tar will probably give a warning > bitching about the nulls at the end, but it won't have any effect on the > restore, the files will still be intact. mt agress that the blocksize is variable: SCSI 2 tape drive: File number=4, block number=0, partition=0. Tape block size 0 bytes. Density code 0x32 (no translation). Soft error count since last status=0 General status bits on (8501): EOF WR_PROT ONLINE IM_REP_EN But dd doesn't seem to like my suggestions for blocksize: [EMAIL PROTECTED] data]$ sudo dd if=/dev/nst1 of=img.raw bs=16M dd: reading `/dev/nst1': Invalid argument OK, so 16M is technically bigger than the stated limits, so: [EMAIL PROTECTED] data]$ sudo dd if=/dev/nst1 of=img.raw bs=16777215 dd: reading `/dev/nst1': Value too large for defined data type [EMAIL PROTECTED] data]$ sudo dd if=/dev/nst1 of=img.raw bs=8M dd: reading `/dev/nst1': Value too large for defined data type But 4M works. ?? And to add insult to injury, that's going at about 70K/s. > In some desparate cases in the past, where some mung brain had data on > Sony Video 8 tapes (!) I ended up mucking around with "mt blksize" and > just using "cat" to try to grab the stuff off the tape without any > blocking, crazy as that sounds, but that was also on SGI IRIX with > Exabyte 820 "Eagle" drives. It was a pain, and there were loads of > errors, but I got most of the data off the tapes. That sounds... fun. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: *Slow* amrestore
On Thu, 11 Mar 2004 at 8:34pm, Gerhard den Hollander wrote > I am assuming the disk you are writing to, the machine the tape drive is > attached to and the machine on which the amrestore is running are all the > same ? Yep. > If not, your network might be the bottleneck. I was originally doing this via amrecover, and thought that may be it. Thus I switched to amrestore writing to a local disk. > If it is, check your syslog (usually /var/log/messages /var/log/allmessages) > and look if there is anything weird. > > Is the disk you are writing to OK ? Nothing strange in the syslog at all, and the disk is plenty quick. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: *Slow* amrestore
Hmm. Check also "mt status" to make sure the drive thinks that the blocksize is "0" if not change it with "mt blksize". The files will be perfectly fine with bs=16M. gzip and/or tar will probably give a warning bitching about the nulls at the end, but it won't have any effect on the restore, the files will still be intact. In some desparate cases in the past, where some mung brain had data on Sony Video 8 tapes (!) I ended up mucking around with "mt blksize" and just using "cat" to try to grab the stuff off the tape without any blocking, crazy as that sounds, but that was also on SGI IRIX with Exabyte 820 "Eagle" drives. It was a pain, and there were loads of errors, but I got most of the data off the tapes. --jonathan Joshua Baker-LePain wrote: On Thu, 11 Mar 2004 at 2:21pm, Jonathan Dill wrote I would try "amrestore -c" to just dump the image off the tape, and then do the uncompress and extraction separately, but you will need enough disk space to do it. Worst case, you could try "amrestore -r" and use "dd bs=32k skip=1 if=dump-file" to skip over the header, and then pipe that to the uncompress and extract processes. The current slow-running process is 'amrestore -r'. I'm guessing it's forcing the block size that's the culprit here -- the amrestore man page isn't quite clear, but it seems to say that if you don't specify -b, it'll default to 32k. I tried just 'dd if=/dev/nst1 of=img.raw' but it said "dd: reading `/dev/nst1': Cannot allocate memory". tapeinfo says this about blocksizes for the drive: MinBlock:2 MaxBlock:16777215 Maybe I should try dd with bs=16M? But will that pad the output file with an unacceptable-to-tar chunk at the end since the tapefile is unlikely to be an exact multiple of 16M? Thanks.
Re: *Slow* amrestore
--On Thursday, March 11, 2004 2:30 PM -0500 Joshua Baker-LePain <[EMAIL PROTECTED]> wrote: [snip] Maybe I should try dd with bs=16M? But will that pad the output file with an unacceptable-to-tar chunk at the end since the tapefile is unlikely to be an exact multiple of 16M? In my experience, dd should read in chunks of 16M but the last block (incomplete) will not be filled in (padded) to complete the block. Even if it was, here is what my gnu tar thinks of a tar with garbage at the end: --- [EMAIL PROTECTED](p1)(0):~/tartest% tar czvf foo.tar.gz foo foo/ foo/bar/ foo/bar/baz/ [EMAIL PROTECTED](p1)(0):~/tartest% ls -l foo.tar.gz -rw-r--r--1 areidstaff 152 Mar 11 14:42 foo.tar.gz [EMAIL PROTECTED](p1)(1):~/tartest% dd if=/dev/zero of=blank bs=1k count=10 10+0 records in 10+0 records out [EMAIL PROTECTED](p1)(0):~/tartest% ls -l total 16 -rw-r--r--1 areidstaff 10240 Mar 11 14:43 blank drwxr-xr-x3 areidstaff 102 Mar 11 14:42 foo/ -rw-r--r--1 areidstaff 152 Mar 11 14:42 foo.tar.gz [EMAIL PROTECTED](p1)(0):~/tartest% cat foo.tar.gz blank > test.tar.gz [EMAIL PROTECTED](p1)(0):~/tartest% tar tzf test.tar.gz gzip: stdin: decompression OK, trailing garbage ignored foo/ foo/bar/ foo/bar/baz/ tar: Child returned status 2 tar: Error exit delayed from previous errors [EMAIL PROTECTED](p1)(2):~/tartest% --- Yes, it does return '2' but the data has been extracted already. Actually, come to think of it, gzip is ignoring the garbage.. - [EMAIL PROTECTED](p1)(0):~/tartest% gunzip foo.tar.gz [EMAIL PROTECTED](p1)(0):~/tartest% cat foo.tar blank > test.tar [EMAIL PROTECTED](p1)(0):~/tartest% tar tvf test.tar drwxr-xr-x areid/staff 0 2004-03-11 14:42:08 foo/ drwxr-xr-x areid/staff 0 2004-03-11 14:42:08 foo/bar/ drwxr-xr-x areid/staff 0 2004-03-11 14:42:08 foo/bar/baz/ [EMAIL PROTECTED](p1)(0):~/tartest% - Tar doesn't even complain on my system here: tar (GNU tar) 1.13.25 Thanks. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University Just my 2cents.. antoine -- Antoine Reid Administrateur Système - System Administrator __ Logient Inc. Solutions de logiciels Internet - Internet Software Solutions 417 St-Pierre, Suite #700 Montréal (Qc) Canada H2Y 2M4 T. 514-282-4118 ext.32 F. 514-288-0033 www.logient.com *AVIS DE CONFIDENTIALITÉ* L'information apparaissant dans ce message est légalement privilégiée et confidentielle. Elle est destinée à l'usage exclusif de son destinataire tel qu'identifié ci-dessus. Si ce document vous est parvenu par erreur, soyez par la présente avisé que sa lecture, sa reproduction ou sa distribution sont strictement interdites. Vous êtes en conséquence prié de nous aviser immédiatement par téléphone au (514) 282-4118 ou par courriel. Veuillez de plus détruire le message. Merci. *CONFIDENTIALITY NOTE* This message along with any enclosed documents are confidential and are legally privileged. They are intended only for the person(s) or organization(s) named above and any other use or disclosure is strictly forbidden. If this message is received by anyone else, please notify us at once by telephone (514) 282-4118 or e-mail and destroy this message. Thank you.
Re: *Slow* amrestore
On Thu, 11 Mar 2004 at 2:21pm, Jonathan Dill wrote > I would try "amrestore -c" to just dump the image off the tape, and then > do the uncompress and extraction separately, but you will need enough > disk space to do it. Worst case, you could try "amrestore -r" and use > "dd bs=32k skip=1 if=dump-file" to skip over the header, and then pipe > that to the uncompress and extract processes. The current slow-running process is 'amrestore -r'. I'm guessing it's forcing the block size that's the culprit here -- the amrestore man page isn't quite clear, but it seems to say that if you don't specify -b, it'll default to 32k. I tried just 'dd if=/dev/nst1 of=img.raw' but it said "dd: reading `/dev/nst1': Cannot allocate memory". tapeinfo says this about blocksizes for the drive: MinBlock:2 MaxBlock:16777215 Maybe I should try dd with bs=16M? But will that pad the output file with an unacceptable-to-tar chunk at the end since the tapefile is unlikely to be an exact multiple of 16M? Thanks. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: *Slow* amrestore
Joshua Baker-LePain wrote: I'm reading some somewhat large (14-18GB) images off of AIT3 tapes, and it's taking *forever*. Some crude calculations show it coming off the tape at around 80 KB/s, whereas it was written out at 11701.6 KB/s. The tapes were written in variable block size mode. What's the best way to read these images more quickly? Are you piping the amrestore output, say to uncompress and extract files? Maybe the extraction process is too slow, and causes the tape to have to stop and reposition while the pipe is emptying out. I wonder if you could put a bigger buffer in the pipe somehow, that might help, say 64 MB to begin--I remember seeing a cat-like program that could buffer a pipe, I think it was with Linux software for doing backups to CD-R. I would try "amrestore -c" to just dump the image off the tape, and then do the uncompress and extraction separately, but you will need enough disk space to do it. Worst case, you could try "amrestore -r" and use "dd bs=32k skip=1 if=dump-file" to skip over the header, and then pipe that to the uncompress and extract processes. --jonathan
Re: *Slow* amrestore
On Thu, 11 Mar 2004 at 2:06pm, Joshua Baker-LePain wrote > I'm reading some somewhat large (14-18GB) images off of AIT3 tapes, and > it's taking *forever*. Some crude calculations show it coming off the > tape at around 80 KB/s, whereas it was written out at 11701.6 KB/s. The > tapes were written in variable block size mode. What's the best way to > read these images more quickly? *sigh* The host OS is Linux, btw. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
