Re: Rsync: Re: password prompts (fwd)
On Sat, Apr 07, 2001 at 08:00:19PM +0100, L. Cranswick wrote: FTP and Rsync via SSH to update files - how many users do this? I don't think I have persuaded one person to do this - they all think it too inconvenient - too much new stuff to learn - and it takes discipline to stick with it. I hate to be a bofh, but... If decent security is "too inconvenient", then I refuse to be responsible for the compromised box. Good security takes discipline, in any implementation. As for the rsync integration: It is only a simple ssh setup and a '-e ssh'. Is that _really_ too much to ask? -drew -- M. Drew Streib [EMAIL PROTECTED], http://dtype.org "Email sigs waste valuable bandwidth." PGP signature
rsync problems with .iso files
Hi, I've made some progress in getting rsync to work on the .iso I'm downloading, but now I've got another problem -- AFAICT, rsync is re-downloading the entire file! To recap briefly, I had downloaded an iso (mandrakefreq) using NCFTP, but the md5sum was bad. I am hoping to use rsync to correct it. After some false starts and overcoming other problems (which I describe below, FWIW), I invoked rsync with the following command line: rsync -c -vv --progress --partial \ carroll.cac.psu.edu::mandrake-iso/mandrakefreq-i586-20010316.iso \ mandrakefreq-i586-20010316.iso (While the PWD is the directory containing the iso with the bad md5sum.) rsync is now giving me a byte count and a % complete (e.g., "18846488 (2%)") and acting like it's downloading the entire file, in other words, not just downloading portions of the file. (My suspicicion that it's downloading the entire file is based on the speed of transfer -- I'm using a 33.6 kbps modem and usually get 9 to 11 megabytes per hour (or less) as a transfer rate. I've had rsync running for about two hours and, as you can see, it's downloaded almost 19 MB.) (As I sent this email, the byte count was up to about 24 MB after about 2 1/2 hours.) I'm using rsync 2.3.2 protocol version 21 (Mandrake 7.0). The rsync server at carroll.cac.psu.edu uses protocol version 24. What am I doing wrong? Theories about my current problem: 1. rsync can't repair an iso file except by retransmitting the entire file -- unlikely -- somebody would have told me this by now, and some of my earlier trials produced results like: wrote 105 bytes read 85 bytes 20.00 bytes/sec total size is 684400640 speedup is 3602108.63 (But the file (and md5sum) did not change.) 2. rsync is transmitting only portions of the iso file, but the overhead involved in doing the checksums on both ends makes the transfer time about the same as a total transfer at the speed of my 33.6 kbps modem -- I would be surprised -- that would be quite a coincidence. 3. There is something wrong in my command line -- this would not surprise me. 4. In the course of transferring the iso with the bad md5sum to a CD-Rom, then reading the CD on another computer (under Windows), then back to this computer using samba, it has become more corrupted, and retransmitting every "packet" is required. (Hmm, I can test this by running md5 on the transferred iso again, and compare it to the original bad iso.) -- Nope -- disproven -- the md5sum is the same (bad) md5sum from before the iso's oddysey. 5. The md5sum is not bad. -- This is sort of obsolete, and pretty much of a non-sequitur in this discussion. -- Not too likely -- this was an earlier theory, disproven now because rsync seems to be retransmitting the entire file. (And the trial after touching the local file to ensure a date mismatch but getting no change in the file contributed useful information at the time, but again, is irrelevant at this point.) 6. The problem is due to the protocol mismatch. -- Not too likely -- this condition prevailed throughout the trials, even when I got the "wrote 105 bytes read 85 bytes 20.00 bytes/sec" results. 7. Other?? Thanks for any help anyone can offer! Randy Kramer PS: Some other information (problems I've overcome, maybe): If you noticed some of my earlier posts, I suspected that a different problem (I got the message "unexpected EOF in read_timeout" and no change to the local copy of the iso file) might be because the local copy of the iso was in a FAT32 partition. I've now converted to an ext2 partition, but after making the conversion I still had the same problem. I was using -vvv as one of my options. I tried deleting one of the v's, and am now getting the behavior described above. So: -there seems to be a problem using -vvv as an option -I don't know whether the FAT32 partition was a problem or not. I suspect not, and when I convert the partition back again, I'll try it again.
Re: rsync problems with .iso files
On Sun, Apr 08, 2001 at 02:42:45PM -0400, Randy Kramer wrote: 1. rsync can't repair an iso file except by retransmitting the entire file -- unlikely -- somebody would have told me this by now, and some of my earlier trials produced results like: This really depends on how the file was damaged. I would venture that this is probably the case though. Everything else you described is as expected, it just seems that rsync is needing to transmit more than you'd hoped. This isn't definite, of course, but without knowing the exact contents of both files and doing some algorithm analysis, we can't really know. I have certainly seen large files that couldn't be "repaired" by rsync and needed to be redownloaded. -drew -- M. Drew Streib [EMAIL PROTECTED], http://dtype.org "Email sigs waste valuable bandwidth." PGP signature
Re: rsync problems with .iso files
On Sun, Apr 08, 2001 at 08:51:40PM +0100, M. Drew Streib wrote: I have certainly seen large files that couldn't be "repaired" by rsync and needed to be redownloaded. Indeed. In this case I wonder if the original was downloaded in ASCII mode instead of binary? That would definitely be a download-from-scratch repair... -- Cheers Jason Haar Unix/Special Projects, Trimble NZ Phone: +64 3 9635 377 Fax: +64 3 9635 417
Re: rsync problems with .iso files
Jason Haar wrote: On Sun, Apr 08, 2001 at 08:51:40PM +0100, M. Drew Streib wrote: I have certainly seen large files that couldn't be "repaired" by rsync and needed to be redownloaded. Indeed. In this case I wonder if the original was downloaded in ASCII mode instead of binary? That would definitely be a download-from-scratch repair... I downloaded the original (iso) file using ncftp, with commands: $ ncftp ncftp open carroll.cac.psu.edu ncftp cd pub/linux/distributions/mandrake/iso/ ncftp get mandrakefreq-i586-20010316.iso Unfortunately, my ISP (who changed the TOS since I signed up), kept cutting me off after nominally 12 hours (but actually 4 to 12 hours), so I had to periodically resume the download. Still, ncftp should have handled that without errors, and, the commands I've listed should not have done an ASCII transfer, and, I've done this before with no problem. (But the previous instance was about 3 months ago and before the ISP started cutting me off. I might have had to resume the previous downloaded isos once or twice each. Anyway, I remain puzzled. And, I'm trying to read Andrew Tridge's thesis. Randy Kramer
Re: rsync problems with .iso files
Drew, Thanks for the response! Very frustrating! I guess an iso consists of a mixture of binary and ASCII data (which I've confirmed somewhat by running less on the file. Lots of @^@^, but occasional patches of readable text. After reading some of the description of how the rsync algorithm works, it's hard to imagine a file transfer (under ncftp) that would be so bad as to have no or few sections transferred correctly (even if I had done it in ASCII mode). I've done iso transfers before using ncftp and have not had any trouble. Anyway. Thanks, Randy Kramer M. Drew Streib wrote: On Sun, Apr 08, 2001 at 02:42:45PM -0400, Randy Kramer wrote: 1. rsync can't repair an iso file except by retransmitting the entire file -- unlikely -- somebody would have told me this by now, and some of my earlier trials produced results like: This really depends on how the file was damaged. I would venture that this is probably the case though. Everything else you described is as expected, it just seems that rsync is needing to transmit more than you'd hoped. This isn't definite, of course, but without knowing the exact contents of both files and doing some algorithm analysis, we can't really know. I have certainly seen large files that couldn't be "repaired" by rsync and needed to be redownloaded. -drew -- M. Drew Streib [EMAIL PROTECTED], http://dtype.org "Email sigs waste valuable bandwidth." --- Part 1.2 Type: application/pgp-signature
Re: rsync problems with .iso files
Here are some possible explanations: 1) you used --partial on an earlier attempt and interrupted the transfer. That would leave you with a partial and potentially much smaller image locally, which would mean a subsequent transfer would send most of the file 2) the corruption is spread throughout the file, for example crlf conversion. If the corruption occurs more often than the block size that rsync chooses (assuming you don't override the choice) then rsync will end up sending the whole file. I suspect that (2) is the problem. To check this try dumping a piece of the two files (perhaps with "od") and looking at what differs between them. If it is a regular difference then you could use a program to "fix" that difference as best you can, then use rsync to fix up any still broken blocks. If you are interested in approaching this problem from an academic point of view then you might like to look in the subsection "Text transformation" in chapter 4 of my thesis. That section looks at how to efficiently handle this type of corruption. The current versions of rsync don't use the method in that section. Cheers, Tridge
