rsync hanging in read-batch mode
Trying to do something like this, in Ubuntu 11.10/12.04:
Location A
(work occurs changing files)
rsync --write-batch /media/bkup/foo.rsb /path/to/files/ /path/to/copy/
Location B
rsync --read-batch /media/bkup/foo.rsb /path/to/files/
rsync --read-batch /media/bkup/foo.rsb /path/to/copy/
(work occurs changing files)
rsync --write-batch /media/bkup/foo.rsb /path/to/files/ /path/to/copy/
Location A
rsync --read-batch /media/bkup/foo.rsb /path/to/files/
rsync --read-batch /media/bkup/foo.rsb /path/to/copy/
return to top
there are two directories I'm trying to do this with, the one that is just
files, and symlinks etc.,
works ok, but the one the includes /home/ hangs during read-batch. I sort of
wonder if it's related to
strange things in the filesystem like .gvfs and filesystem mount points - just
speculation.
using a recent compile of rsync:
git log | head -3
commit d51a3adb4fca3e6b1b046c6e570828f3bca8fe36
Author: Wayne Davison ---@samba.org
Date: Sat May 5 08:01:09 2012 -0700
cat rsync_dbg.sh
#!/bin/sh
ulimit -c unlimited
# Some systems have truss or tusc instead of strace.
# The -f option tells strace to follow children too.
# The -o option tells strace where to send its output.
strace -f -o /tmp/rsync-$$.out /home/tbrown/Package/rsync/rsync/rsync ${@}
. rsync_dbg.sh --exclude .smb --exclude .gvfs --exclude 'Ubuntu One' --exclude
'Dropbox' --one-file-system --verbose --archive --progress --human-readable
--archive --delete --ignore-errors --read-batch
/media/bkup64/bkup/trans/usr1.rsb /mnt/usr1/usr1/
(the same flags were used for the write batch)
after a few files being synced, everything seems to stop and dstat confims
nothing is happening.
netstat -p | grep sync
unix 3 [ ] STREAM CONNECTED 268012 3478/rsync
unix 3 [ ] STREAM CONNECTED 268011 3477/rsync
unix 3 [ ] STREAM CONNECTED 272995 3477/rsync
unix 3 [ ] STREAM CONNECTED 272994 3477/rsync
Ctrl-C in original terminal
home/tbrown/.cwe.remind
4.85K 100% 94.69kB/s 0:00:00 (xfr#10, ir-chk=2562/5267)
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(613)
[receiver=3.1.0dev]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(613)
[generator=3.1.0dev]
I don't want to post the output of strace to the list, it contains listings of
contents of /home/*
but I'll send it to a developer if they ask for it.
Cheers -Terry
rsync: [receiver] write error: Broken pipe (32)
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync file corruption when destination is a SAN LUN (Solaris 9 10)
I have run into a problem using 'rsync' to copy files from local disk to a SAN mounted LUN / file-system. The 'rsync' seems to run fine and it reports no errors, but some files are corrupted (check-sums don't match originals, and file data is changed). So far, I have found this problem on both Solaris 9 and Solaris 10 OSes and on several different models of Sparc systems using different versions of 'rsync' (2.6.8, 3.0.2, and 3.0.6). All of these systems are using QLogic HBAs and connect to QLogic FC switches and the SAN storage is on Sun/StorageTek arrays. My quick example of the problem: SAN mounted LUN / file-system == /apps local disk has OS system files == / mkdir /apps/junk rsync -avcHS /sbin/. /apps/junk/. no errors reported no errors reported in system logs then immediately do the same 'rsync' again rsync -avcHS /sbin/. /apps/junk/. it finds 2-3 files where the check-sums don't match and it re-copies them. And if I do a 3rd 'rsync', it re-copies the same 2-3 files. If I don't use the sparse files option, -S, the copies are successful and the data matches between the original files and the 'rsync'-ed copies. But I need to use the sparse files processing for the files that I need to copy. I do not see this problem if the 'rsync'-s are from: - local-disk to local-disk - local-disk to NFS file-system - NFS file-system to local-disk What other data would be useful to debug this problem? = Terry Countryman terry.country...@oit.gatech.edu -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
large number of small files to move
Hello, I have a large migration of a single system going from windows to linux. We are moving 9 TB and 60 million files. I am planning on using rsync to move this data. Are there any special considerations I should have in mind when moving this much data and more importantly, this many small files? I am mounting the volumes out of fstab like this: //dss01/f$/mnt/dss01/f cifs credentials=/etc/asdf,ro 0 0 And rsync command is: rsync -rt --delete --stats /mnt/dss01/f /data00 Any ideas? Thanks! Terry -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
summary only in output
Hello, I don't want my output to include every file like -v does. I just want the nice little summary at the end of the operation: sent 691442 bytes received 2462 bytes 462602.67 bytes/sec total size is 683504 speedup is 0.99 I have about 50 million files I need to move from one machine to the other and don't want the overhead associated with printing every file that is being moved but I would like a summary at the end of the job. I also don't want to run my output through a filter like grep for obvious reasons. --output-log= puts a newline instead of the information which isn't great either. Any ideas? Here is my command: rsync -rt --delete foo/ bar/ Thanks! -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: summary only in output
On Feb 13, 2008 2:24 PM, Wayne Davison [EMAIL PROTECTED] wrote: On Wed, Feb 13, 2008 at 02:02:26PM -0600, Terry wrote: I don't want my output to include every file like -v does. I just want the nice little summary at the end of the operation: Try the --stats option. It includes some extra statistics at the end of the run, followed by that nice little summary. ..wayne.. That's perfect. Overlooked that option. :\ Thanks! -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Linux to Windows
John Jablonski wrote: Hmmm. Oddly (?) enough, you can make a .FILE using a dos prompt (cmd), but not using the win GUI. RMB-Create new-Text file on your desktop (or anywhere using windows explorer) and try and create a file named .dotfile. You can't do it (in XP-pro anyways). This is just an example of one of the many hard-coded string match traps in Windows Explorer that bear little relation to the capabilities of the file system. If i remember it correctly, the characters \ / : * ?| are restricted at a filesystem level, rather than by explorer. You can create a file .test in dos/cygwin and select it in explorer and press f2 to rename it. Explorer will accept the existing name, but whine at you if you try to call it something else, like .test2 Terry. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: win32, ssh and rsync
Donald Orbin wrote: Hi all I have been trying to get myhead around this for smetime now and hope one of the gurus here can assist. I would like to connect to a server(that is not running a rsync daemon) to push files from a win32 (2003 server) to an OBSD machine. The command I have successfully used is: rsync -Rrazvvu -e ssh -i .ssh/pvt /cygdrive/c/dirname [EMAIL PROTECTED]:BACKUP [where BACKUP is a directory /home/username/BACKUP ] This reports opening conection using : ssh -i .ssh/pvt -l username rsync --server -vvulogDtprRz . BACKUP [and youcan see the process on the server... ie: rsync --server -vvulogDtprRz . BACKUP ] After which te synchronisation takes place. I have written a compiled vb wrapper programto perform this operation - however it does not run from the windows 2003 scheduler. I do a similar thing using a batch file wrapper to execute a shell script (So I can pass command-line arguments to shell scripts from scheduled task) Do you have any logging output from the task? What happens if you right-click the job in the scheduler list and Run now? Scheduled tasks do run in a different environment that interactive tasks. Things to check are whether or not the wrapper can find the executables; if the job is running as a user with all the necessary permissions; If the wrapper needs access to the console to run and, generally, if the server is available at the time the task is set to run. I thus want to know if the command can be broken into 2 parts... the ssh connect part... and the rsync part. I do not beleive this is the same as the ssh tunneling since the rsyncd.conf file does not exist on the server and I am notrunning rsync as a daemon. Has anyone else had experience with this. Regards Donald ___ How much free photo storage do you get? Store your holiday snaps for FREE with Yahoo! Photos http://uk.photos.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Two folders !
BASING Tibor Vovcak wrote: Hello !! I need to sync two folders in fedora linux. One is at /ftp/test/test1 Second is at /ftp/test/test1/test3 I need to sync /ftp/test/test1/test3 that new files uploaded will be seen in /ftp/test/test1 every hour or a day . How do i do that ? I admit I may be misunderstanding the purpose here but, from your description, it sounds lke you're setting up an hourly mirror of /ftp/test/test1/test3 in /ftp/test/test1 The command line: rsync -av /ftp/test/test1/test3/* /ftp/test/test1/ Should be all you need to do, but it depends on how the folders are structured. If you have a folder called /ftp/test/test1/test3/test3 then the above command line would likely break horribly. Can I ask why you want to mirror the contents of a folder into its own parent folder? Terry. Thank YOU ALL ! Tibor Vovak Slovenia -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is rsync right for us?
Bruce Therrien wrote: Hi, What we want to do is synchronize our music files between our 2 servers like every 30 minutes or so. Been trying to find scripts to do this, but no success. The files are created on our Mac G4 running OS X server, and then tranferred manually to the IBM server in the same office on the same network. So long as the synchronisation is one-way and you're not trying to merge changes made on the IBM to the G4 at the same time, then rsync is absolutely ideal for your needs. We would like to automate the process, by checking the G4 about every 30 minutes for changed files and then uploading them to the IBM, using the same directory structure, if possible.. It should also update any existing files on the IBM. Rsync does all of the checking for you and will only upload the new files or the changed parts of the existing ones, thereby minimising network traffic. You just give it a local source directory and a remote destination and it will do the rest. It's probably the quickest way you can do what you've specified. Terry. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Limit the total bytes transfered?
Michael Best wrote: I ask because a client had a broken filesystem that occasionally has 2T+ files on it (broken filesystem, so they weren't actually that big) but we happily ran up a huge b/w bill with rsync. For this specific example you could probably wildcard match the files with a --exclude= argument. Borked filesystems usually generate matchable files. For a more generic test: if [ `du -sb root` -gt size limit ] ; then barf; else rsync; fi Which is crude, time-consuming and won't transfer anything if you step even one byte over the limit. How evil do you want to be? :) You could get your script to parse file-from before hand and maybe only backup the entries that come in under the limit. Perhaps build the full file list yourself, incrementing a counter each file until you reach the size limit then fire off rsync with that new list passed to --files-from. Again, not very elegant, but it'd do the job. This also assumes you're running something unixy (Doing this stuff in batch a whole new level of hell). -Mike -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to control rsync log?
If I specify this option on a 2.6.3 server config, will it still be effective even if the clients are 2.6.0 (or possibly 2.5.6, I'm still pestering them to upgrade) Terry. Wayne Davison wrote: On Thu, Feb 24, 2005 at 12:35:21PM -0700, [EMAIL PROTECTED] wrote: Is there a way to let rsync server to have the ability to control if it wants to log those directory changes or not, like transfer logging does for files? Add this line to your rsyncd.conf (in either the global-settings section or an individual module): max verbosity = 0 (That setting is present in 2.6.3, but not documented.) That will keep the verbose flag from getting set on the server even when the user chooses to specify -v for themselves. As long as the user isn't sending files to the server, they shouldn't notice a difference in the output. When files are being sent to the server, that setting will prevent the generator from outputting about new directories and symlinks in 2.6.3 (but 2.6.4 will handle this properly). ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Redundant Rsync?
I use the following bit of shell to check whether an rsync server is up:
#!/bin/bash
# Prod the rsync server...
check=$(echo -e \n |netcat server1.example.com 873);
if [ $check != @RSYNCD: 26 ]; then {
echo Unable to connect to rsync server.
# Perform your failsafe measures here
} fi
If you're not running rsyncd and connect via SSH, then:
check=$(echo -e \n |netcat server1.example.com 22 |head -1);
with $check needing to be whatever your SSH server should return. (Mine
is SSH-2.0-OpenSSH_3.8p1 Debian 1:3.8p1-2)
You can wrap that up in a for loop to iterate through a whole bunch of
servers until you find one that's up if you need to.
Terry.
Ernie Oporto wrote:
Is there a way to tell rsync that if it does not have contact with a
certain rsync server to continue on to another for file requests? I
imagine something like this, where you define two servers as the
source...
/usr/local/bin/rsync {serv1,serv2}::src dest
...proposing that
1. Should serv1 be down when the sync is originally started, that
serv2 be contacted after a timeout.
2. Not so important, but probably a desirable thing once the above is
available, if serv1 or its connection dies in the middle of a sync,
that the sync continue on to serv2.
I'm not sure that I'm asking for something as complex as a failover
mode between paired servers, but just something in the client that
lets it know to move on.
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: cron/rsync issue
It could be a timing issue that means there's no route or server
available at that time. Given the length of the command line, I'd be
tempted to put it in a shell script with a connection checker:
#!/bin/bash
# Prod the rsync server...
check=$(echo -e \n |netcat sunsite.uio.no 873);
if [ $check != @RSYNCD: 26 ]; then {
echo Unable to connect to rsync server. Dumping traceroute:
traceroute -n sunsite.uio.no
# or whatever other tests you want to run.
} else {
# rsync command...
} fi
I use something like this before any unattended rsyncs.
Adding a bit more - might give you some more info, too :)
Mark Watts wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I've got an rsync command...
rsync -av --stats --progress --partial --delete-after --bwlimit=2000
rsync://sunsite.uio.no/Mandrakelinux /export/ftp/mandrakelinux
This works just fine when runfrom the command line (as user 'mwatts').
I've put it in the 'mwatts' users' crontab:
MAILTO=[EMAIL PROTECTED]
0 3 * * * rsync -av --stats --progress --partial --delete-after --bwlimit=2000
rsync://sunsite.uio.no/Mandrakelinux /export/ftp/mandrakelinux
However, Every day I get the following email from cron:
rsync: failed to connect to sunsite.uio.no: Connection timed out
rsync error: error in socket IO (code 10) at clientserver.c(88)
Why would I be getting this if it works on the command line?
Mark.
- --
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFA7lFtBn4EFUVUIO0RAvWEAJ9FemOwR5PDBK67bGmRwPErMTzUfgCg997T
B2PrqIKIOhSv/eQWNidrSlc=
=bgFJ
-END PGP SIGNATURE-
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: wildcard error in source path?
As you say, it does delete files that don't exist on the sending side
and I can see why the behaviour looks wrong at first glance
The difference between dump/* and dump/ is the list of files you are
giving to rsync in the first place.
With dump/*:
For each file in dump/, do {
Comapre file with remote copy, and upload file or changes.
}
With dump/:
Get list files in local dump/
Get list of files in remote dump/
for each entry in remote list, do {
If local counterpart doesn't exist, delete.
Else compare local-remote and upload file or changes.
}
The former method doesn't allow for the concept of missing local file.
It's not broken, really, you're just asking it to do something else. If
there are any subdirectories inside dump/ then --delete should work as
you expect for those.
Is there a particular reason why you can't use dump/?
(Previous disclaimer still applies)
Terry.
Stuart Halliday wrote:
Logically, this is correct behaviour, I think.
dump/* is a wildcard that matches every _existing_ local file in the
dump/ directory. Since the file you deleted doesn't exist, it isn't
considered by rsync.
dump/ tells rsync to compare the contents of the local dump/ directory
with those of the remote one and, in your case, will delete on the
remote host any files that don't exist locally.
Disclaimer: I haven't used --delete myself, so I could be wrong.
Yes your statement sounds logic. But the use of --delete is stated as being:
delete files that don't exist on the sending side
Which I assume means those on the remote side.
Since it isn't deleting the file on the remote side then rsync is a touch broken.
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question
I had this problem trying to script an unattended backup. (rsync 2.6.1 on cygwin) I found that if you need to pass command line arguments to ssh you need to use: rsync --rsh=ssh -i key Using -e, if I remember it correctly, just tries to execute a command called ssh -i key which, obviously, doesn't exist. I think this is a feature rather than a bug, though. Maybe :) Terry. hello i looked in google , faq etc but didnt found a answer. sorry if i overseen something. how i can auth. via a hostkey without make a config in ~/.ssh normaly ssh has support with ssh -i /keyfile is there any way to combine it via rsync , with rsync -e ssh -i key ..etc etc does not work thx bye -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug reporting
John wrote: Possibly rsync can use ssh to forward a local port chosen the same way ftp chooses a port fo active ftp. Then local rsync opens a connexion to 127.0.0.1:port at the local end, and ssh forwards the stream to rsync running as a daemon (on another randomly-chosen port? a user-specified port?) a the destination. Bear in mind that this way you'll be encrypting the entire session instead of just the Please start rsyncd for me bit, so you'll likely suffer a speed hit depending on how powerful the two machines are. John also wrote... I keep getting timeouts transferring between two ADSL accounts and I've not decide whether to blame Arachnet (the ISP at both ends) or rsync or even to suppose I'm expecting too much. As an aisde, I've had a problem similar to the one you describe with an unrelated service. Turns out the NAT table was dropping entries that went unused for 5 minutes, which meant the firewall started blocking return traffic as it appeared to be unrelated to any outgoing connection. I don't know if this would ever be an issue with rsync but it might be worth a look. Terry. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync-2.6.1pre-1 hang
The backtrace you cite looks to be impossible (nested calls to send_file_list() can't happen), so I assume that the list is not valid for some reason. I haven't seen such a hang as you describe. Have you delved into this any further? Well what I did after my last post was recompiling rsync with different optimisation options, and also with all the optimisations disabled (-O0). the directory tree that is being synchronised is a so-called gentoo linux's 'portage', consisting of lots of directories (~14k) and files (~68k), most of which are pretty small (5k) whats further interesting, that after todays update of the tree from the upstream server, all of the sudden i can not reproduce described behaviour anymore... is it possible that it was something in the tree causing rsync to act so weird ? regards, terry -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync-2.6.1pre-1 hang
=0x3, s=0x3) at batch.c:202 #102 0x0805d2be in read_longint (f=873) at io.c:710 #103 0x08062f73 in read_batch_csum_info (flist_entry=0, s=0x0, checksums_match=0x1) at batch.c:256 #104 0x0805219b in do_server_recv (f_in=0, f_out=0, argc=1501700484, argv=0x2a2514bc) at main.c:518 #105 0x2a287cf9 in ?? () #106 0x in ?? () #107 0x in ?? () #108 0x59822184 in ?? () #109 0x2a2514bc in ?? () #110 0x0003 in ?? () #111 0x0804a180 in ?? () #112 0x in ?? () #113 0x0804a1a1 in ?? () #114 0x08051f20 in do_recv (f_in=134553376, f_out=3, flist=0x59822174, local_name=0x806bcd0 U\211WV1S\203\f) at main.c:439 Please let me know if you need further information, best regards, Terry -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re[2]: how rsync works
On Monday, November 24, 2003 at 7:04 PM, jw wrote: snipped The intent was to, if there is interest, get feedback to polish the document for placement on the web site. Or to let someone who is perhaps a better writer take it over. Not to self-publish the thing. Perhaps i had to high of expectations but i expected at least one or two would say something, even if it was just yuck or its a start before knowledge of it subsided into archives. I'll put it back up if i hear more than just a whimper. I just joined the list and I, for one, would be very interested in reading the document and giving you feedback. I think such a document would be very useful. I've spent the last couple of days reading all things rsync on the Internet, and I still have questions that would, most likely, be answered by such a document. So please consider this as more than just a whimper. -- Terry -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: A checksum question
I'm sorry if you mis-understood me on this issue. I'm not trying to denigrate Rsync, as we are convinced that it is a highly capable tool that we can adapt for our needs. It is just that in certain situations the checksum processing absorbs a lot of CPU time. I have a user that has been using the 2.4.6 version for some considerable time and is very happy with it. However, when I moved him to the 2.5.5 version he found that his transfers took considerably longer. This appears to be due to the introduction of the MD4 checksum, I believe was introduced at release 2.5. We have a large and disparate computer centre, with virtually every flavour of unix, where we find that Rcp is less than reliable for moving data between them. This unreliability has prompted us to look at other solutions, with Rsync at the top of the list, as it implements many of the features that we have identified for our needs, checksumming (for data integrity) being one of them. Regards, Terry - jw schultz wrote: On Tue, Mar 25, 2003 at 04:31:00PM +, Terry Raggett wrote: I'd like to know a little about the internals of RSYNC. I am a little confused as to why RSYNC is using both the simple 32 bit algorithm and the MD4 checksum function on the same files. From my testing this causes a vast overhead that is clearly not represented by RCP (fairly obvious!). Removing checksumming from a secure whole-file LAN transfer brings RSYNC in line with the general performance of RCP. Can someone explain to me the rationale behind the checksum algoritm and use within the RSYNC protocol? I'm asking this question as I am finding it rather difficult to convince some of our users that RSYNC is a viable replacement for RCP functionality, which is necessary to resolve some of the limitations we encounter with the standard RCP. Many thanks in anticipation, Terry Raggett Rsync is not a replacement for rcp. Rcp is a remote-copy utility. Rsync is a remote-update utility. They each exist for completely different purposes. Without knowing what limitations you are encountering with rcp or to the specific purpose of use i couldn't speak to the suitability of rsync. There are many times that rsync is not the best or most efficient tool for the job. If all you want is a fast light-weight copy utility, rsync isn't it. There are even times when rsync is ill suited to an update operation. The tool-box was invented for a reason. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
A checksum question
I'd like to know a little about the internals of RSYNC. I am a little confused as to why RSYNC is using both the simple 32 bit algorithm and the MD4 checksum function on the same files. From my testing this causes a vast overhead that is clearly not represented by RCP (fairly obvious!). Removing checksumming from a secure whole-file LAN transfer brings RSYNC in line with the general performance of RCP. Can someone explain to me the rationale behind the checksum algoritm and use within the RSYNC protocol? I'm asking this question as I am finding it rather difficult to convince some of our users that RSYNC is a viable replacement for RCP functionality, which is necessary to resolve some of the limitations we encounter with the standard RCP. Many thanks in anticipation, Terry Raggett -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Problem with checksum failing on large files
-Original Message- From: Derek Simkowiak [mailto:[EMAIL PROTECTED]] Sent: Saturday, October 12, 2002 2:14 PM To: Craig Barratt Cc: Terry Reed; Donovan Baarda; '[EMAIL PROTECTED]' Subject: Re: Problem with checksum failing on large files My theory is that this is expected behavior given the check sum size. Craig, Excellent analysis! Assuming your hypothesis is correct, I like the adaptive checksum idea. But how much extra processor overhead is there with a larger checksum bit size? Is it worth the extra code and testing to use an adaptive algorithm? I'd be more inclined to say This ain't the 90's anymore, realize that overall filesizes have increased (MP3, MS-Office, CD-R .iso, and DV) and that people are moving from dialup to DSL/Cable, and then make either the default (a) initial checksum size, or (b) block size, a bit larger. Terry, can you try his test (and also the -c option) and post results? I tried --block-size=4096 -c --block-size=4096 on 2 files (2.35 GB 2.71 GB) still had the same problem - rsync still needed to do a second pass to successfully complete. These tests were between Solaris client AIX server (both running rsync 2.5.5). As I mentioned in a previous note, a 900 MB file worked fine with just -c (but required -c to work on the first pass). I'm willing to try the fixed md4sum implementation, what do I need for this? I cannot try these tests on a Win32 machine because Cygwin does not support files 2 GB I could only find rsync as part of Cygwin. I don't have the time nor the patience to try to get rsync to compile using MS VC++ :-) Is there a Win32 version of rsync with large file support available? I do not have any Linux boxes available to test large files. Thanks. -- Terry -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Problem with checksum failing on large files
Would you mind trying the following? Build a new rsync (on both sides, of course) with the initial csum_length set to, say 4, instead of 2? You will need to change it in two places in checksum.c; an untested patch is below. Note that this test version is not compatible with standard rsync, so be sure to remove the executables once you try them. Craig I changed csum_length=2 to csum_length=4 in checksum.c this time rsync worked on the first pass for a 2.7 GB file. I'm assuming that this change forced rsync to use a longer checksum length on the first pass, what checksum was actually used? Here is the verbose output: opening connection using ssh 10.200.1.1 -l twr4321 /home/twr4321/rsync-src/rsync-2.5.5-mod/rsync --server --sender -vvvlogDtpr . /rsync.guest/SUBMcopy.txt.7 receiving file list ... server_sender starting pid=67130 make_file(1,SUBMcopy.txt.7) expand file_list to 4000 bytes, did move recv_file_name(SUBMcopy.txt.7) received 1 names 1 file to consider recv_file_list done get_local_name count=1 SUBMcopy.txt.7 recv_files(1) starting generator starting pid=8128 count=1 recv_generator(SUBMcopy.txt.7,0) send_file_list done send_files starting sending sums for 0 send_files(0,/rsync.guest/SUBMcopy.txt.7) generate_files phase=1 send_files mapped /rsync.guest/SUBMcopy.txt.7 of size 2715101559 recv_files(SUBMcopy.txt.7) recv mapped SUBMcopy.txt.7 of size 2710310258 SUBMcopy.txt.7 calling match_sums /rsync.guest/SUBMcopy.txt.7 built hash table hash search b=16384 len=2715101559 match at lines snipped 2715101559 100%2.31MB/s0:18:39 done hash search sending file_sum got file_sum renaming .SUBMcopy.txt.7._iaq4p to SUBMcopy.txt.7 set modtime of SUBMcopy.txt.7 to (1032979931) Wed Sep 25 11:52:11 2002 false_alarms=188029 tag_hits=661854315 matches=121690 sender finished /rsync.guest/SUBMcopy.txt.7 recv_files phase=1 send_files phase=1 generate_files phase=2 send files finished total: matches=121690 tag_hits=661854315 false_alarms=188029 data=721332599 recv_files finished Number of files: 1 Number of files transferred: 1 Total file size: 2715101559 bytes Total transferred file size: 2715101559 bytes Literal data: 721332599 bytes Matched data: 1993768960 bytes File list size: 79 Total bytes written: 1323432 Total bytes read: 730229860 wrote 1323432 bytes read 730229860 bytes 576253.90 bytes/sec total size is 2715101559 speedup is 3.71 _exit_cleanup(code=255, file=main.c, line=925): about to call exit(255 Thanks. -- Terry -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Problem with checksum failing on large files
I'm having a problem with large files being rsync'd twice because of the checksum failing. The rsync appears to complete on the first pass, but then is done a second time (with second try successful). When some debug code was added to receiver.c, I saw that the checksum for the remote file the temp file do not match on the first try, so (as expected) it repeats the rsync the checksums do match after the retry. I have reproduced this behavior with various files ranging from 900 MB to 2.7 GB (though smaller files in several MB range work fine on the first pass). I just need to rsync a single large file for this scenario. The client is rsync 2.5.5 on Solaris 9 the server is rsync 2.5.5 on AIX 4.3.3. I see the same results with Solaris 9 rsync 2.5.2 client Solaris 7 rsync 2.5.2 server, as well as Solaris 9 rsync 2.5.2 client with AIX 2.5.5 server. I also see the same results for rsync in daemon mode over SSH. I used gcc to compile rsync in all cases. Is there a different checksum mechanism used on the second pass (e.g., different length)? If so, perhaps there is an issue with large files for what is used by default for the first pass? Here is the verbose output: === $ rsync -avvv --stats --progress [EMAIL PROTECTED]::rsync/gb-testfile gb-testfile opening tcp connection to 10.200.1.1 port 873 Password: receiving file list ... recv_file_name(gb-testfile) received 1 names 1 file to consider recv_file_list done get_local_name count=1 gb-testfile generator starting pid=2500 count=1 recv_generator(gb-testfile,0) recv_files(1) starting sending sums for 0 generate_files phase=1 recv_files(gb-testfile) recv mapped gb-testfile of size 432402669 gb-testfile 915211191 100%3.61MB/s0:04:01 got file_sum renaming .gb-testfile.nDaO4e to gb-testfile set modtime of gb-testfile to (1034093089) Tue Oct 8 09:04:49 2002 redoing gb-testfile(0) recv_generator(gb-testfile,0) recv_files phase=1 sending sums for 0 generate_files phase=2 recv_files(gb-testfile) recv mapped gb-testfile of size 915211191 gb-testfile 915211191 100% 10.23MB/s0:01:25 got file_sum renaming .gb-testfile.oDaO4e to gb-testfile set modtime of gb-testfile to (1034093089) Tue Oct 8 09:04:49 2002 recv_files finished Number of files: 1 Number of files transferred: 2 Total file size: 915211191 bytes Total transferred file size: 1830422382 bytes Literal data: 482837431 bytes Matched data: 1347584951 bytes File list size: 46 Total bytes written: 1275722 Total bytes read: 483225574 wrote 1275722 bytes read 483225574 bytes 1154949.45 bytes/sec total size is 915211191 speedup is 1.89 _exit_cleanup(code=0, file=main.c, line=925): about to call exit(0) === Thanks. -- Terry -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Problem with checksum failing on large files
-Original Message- From: Derek Simkowiak [mailto:dereks;itsite.com] Sent: Friday, October 11, 2002 1:51 PM To: Terry Reed Cc: '[EMAIL PROTECTED]' Subject: Re: Problem with checksum failing on large files I'm having a problem with large files being rsync'd twice because of the checksum failing. I think this was reported recently. Please try using the -c option (always checksum) and see if the makes the problem go away. This is a high priority bug for me (although I have not yet experienced it). --Derek Using -c helps for the smallest file (900 MB), but has no effect on the larger files (e.g, 2.7 GB). Most of my files are between 1.5 GB 3 GB. Any other suggestions? Thanks. -- Terry -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
getaddrinfo problem
Can anyone tell me what the work around fix was for the getaddrinfo host not found problem? It's something to do with the port number I believe. Terry - -- +---+--+-+ | Terry Raggett | | /~~\ /~~\ | | ECMWF | e-mail: [EMAIL PROTECTED] | /\/\| | Shinfield Park| Tel:(+44 118) 949 9358 | ECMWF ECMWF | | Reading | Fax:(+44 118) 986 9450 | ECMWF ECMWF | | Berkshire RG2 9AX | Telex: (+44 118) 984 7908 | \/\/| | England | | \__/ \__/ | +---+--+-+ | ECMWF is the European Centre for Medium-Range Weather Forecasts| ++ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: getaddrinfo() problem with AIX 4.3.3 and rsync 2.5.2?
I have built rsync on Irix 6.5, Fujitsu something, AIX 4.3, Sus Linux 7.x, HPUX
something. The only one I've had a problem with is AIX 4.3. IBM tell me
getaddrinfo is working as designed and that the port number must be assigned in
the /etc/services file. I can't see why this should be so, but they aren't
going to change it.
Terry
-
On Apr 1, 15:26, Allen, John L. wrote:
Subject: getaddrinfo() problem with AIX 4.3.3 and rsync 2.5.2?
I had to apply the following patch to the clientname.c check_name() function
before I could successfully use hosts allow in rsyncd.conf. Without the
patch I kept getting rsync: forward name lookup for ... failed: Host not
found
errors. These errors were produced by the rsync daemon, and putting in a
debugging statement revealed that the port_buf being used had a large port
number
in it (like 32835). It seems wrong-headed to pass that port to getaddrinfo
to check for host name validity, but since I don't know much network
programming,
I'll defer to almost anyone :-)
Perhaps it's just AIX 4.3.3 getaddrinfo() that's busted?
$ gdiff -u clientname.c.orig clientname.c
--- clientname.c.orig Fri Jan 25 17:59:37 2002
+++ clientname.cMon Apr 1 13:32:10 2002
@@ -240,7 +240,7 @@
hints.ai_family = ss_family;
hints.ai_flags = AI_CANONNAME;
hints.ai_socktype = SOCK_STREAM;
- error = getaddrinfo(name_buf, port_buf, hints, res0);
+ error = getaddrinfo(name_buf, NULL, hints, res0);
if (error) {
rprintf(FERROR,
RSYNC_NAME : forward name lookup for %s failed:
%s\n,
--
John L. Allen, Dept 0631, EMS
516-346-8456, MS C06-001
Northrop Grumman Information Technology
Internal Information Services
--
JOHN.L.ALLEN.vcf
[ Attachment (application/octet-stream): JOHN.L.ALLEN.vcf 836 bytes ]
-- End of excerpt from Allen, John L.
--
+---+--+-+
| Terry Raggett | | /~~\ /~~\ |
| ECMWF | e-mail: [EMAIL PROTECTED] | /\/\|
| Shinfield Park| Tel:(+44 118) 949 9358 | ECMWF ECMWF |
| Reading | Fax:(+44 118) 986 9450 | ECMWF ECMWF |
| Berkshire RG2 9AX | Telex: (+44 118) 984 7908 | \/\/|
| England | | \__/ \__/ |
+---+--+-+
| ECMWF is the European Centre for Medium-Range Weather Forecasts|
++
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
rsynd-2.5.1 Makefile
Make fails with make: fatal error: line 65: Syntax error. when building on a Fujitsu VPP system (system_V). Make doesn't like the continuation to a blank line for LIBOBJ. Distribution version: LIBOBJ=lib/fnmatch.o lib/compat.o lib/snprintf.o lib/mdfour.o \ lib/permstring.o \ ZLIBOBJ=zlib/deflate.o zlib/infblock.o zlib/infcodes.o zlib/inffast.o \ zlib/inflate.o zlib/inftrees.o zlib/infutil.o zlib/trees.o \ zlib/zutil.o zlib/adler32.o Corrected version: LIBOBJ=lib/fnmatch.o lib/compat.o lib/snprintf.o lib/mdfour.o \ lib/permstring.o ZLIBOBJ=zlib/deflate.o zlib/infblock.o zlib/infcodes.o zlib/inffast.o \ zlib/inflate.o zlib/inftrees.o zlib/infutil.o zlib/trees.o \ zlib/zutil.o zlib/adler32.o Terry - -- +---+--+-+ | Terry Raggett | | /~~\ /~~\ | | ECMWF | e-mail: [EMAIL PROTECTED] | /\/\| | Shinfield Park| Tel:(+44 118) 949 9358 | ECMWF ECMWF | | Reading | Fax:(+44 118) 986 9450 | ECMWF ECMWF | | Berkshire RG2 9AX | Telex: (+44 118) 984 7908 | \/\/| | England | | \__/ \__/ | +---+--+-+ | ECMWF is the European Centre for Medium-Range Weather Forecasts| ++
rsync-2.5.0 getaddrinfo in AIX4.3?
Anyone have problems with getaddrinfo under AIX4.3? I have built 2.5.0 under linux, HP and VPP without too much in the way of problems. However, AIX4.3 version builds OK but doesn't execute getaddrinfo correctly. Any ideas? Terry Raggett - -- +---+--+-+ | Terry Raggett | | /~~\ /~~\ | | ECMWF | e-mail: [EMAIL PROTECTED] | /\/\| | Shinfield Park| Tel:(+44 118) 949 9358 | ECMWF ECMWF | | Reading | Fax:(+44 118) 986 9450 | ECMWF ECMWF | | Berkshire RG2 9AX | Telex: (+44 118) 984 7908 | \/\/| | England | | \__/ \__/ | +---+--+-+ | ECMWF is the European Centre for Medium-Range Weather Forecasts| ++
RSYNC Tutorial
I tried to retrieve the RSYNC tutorial via the WEB page http://samba.org/rsync/resources.html. This fails (for me) for the following reason: Unable to resolve IP address for everythinglinux.org. Can anyone suggest an alternative? Many thanks Terry -
