RE: Rsync: Re: patch to enable faster mirroring of large filesystems
I, too, was disappointed with rsync's performance when no changes were required (23 minutes to verify that a system of about 3200 files was identical). I wrote a little client/server python app which does the verification, and then hands rsync the list of files to update. This reduced the optimal case compare time to under 30 seconds. Here's what it does, and forgive me if these sound similar to the stuff you're doing: - The client and server cache checksums (MD5, since there is no MD4 implementation conveniently available for Python that I know of) on a per-directory basis. These are kept in a .checksum file in the directory, so they persist from session to session. This is especially handy for the server, where (in my particular case) the files don't change very often. - On the initial compare the client sends the checksum of each .checksum file; if they match, it's not necessary to send the .checksum file, and we just culled an entire directory for a cost of about a 32 byte transfer. - If there's a mismatch, the client sends over the entire .checksum file. The server does the compare and sends back a list of files to delete and a list of files to update. (And now I think of it, it would probably be better if the server just sent the client back the list of files and let the client figure out what it needed, since this would distribute the work better.) - The client deletes the delete files, and uses rsync to update the update files. The ideal case is when all checksums are up to date. The worst-case is when the checksum cache needs to be built completely -- but this still only takes a couple of minutes, easily an order of magnitude better than the best-case I experienced with raw rsync. -Original Message- From: Alberto Accomazzi [mailto:[EMAIL PROTECTED]] Sent: Thursday, November 29, 2001 10:02 AM To: Dave Dykstra Cc: [EMAIL PROTECTED] Subject: Re: Rsync: Re: patch to enable faster mirroring of large filesystems
RE: Rsync: Re: patch to enable faster mirroring of large filesyst ems
I was at first, but then removed it. The results were still insufficiently fast. Were you using the -c option of rsync? It sounds like you were and it's extremely slow. I knew somebody who once went to extraordinary lengths to avoid the overhead of -c, making a big patch to rsync to cache checksums, when all he had to do was not use -c.
Re: Rsync: Re: patch to enable faster mirroring of large filesyst ems
On Thu, Nov 29, 2001 at 12:59:00PM -0600, Keating, Tim wrote: I was at first, but then removed it. The results were still insufficiently fast. Were you using the -c option of rsync? It sounds like you were and it's extremely slow. I knew somebody who once went to extraordinary lengths to avoid the overhead of -c, making a big patch to rsync to cache checksums, when all he had to do was not use -c. 23 minutes to check 3200 files is definitely unexpected. What options did you end up using? Normally rsync will only check the modification timestamps and the sizes of the files on both sides (that is, only a stat()) and if they match it will not do anything else. - Dave Dykstra
Re: Rsync: Re: patch to enable faster mirroring of large filesyst ems
It seems to me the new options --read-batch and --write-batch should go a long way towards reducing any time spent in creation of checksums and file lists, so you should definitely give 2.4.7pre4 a try. This is just a guess since I haven't actually used those options myself, but seems worth looking into. BTW, could we please have some real documentation about these options? What's in the man page doesn't come nearly close to telling what is cached and how to make use of it. Some examples of how people are using this option may be illuminating for those of us who don't have the time or inclination to figure it out from the code. -- Alberto In message [EMAIL PROTECTED], Keating, Tim writes: I was at first, but then removed it. The results were still insufficiently fast. Were you using the -c option of rsync? It sounds like you were and it's extremely slow. I knew somebody who once went to extraordinary lengths to avoid the overhead of -c, making a big patch to rsync to cache checksums, when all he had to do was not use -c. Alberto Accomazzi mailto:[EMAIL PROTECTED] NASA Astrophysics Data System http://adsabs.harvard.edu Harvard-Smithsonian Center for Astrophysicshttp://cfawww.harvard.edu 60 Garden Street, MS 83, Cambridge, MA 02138 USA
RE: Rsync: Re: patch to enable faster mirroring of large filesyst ems
Keating, Tim [[EMAIL PROTECTED]] writes: - If there's a mismatch, the client sends over the entire .checksum file. The server does the compare and sends back a list of files to delete and a list of files to update. (And now I think of it, it would probably be better if the server just sent the client back the list of files and let the client figure out what it needed, since this would distribute the work better.) Whenever caching checksums comes up I'm always curious - how do you figure out if your checksum cache is still valid (e.g., properly associated with its file) without re-checksumming the files? Are you just trusting size/timestamp? I know in my case I've got database files that don't change timestamp/size and yet have different contents. Thus I'd always have to do full checksums so I'm not sure what a cache would buy. -- David /---\ \ David Bolen\ E-mail: [EMAIL PROTECTED] / | FitLinxx, Inc.\ Phone: (203) 708-5192| / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \---/
Re: problem with 2.4.7pre4
Linux. Other files are ok, even files that are simular in size, I have a 700meg file that seems ok. I'm aware of the 2 gig limit, but these files aren't close to 2gigs. Note that I get very bad behavior with 2.4.7pre4, it doesn't even attempt to copy, 2.4.6 just seems to fail at the end. -jeremy On Thu, 29 Nov 2001, Dave Dykstra wrote: What operating system? Can you copy smaller files OK? I think the limit for 32 bits is 2GB so you shouldn't be running into that. - Dave Dykstra On Thu, Nov 29, 2001 at 02:24:14PM -0500, Jeremy Hansen wrote: yOn Thu, 29 Nov 2001, Jeremy Hansen wrote: rsync: connection unexpectedly closed (0 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(139) Just a note here, if I move the rsync server back to 2.4.6, I'm at least able to initiate an actual copy. 2.4.7pre4 just fails right away with the above. Also, on 2.4.6 I was getting this: access_log write failed on access_log : Success unexpected EOF in read_timeout THis is about an 800meg file I'm trying to copy. Thanks -jeremy -- The trouble with being poor is that it takes up all your time. -- The trouble with being poor is that it takes up all your time.
RE: Rsync: Re: patch to enable faster mirroring of large filesyst ems
In my particular case, it is reasonable to assume that the size and timestamp will change when the file is updated. (We are looking at it as a patching mechanism.) Right now it's actually using update time only, I should modify it to check the file size as well. Is there a way you could query your database to tell you which extents have data that has been modified within a certain timeframe? -Original Message- From: David Bolen [mailto:[EMAIL PROTECTED]] Sent: Thursday, November 29, 2001 2:12 PM To: 'Keating, Tim' Cc: [EMAIL PROTECTED] Subject: RE: Rsync: Re: patch to enable faster mirroring of large filesyst ems Keating, Tim [[EMAIL PROTECTED]] writes: - If there's a mismatch, the client sends over the entire .checksum file. The server does the compare and sends back a list of files to delete and a list of files to update. (And now I think of it, it would probably be better if the server just sent the client back the list of files and let the client figure out what it needed, since this would distribute the work better.) Whenever caching checksums comes up I'm always curious - how do you figure out if your checksum cache is still valid (e.g., properly associated with its file) without re-checksumming the files? Are you just trusting size/timestamp? I know in my case I've got database files that don't change timestamp/size and yet have different contents. Thus I'd always have to do full checksums so I'm not sure what a cache would buy. -- David /- --\ \ David Bolen\ E-mail: [EMAIL PROTECTED] / | FitLinxx, Inc.\ Phone: (203) 708-5192| / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \- --/
RE: Rsync: Re: patch to enable faster mirroring of large filesyst ems
Keating, Tim [[EMAIL PROTECTED]] writes: Is there a way you could query your database to tell you which extents have data that has been modified within a certain timeframe? Not in any practical way that I know of. It's not normally a major hassle for us since rsync is used for a central backup that occurs on a large enough time scale that the timestamp does normally change from the prior time. So our controlling script just does its own timestamp comparison and only activates the -c rsync option (which definitely increases overhead) if they happen to match. Although I will say that the whole behavior (the transaction log always has an appropriate timestamp, it's just the raw database file itself that doesn't) sure caught me by surprise in the beginning after finding what I thought was a valid backup wouldn't load :-) -- David /---\ \ David Bolen\ E-mail: [EMAIL PROTECTED] / | FitLinxx, Inc.\ Phone: (203) 708-5192| / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \---/
Re: cannot create .hosts.b0WX1x : File exists
Hello Dave, On Thu, 29 Nov 2001, Dave Dykstra wrote: What version of rsync are you using? The cannot create message is coming from receiver.c. Assuming you're using a released version of rsync and not a development version, it appears that the mktemp library call is creating the file and rsync isn't expecting that. Are you sure the Linuxes are exactly the same; do you perhaps have a different version of libc on that one machine (I'm guessing that maybe mktemp has been changed to create the file instead of just returning its name)? Andrew Tridgell changed the relevant section of code in the development version of rsync 6 months ago to use mkstemp (when configure determines it to be secure) instead of mktemp but he didn't say why. This is it. All working machines have 2.2 lnx kernel and libc-2.1.3.so nonworking one has 2.4 lnx kernel and libc-2.2.1.so. I changed line 121 of syscall.c to use: mktemp(template); instead of mkstemp(template); I got his linker warning, but rsync was linked OK and is working fine now. syscall.o: In function `do_mktemp': syscall.o(.text+0x32f): the use of `mktemp' is dangerous, better use `mkstemp' Dave, thank You for Your help. You guys are really great. Rok Krulec Portoroz/Slovenia, 29.Nov 2001 @ 23:12 CET Public GnuPG key @ http://rok.fpp.edu/pgp.key On Thu, Nov 29, 2001 at 12:34:04AM +0100, Rok Krulec wrote: Hello Dave, thank You for Your time; sorry I didn't included my rsyncd.conf. As I said, on two other machines with EXACTLY the same config file and EXACTLY the same passwd entries for nobody and nogroup = same ID's, it works perfectly. EXECTLY the same permissions on target directory alos ofcourse. This is my /etc/rsyncd.conf: --- CUT HERE --- motd file = /etc/issue.net max connections = 0 syslog facility = local3 [bkp] comment = Backup area path = /_silos read only = no list = no uid = nobody gid = nogroup hosts allow = XXX.XXX.XXX.XXX --- CUT HERE --- Portoroz/Slovenia, 29.Nov 2001 @ 00:24 CET Public GnuPG key @ http://rok.fpp.edu/pgp.key On Wed, 28 Nov 2001, Dave Dykstra wrote: You probably need to set read only = no in rsyncd.conf. - Dave On Wed, Nov 28, 2001 at 04:29:35PM +0100, Rok Krulec wrote: Hello, when I do: /opt/rsync/bin/rsync /etc/hosts targethost::bkp/ I get: cannot create .hosts.b0WX1x : File exists I check the targethost and I get empty file .hosts.b0WX1x When trying with other targethost-s it works, but on this one it doesn't. On the other targethosts I have exactly the same LinuX distribution, permissions and users than on the problem targethost. I have exactly the same configuration than on the other machines, except on probelm targethost I have SCSI hard disk. When i strace the rsync --deamon i get: 4697 open(.hosts.b0WX1x, O_RDWR|O_CREAT|O_EXCL|0x8000, 0600) = 4 4697 open(.hosts.b0WX1x, O_WRONLY|O_CREAT|O_EXCL|0x8000, 0600) = -1 EEXIST (File exists) It seems like it opens the file ReadWrite and then tries to open it readonly. I don't know why. It bangs then. Thanks for any hint, Rok Krulec
Re: cannot create .hosts.b0WX1x : File exists
On Thu, Nov 29, 2001 at 11:46:30PM +0100, Rok Krulec wrote: Hello Dave, On Thu, 29 Nov 2001, Dave Dykstra wrote: What version of rsync are you using? The cannot create message is coming from receiver.c. Assuming you're using a released version of rsync and not a development version, it appears that the mktemp library call is creating the file and rsync isn't expecting that. Are you sure the Linuxes are exactly the same; do you perhaps have a different version of libc on that one machine (I'm guessing that maybe mktemp has been changed to create the file instead of just returning its name)? Andrew Tridgell changed the relevant section of code in the development version of rsync 6 months ago to use mkstemp (when configure determines it to be secure) instead of mktemp but he didn't say why. This is it. All working machines have 2.2 lnx kernel and libc-2.1.3.so nonworking one has 2.4 lnx kernel and libc-2.2.1.so. I changed line 121 of syscall.c to use: mktemp(template); instead of mkstemp(template); I got his linker warning, but rsync was linked OK and is working fine now. syscall.o: In function `do_mktemp': syscall.o(.text+0x32f): the use of `mktemp' is dangerous, better use `mkstemp' Dave, thank You for Your help. You guys are really great. What version of sources is that which had mkstemp at line 121 of syscall.c? It's surprising that you could just replace one with the other, as mkstemp is supposed to open the file and mktemp is not supposed to. It sounds like you have some inconsistent version of the sources. - Dave
Re: cannot create .hosts.b0WX1x : File exists
Hello Dave, What version of sources is that which had mkstemp at line 121 of syscall.c? It's surprising that you could just replace one with the other, as mkstemp is supposed to open the file and mktemp is not supposed to. It sounds like you have some inconsistent version of the sources. rsync --version returns: rsync version 2.4.6 protocol version 24 Rok Krulec Portoroz/Slovenia, 30.Nov 2001 @ 00:03 CET Public GnuPG key @ http://rok.fpp.edu/pgp.key
Re: Rsync: Re: patch to enable faster mirroring of large filesystems
On Thu, Nov 29, 2001 at 11:02:07AM -0500, Alberto Accomazzi wrote: ... These numbers show that reading the filenames this way rather than using the code in place to deal with the include/exclude list cuts the startup time down to 0 (from 1hr). The actual sending of the filenames is down from 2h 15m to 1h 40m. The reason this isn't better is due to the fact that turning buffering on only helps the client, while the server still has to do unbuffered reads because of the way the list is sent across. Are you sure about that? I don't see any unbuffered reads. 2.3.2 did have the read_check() hack which was there to prevent SSH pipes from getting stuck, maybe that's what you're seeing. That was taken out in 2.4.0 so maybe that would greatly speed it up. As far as I can tell there is no way to get around the buffering without a protocol change or a different approach to sending this list. Given the data above, I think implementing --files-from this way would be the wrong way to go, for a number of reasons: I've been starting to think along those lines too. It should be a protocol change to just send the files and not treat it like excludes. In fact, the file list is normally sent from the sender to the receiver, but if the client is the receiver maybe we could figure out a way to have --files-from only send the list in the other direction. - Dave Dykstra
Re: problem with 2.4.7pre4
On 29 Nov 2001, Jeremy Hansen [EMAIL PROTECTED] wrote: Linux. By the way when reporting bugs like this it is good to give a more specific description, like RedHat 7.1 on x86. Other files are ok, even files that are simular in size, I have a 700meg file that seems ok. I'm aware of the 2 gig limit, but these files aren't close to 2gigs. Note that I get very bad behavior with 2.4.7pre4, it doesn't even attempt to copy, 2.4.6 just seems to fail at the end. 2.4.7 should not have a 2GB limit unless the underlying OS does. On Linux this means that you need a moderately recent version of both the kernel (2.4?) and C library. If you have that, despite running on a mostly-32-bit processor it should work properly. rsync --version will tell you whether rsync *thinks* it can handle big files, but it's possible that they will still fail. access_log write failed on access_log : Success unexpected EOF in read_timeout That error often means the ssh connection is failing. Can you please post the exact command you're running and the rsyncd configuration file? -- Martin
Re: Rsync: Re: patch to enable faster mirroring of large filesystems
In message [EMAIL PROTECTED], Dave Dykstra writes: On Thu, Nov 29, 2001 at 11:02:07AM -0500, Alberto Accomazzi wrote: ... These numbers show that reading the filenames this way rather than using the code in place to deal with the include/exclude list cuts the startup time down to 0 (from 1hr). The actual sending of the filenames is down from 2h 15m to 1h 40m. The reason this isn't better is due to the fact that turning buffering on only helps the client, while the server still has to do unbuffered reads because of the way the list is sent across. Are you sure about that? I don't see any unbuffered reads. Actually I'm not sure that the code intends to do unbuffered reads, but that's what's happening for sure from the trussing I've done on the server side. I'm not sure how the buffering should take place since the include/exclude file names are sent over the wire one at a time rather than as a chunk of data, but maybe buffering is done at a higher level. 2.3.2 did have the read_check() hack which was there to prevent SSH pipes from getting stuck, maybe that's what you're seeing. That was taken out in 2.4.0 so maybe that would greatly speed it up. Possible. Another reason why I don't think it's worth spending any more time patching 2.3.2 anyways... As far as I can tell there is no way to get around the buffering without a protocol change or a different approach to sending this list. Given the data above, I think implementing --files-from this way would be the wrong way to go, for a number of reasons: I've been starting to think along those lines too. It should be a protocol change to just send the files and not treat it like excludes. In fact, the file list is normally sent from the sender to the receiver, but if the client is the receiver maybe we could figure out a way to have --files-from only send the list in the other direction. Right. The point is that when Tridge wrote the code he was obviously envisioning a client sending a short exclude list to the server and then the server sending a massive list back to the client. Therefore no optimization nor compression has ever been included to ensure the fast trasfer of the exclude list, so patching things this way goes against the original design of the protocol. So probably the best thing to do is stick the file list right after the exclude list, turning on compression if -z has been selected and bump up the protocol so that we can be backwards compatible. At least that's my take. -- Alberto Alberto Accomazzi mailto:[EMAIL PROTECTED] NASA Astrophysics Data System http://adsabs.harvard.edu Harvard-Smithsonian Center for Astrophysicshttp://cfawww.harvard.edu 60 Garden Street, MS 83, Cambridge, MA 02138 USA
Re: problem with 2.4.7pre4
On Fri, 30 Nov 2001, Martin Pool wrote: On 29 Nov 2001, Jeremy Hansen [EMAIL PROTECTED] wrote: Linux. By the way when reporting bugs like this it is good to give a more specific description, like RedHat 7.1 on x86. Sorry. This is a Red Hat 6.2 machine. 2.2.19 kernel. Both ends are the same. Other files are ok, even files that are simular in size, I have a 700meg file that seems ok. I'm aware of the 2 gig limit, but these files aren't close to 2gigs. Note that I get very bad behavior with 2.4.7pre4, it doesn't even attempt to copy, 2.4.6 just seems to fail at the end. 2.4.7 should not have a 2GB limit unless the underlying OS does. On Linux this means that you need a moderately recent version of both the kernel (2.4?) and C library. If you have that, despite running on a mostly-32-bit processor it should work properly. rsync --version will tell you whether rsync *thinks* it can handle big files, but it's possible that they will still fail. access_log write failed on access_log : Success unexpected EOF in read_timeout That error often means the ssh connection is failing. The server is running rsyncd and client connections are going over an stunnel. The daemon is actually running using DJB's ucspi-tcp/daemontools/supervise. Here is the command line of the server: exec envuidgid root \ tcpserver -DRUvX -x /etc/tcpcontrol/rsyncd.cdb 0 rsync rsync -vvv --daemon Can you please post the exact command you're running and the rsyncd configuration file? Here is the command line of the client: rsync -avz --progress rsync://localhost/apache_logs/access_log . localhost because this is going over an stunnel. Here is the rsyncd.conf: use chroot = yes hosts allow = 127.0.0.1 uid = nobody gid = nobody [apache_logs] uid = root gid = root path = /usr/local/apache/logs comment = apache server log files, etc. read only = yes Thank You. -jeremy -- The trouble with being poor is that it takes up all your time.
Re: problem with 2.4.7pre4
On 29 Nov 2001, Jeremy Hansen [EMAIL PROTECTED] wrote: Sorry. This is a Red Hat 6.2 machine. 2.2.19 kernel. Both ends are the same. OK, thanks. unexpected EOF in read_timeout That error often means the ssh connection is failing. The server is running rsyncd and client connections are going over an stunnel. The daemon is actually running using DJB's ucspi-tcp/daemontools/supervise. In this case it may be that the stunnel is collapsing. It could be good to turn up the verbosity on stunnel, if possible. Failing that perhaps try attaching to it with # strace -p PIDOFSTUNNEL -o /tmp/stunnel.trace to see whether it exits or seems to get an error. The command and config file look OK. -- Martin
Re: problem with 2.4.7pre4
On Fri, 30 Nov 2001, Martin Pool wrote: On 29 Nov 2001, Jeremy Hansen [EMAIL PROTECTED] wrote: rsync -avz --progress rsync://localhost/apache_logs/access_log . tridge just reminded me that -a does *not* detect hardlinks. You need -H too. Hmm, there are no hard links involved here. I think you're confusing me with another poster. I'll do the trace on stunnel. Thanks -jeremy A documentation bug implied that -a was enough. I've just fixed it. -- The trouble with being poor is that it takes up all your time.
Re: problem with 2.4.7pre4
Actually, I did it this time without stunnel and got the same results: Nov 29 16:49:49 rio rsyncd[24007]: transfer interrupted (code 20) at rsync.c(229) It's literally right at the end of the transfer. Somewhere in renaming the temp file or something?? -jeremy On Thu, 29 Nov 2001, Jeremy Hansen wrote: So...it seems I did receive an error from stunnel. So now do I have to go off to the stunnel list :-) Nov 29 16:26:04 rio stunnel[20970]: SSL_write (ERROR_SYSCALL): Broken pipe (32) Nov 29 16:26:04 rio stunnel[20970]: Connection reset: 53424205 bytes sent to SSL, 104 bytes sent to socket Nov 29 16:26:04 rio rsyncd[20971]: transfer interrupted (code 20) at rsync.c(229) Thanks -jeremy On Thu, 29 Nov 2001, Jeremy Hansen wrote: On Fri, 30 Nov 2001, Martin Pool wrote: On 29 Nov 2001, Jeremy Hansen [EMAIL PROTECTED] wrote: rsync -avz --progress rsync://localhost/apache_logs/access_log . tridge just reminded me that -a does *not* detect hardlinks. You need -H too. Hmm, there are no hard links involved here. I think you're confusing me with another poster. I'll do the trace on stunnel. Thanks -jeremy A documentation bug implied that -a was enough. I've just fixed it. -- The trouble with being poor is that it takes up all your time.
Re: problem with 2.4.7pre4
On 29 Nov 2001, Dave Madole [EMAIL PROTECTED] wrote: It also seems that at one point rsync wasn't recognizing that two of the 800M files were actually hard linked together, although in the same run it did fine with smaller files (of the same name in different directories). That's an interesting bug. Can you write a script to reproduce it? Was anything else happening on the disk at the same time? Is it possible you didn't give -a or -H? -- Martin
Re: Why does one of there work and the other doesn't
On 29 Nov 2001, Ian Kettleborough [EMAIL PROTECTED] wrote: 1. How much memory does each file to be copied need. Obvisiouly I have too many files. Hard to say exactly. On the order of a hundred bytes per file. 2. Why does this command work: rsync -ax /usr/xx /backup/usr/ when: rsync -ax /usr/xx/ /backup/usr/ refuses to create the directory xx in /backup/usr and copies the contents of the directory to /backup Actually that's a feature not a bug: /usr/xx means the directory xx so it creates /backup/usr/xx /usr/xx/ means the contents of xx so it copies the contents directly into /backup/usr/ without creating an xx destination directory. Just use whichever one is appropriate. -- Martin
