Re: remote files not being deleted
On Thu, Feb 26, 2004 at 08:05:30AM -0600, Trey Nolen wrote: It may be that you are running out of memory. --delete causes the a second file list (about 100bytes/file) to be created contianing all the files existing on the receiver. Combined with --delete-after this happens at the time when the memory usage is at its greatest due to copy-on-write and fragmentation. Although it looks like my problem was caused by a path problem with 2.5.6 (I need to specify /. instead of / for the source), I'm interested in this memory issue. Is this file list created on the client side or the server side? Neither. This delete file list exists only breifly on the receiver. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: remote files not being deleted
On Wed, Feb 25, 2004 at 08:36:26PM -0600, Trey Nolen wrote: I've got an issue with remote files being deleted after the local file has been deleted. For some reason, this isn't happening. I'm running rsync 2.5.6 protocol 26 (yes, I know there are newer versions, but logistics dictates that I can't upgrade right now). I have used the --delete, --delete-after, and --ignore-errors options in all sorts of combinations. An example of the calling command follows: rsync -avR -e sh --numeric-ids --delete --progress --delete-after--ignore-errors --exclud e /proc/ / [EMAIL PROTECTED]:/ OK, the weird part is that it will delete files if I do a relatively small part of the file system (say /root/ for instance). I'm not sure what the limit is on where it quits working. Is this something that's been seen before? I looked through the archives but didn't find anything relating to this. Any suggestions? I guess I could run multiple jobs and do each directory separately, but that is less than optimal. This is occurring on SEVERAL computers. The client and server side are both Linux with ext3 filesystems. I have some client machines running rsync under Cywin, but I haven't checked to see if they have the issue yet. It may be that you are running out of memory. --delete causes the a second file list (about 100bytes/file) to be created contianing all the files existing on the receiver. Combined with --delete-after this happens at the time when the memory usage is at its greatest due to copy-on-write and fragmentation. You may want to try without --delete-after. That would reduce the memory requirements somewhat and if you do run out of memory with --delete it will be more obvious because the rsync will fail outright. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Deleted files
On Mon, Feb 23, 2004 at 05:04:00PM -0700, Walls Rob W Contr 75 CS/SCBS wrote: Thought my troubles were fixed, but files disappeared again and reappeared after a few dozen rsync cycles. I am mapping a Win2K directory to Linux with Samba. That mounted directory is then Rsynced to another Linux box on a schedule (Linux to Linux, because I can't run rsync on the windows machine). For some unknown reason, several thousand files are periodically (after several days) deleted from the remote Linux box. Running Rsync a few dozen times seems to get things back to normal, but causes great stress to myself and my users. I am using the delete option, but these files should not be deleted, because they haven't changed. Files eventually get back in sync. I can't figure out why they are deleted in the first place. Any ideas on what to check? Firstly, smbfs on linux is related to samba but is not samba. By mapping a Win2K directory to Linux with Samba i assume you mean mounting a smb/cifs share from a W2K server with smbfs on linux. Assuming that clarification is correct we can now procede. Given that this simply doesn't seem to be happenning with other filesystems i expect that either smbfs or the W2K server are intermittantly dropping entries as readdir is done. It may be there is a bug in smbfs or even in W2K but it may simply be that directory reading over smb cannot be done coherently if there are long delays between readdir() calls on largish directories. If this were happening with local filesystems i would suggest send_directory() be refactored (or do it myself) to reduce the time spent with a directory open. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Log formatting: quoting format option flags in Perl
On Thu, Feb 19, 2004 at 07:19:52PM -0800, Shifra Raffel wrote: Hello, I'm having trouble with setting desired punctuation and spaces with log formatting in a perl script. The log-format options I am using are --log-format=%h%a%l%o%f%t%u%c. Full rsync command at end of this message. When I run the rsync command and write the results to a file, $results = `$rsyncstr 21`, my log format options are ignored if I have the options quoted in the command, and the log is unreadably compressed if I leave off the quotes. I've tried qq//, escaped \, escaped \', and many other ways of escaping quotes. Any suggestions? Full rsync string: rsync -Cclrtvzn --log-format=%h %a %l %o %f %t %u %c /localdir [EMAIL PROTECTED]:/remotedir This is not an rsync issue. It is a perl issue. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: open files
On Thu, Feb 19, 2004 at 07:40:14PM -0800, Tarun Karra wrote: hi guyz, One simple question. What does rsync do when it encounters open files. Do we have to use open file manager(like st bernard) to back up open files or is there any open source open file manager or can rsync backup open files by itself. Rsync is a POSIX/SUS compliant utility. It doesn't know from open files. I assume you are speaking from the context of some brain-damaged single-user OS that doesn't allow files to be opened by multiple processes simultaniously. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync and rdist-like specials
On Wed, Feb 18, 2004 at 09:54:01PM -0500, Joseph McDonald wrote: Greetings, We manage a large farm of systems and push changes, patches, new software, etc. out to our clients using kerberized rdist. It's clunky and slow but it's got the magic of specials. Using a special one can tell rdist that if a particular file changes, preform this command. So, if we make a change to httpd.conf, we can tell rsync to bounce httpd. If httpd.conf hasn't change, the process is left alone. It's perfect for pushing out updated service config files. Is there anything similar in rsync? Thanks! Not built in. rsync -v will report to stdout all the files updated. That output can be post-processed to do such things. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature request: case insensitivity
On Tue, Feb 17, 2004 at 12:58:25AM -0500, Ethan Tira-Thompson wrote: I sync files to a memory stick fairly frequently. The memory stick uses a basic FAT format, which kills case. What's more, on some platforms (Windows), the drivers make all filenames uppercase, whereas on others (linux, mac) all the filenames are lowercase. So I currently have a script which will go through the source directory, rename all of the files to all upper or all lower case, and then run rsync. Otherwise, rsync will try to copy some files every time, even though they haven't changed. (for obvious reasons - it thinks the filenames don't match) It would make my life, scripts, and portability much easier if there was an option for rsync to treat files with the same name, ignoring case, to be treated as a filename match. This seems like something that would be useful for the several other platforms which do store case, but ignore it for OS file matches. (I did look around for any other references to this idea before I posted) No chance. Doing this in kernel space is ugly, slow and prone to inconsistant results due to aliases. Doing case insensitivity in something like rsync is even worse. What you seem to be asking for is less that of case insensitivity than of tolerating filesystems that don't preserve case. You could try a patch that would force case in the filelist as it is received but that would probably be fraught with problems. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature Request - Load Throttling
On Tue, Feb 17, 2004 at 03:16:32PM -0800, Marc Perkel wrote: If you aleady are doing this - then thanks. But here's a feature I'd like to see. Rsync often puts a lot of load on the system because of high disk access. I'd like to see rsync become load aware and to slow itself down under high load levels. Seems to me that it could be something fairly easy to add to have it check the load levels and back down so as not to slow down the system. This is what process and i/o schedulars are for. In most cases rsync is i/o bound. Either disk or network. That means it spends most of its time sleeping already. This becomes increasingly true as the performance gulf grows between CPU and that of memory, disk and network. Perhaps you have a suggestion for defining what constitutes high load and how to determine that on all the different platforms? Hint: load average is meaningless. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature Request - Load Throttling
On Tue, Feb 17, 2004 at 06:47:19PM -0800, Marc Perkel wrote: Actually - the problem is disk IO. And the disk IO is what makes the load levels go up. The load level is something that's readable can can be used to have rsync slow itself down. Nice doesn't do the trick. Nice helps - but even at nice +19 it still slows the system to a crawl when backing up from one drive to another. Is that is on AIX with 12 AS400 CPUs or the VMS SSI cluster? Or is that a single CPU linux box with a 2.4.?? kernel? So - if rsync could watch the load levels and pause every now and then to put a little space between disk access at high load levels it would make it a lot friendlier to the system. The reason nice doesn't work is that once the system call is made to access the disk - nice doesn't apply. What load levels? Do you have some nice C code that can do that for ALL the platforms without misreading? This is what process and i/o schedulers are for. Maybe you should contact the people responsible for whatever kernel it is you are running. jw schultz wrote: On Tue, Feb 17, 2004 at 03:16:32PM -0800, Marc Perkel wrote: This is what process and i/o schedulars are for. In most cases rsync is i/o bound. Either disk or network. That means it spends most of its time sleeping already. This becomes increasingly true as the performance gulf grows between CPU and that of memory, disk and network. Perhaps you have a suggestion for defining what constitutes high load and how to determine that on all the different platforms? Hint: load average is meaningless. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature Request - Load Throttling
On Tue, Feb 17, 2004 at 07:34:06PM -0800, Marc Perkel wrote: jw schultz wrote: On Tue, Feb 17, 2004 at 06:47:19PM -0800, Marc Perkel wrote: Actually - the problem is disk IO. And the disk IO is what makes the load levels go up. The load level is something that's readable can can be used to have rsync slow itself down. Nice doesn't do the trick. Nice helps - but even at nice +19 it still slows the system to a crawl when backing up from one drive to another. Is that is on AIX with 12 AS400 CPUs or the VMS SSI cluster? Or is that a single CPU linux box with a 2.4.?? kernel? It's on a dual xeon with 4 gigs of ram and a pair of 250 gig serial ata drives. So - if rsync could watch the load levels and pause every now and then to put a little space between disk access at high load levels it would make it a lot friendlier to the system. The reason nice doesn't work is that once the system call is made to access the disk - nice doesn't apply. What load levels? Do you have some nice C code that can do that for ALL the platforms without misreading? This is what process and i/o schedulers are for. Maybe you should contact the people responsible for whatever kernel it is you are running. What happens is that the server is cooking along just fine serving about 2 million hits a day. Load level - according to top is running around 0.6 to 2.3 or so - and then rsync kicks in doing a backup between the drives and even though I'm running at nice +19 the load kicks up to around 50 and several services almost stop. That's why I'm asking for this feature. By load do you mean that fuzzy approximation of the average number of processes in the run queue. I've been on systems where a load average of 50 is normal and quite zippy. Load average is not a reliable number without context. What you have is a failure of the scheduler. The kernel guys are working on that. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Tru64 sockets bug?
On Fri, Feb 13, 2004 at 01:47:00PM -0500, Jonathan Burelbach wrote: We are using rsync to maintain a warm mirror of our boot disks. I just updated to rsync 2.6.0 and noticed that the rsync runs are now getting errors when copying sockets. The system is running Tru64 5.1A pk 4 on an AlphaServer 4100. Is this a bug? Is there some way to not copy sockets or should I find all of them and add them to the exclude file? rsync was built using a straight ./configure with no options and passed all the tests once built. TIA, Jon This looks like it may be the same problem that freeBSD has. You may want to file a bug report with HP/Compaq/DEC. Rsync treats sockets and pipes as devices so enabling devices (-d, included in -a) enables sockets. So to avoid the warnings you could add the sockets to the exclude list or not enable devices. If syncing the sockets is a priority (unlikely) there are patches in the list archives that do socket and pipe creation using the socket and pipe specific syscalls. rsync --version rsync version 2.6.0 protocol version 27 Copyright (C) 1996-2004 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6, 32-bit system inums, 64-bit internal inums rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. ls -l /usr/var/cluster/members/member0/agentx/master srwxr-xr-x 1 root system 0 Feb 4 13:10 /usr/var/cluster/members/member0/agentx/master= rsync output: building file list ... done adm/rsync.system.log adm/sialog mknod /bkvar/cluster/members/member0/agentx/master failed: Invalid argument cluster/members/member0/evm/evmlog/evmlog.20040213 mknod /bkvar/cluster/members/member0/evm/sockets/evmd failed: Invalid argument cluster/members/member0/run/sudo/jburelba/10 wrote 27550211 bytes read 420 bytes 1281424.70 bytes/sec total size is 555819562 speedup is 20.17 rsync error: some files could not be transferred (code 23) at main.c(632) var rsync completed 02/13/04 13:21:46 -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Sat Feb 14 01:43:24 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv23946 Modified Files: TODO Log Message: Remove Conditional -z for old protocols because those old protocols are no longer supported. Revisions: TODO1.75 = 1.76 http://www.samba.org/cgi-bin/cvsweb/rsync/TODO.diff?r1=1.75r2=1.76 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
Re: error in rsync protocol data stream
On Thu, Feb 12, 2004 at 02:14:12PM -0500, Ray Lischner wrote:
On Thursday 12 February 2004 12:44 pm, Ray Lischner wrote:
I am trying to backup an entire disk to a USB-mounted disk, for backup
purposes. Many files are copied, but eventually rsync dies:
I tried it again, and this time it fails at a different point and with a
different error:
$ rsync -axHSv --delete /backup /media/sda1
...
rsync: writefd_unbuffered failed to write 32768 bytes: phase unknown:
No buffer space available
rsync error: error in rsync protocol data stream (code 12) at io.c(666)
As I mentioned in my previous post, the system has 192MB RAM, 400MB
swap. The file system I am trying to copy takes up about 15GB.
You have run out of memory. Per the SuSE distributed
write(2) manpage ENOBUFS is not supposed to happen but SUSv3
does list such an error.
Try this patch which adds an arbitrary 100ms back-off delay
to hopefully give the system a chance to recover.
Index: io.c
===
RCS file: /data/cvs/rsync/io.c,v
retrieving revision 1.115
diff -u -p -r1.115 io.c
--- io.c26 Dec 2003 23:04:01 - 1.115
+++ io.c12 Feb 2004 23:33:42 -
@@ -655,6 +655,11 @@ static void writefd_unbuffered(int fd,ch
continue;
}
+ if (ret == -1 errno == ENOBUFS) {
+ msleep(100); /* system overload, back off */
+ continue;
+ }
+
if (ret = 0) {
/* Don't try to write errors back
* across the stream */
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: error in rsync protocol data stream
On Thu, Feb 12, 2004 at 10:28:31PM -0500, Ray Lischner wrote: On Thursday 12 February 2004 06:38 pm, jw schultz wrote: You have run out of memory. According to the FAQ-o-matic, rsync needs about 100 bytes/file. I am copying about 1,000,000 files, so that comes out to roughly 100MB. The system has 192MB of physical RAM. Of course, the kernel and all the other daemons and what-not need memory, but there should still be enough for rsync. I was successful when copying fewer files at a time. When I copy 300,000 files, rsync's RSS is 50MB after building the file list, and while it is copying, the RSS rises to about 75MB. I conclude that rsync actually needs as much as 250 bytes/file. This explains why 192MB is insufficient when I want to copy 1M files. That 100 bytes/file is per file usage for the purpose of scaling. Subject to variability according to actual filename lengths and system wordsize, and does not account for the additional 72bytes/file that were needed if -H was in play. Don't confuse per-file with flat overhead and data requirements for one-file-at-a-time transfers that get reused every file. Try the patch and let us know. That should allow you to function. CVS (which does not have that patch) requires about 20 bytes less per-file. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Please help : Strange 'file has vanished' behaviour with rsync on windows.
Your question has diddly-squat to do with the thread or subject line. And no, rsync does not support both ends being remote. You need to make it so one end or the other is local to rsync. If mypc2 and mypc1 have no way to connect one to the other regardless of initiator ther are books and howtos on ssh that will show you how to forward ports or do tunneling. On Wed, Feb 11, 2004 at 05:01:51PM -0800, [EMAIL PROTECTED] wrote: Actually I would want to use ssh and do copy between two remote servers from my machine. But the following did not work: rsync -e ssh -av [EMAIL PROTECTED]:/myshare1/dir1 [EMAIL PROTECTED]:/myshare2/dir2 I was not able to successfully rsync with both, source and destination, as remote locations. I would appreciate if you can tell me whether there is a limitation in rsync or am I doing something wrong? regards, -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.1cvs dumps core when preserve_hard_links and backup
On Tue, Feb 10, 2004 at 01:40:48PM +0900, Shinichi Maruyama wrote:
I tried rsync 2.6.1cvs on my FreeBSD 4.X boxes.
It dumps core when preserve_hard_links and backup.
And SUPPORT_HARD_LINKS is true.
rsync -avH --delete --backup --backup-dir=/some/backup/dir \
srchost::dir/ dir/
The core says...
Program terminated with signal 11, Segmentation fault.
#0 0x80536c0 in keep_backup (fname=0x808f060 rsync.core) at backup.c:266
266 if (!kept preserve_hard_links file-link_u.links
(gdb) where
#0 0x80536c0 in keep_backup (fname=0x808f060 rsync.core) at backup.c:266
#1 0x80537c9 in make_backup (fname=0x808f060 rsync.core) at backup.c:297
#2 0x804c003 in delete_files (flist=0x8096100) at receiver.c:112
#3 0x805028c in do_recv (f_in=3, f_out=3, flist=0x8096100, local_name=0x0)
at main.c:434
#4 0x805093c in client_run (f_in=3, f_out=3, pid=-1, argc=1, argv=0x8095004)
at main.c:666
(gdb) p *file
$1 = {modtime = 1076386251, length = 577536, mode = 33152, u = {rdev = 0, sum = 0x0,
link = 0x0}, link_u = {idev = 0x80d1734, links = 0x80d1734},
basename = 0x80d1744 rsync.core, dirname = 0x0, basedir = 0x0, uid = 0, gid = 0,
flags = 0 '\000'}
In file backup.c.
266 if (!kept preserve_hard_links file-link_u.links
267 hlink_list[file-F_HLINDEX] != file) {
268 if (verbose 1)
269 rprintf(FINFO, %s is a hard link\n, f_name(file));
270 }
file-link_u.links (and also file-F_HLINDEX) has some values,
but no meanings because it's only an union with file-link_u.idev.
It may be
if (!kept preserve_hard_links file-u.link) {
or some ?
CVS has a simplified test now that should avoid the segfault.
It is just for a verbose message.
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: checksum_seed
On Tue, Feb 10, 2004 at 09:56:00PM -0800, Craig Barratt wrote: On Mon, Feb 09, 2004 at 09:14:06AM -0500, Jason M. Felice wrote: I got the go-ahead from the client on my --link-by-hash proposal, and the seed is making the hash unstable. I can't figure out why the seed is there so I don't know whether to cirumvent it in my particular case or calculate a separate, stable hash. I believe the checksum seed is meant to reduce the chance that different data could repeatedly produce the same md4 digest over multiple runs. If a collision happens the hope is that a different checksum seed will break the collision. However, my guess is that it doesn't make any difference. Certainly adding the seed at the end of the block won't change a collision even if the seed changes over multiple runs. File MD4 checksums add the seed at the beginning, which might help breaking collisions, although I'm not sure. Wayne Davison writes: There was some talk last year about adding a --fixed-checksum-seed option, but no consensus was reached. It shouldn't hurt to make the seed value constant for certain applications, though, so you can feel free to proceed in that direction for what you're doing for your client. FYI, I just checked in some changes to the checksum_seed code that will make it easier to have other options (besides the batch ones) specify that a constant seed value is needed. I would really like a --fixed-csumseed option become a standard feature in rsync. Just using the batch value (32761) is fine. Can I contribute a patch? The reason I want this is the next release of BackupPC will support rsync checksum caching, so that backups don't need to recompute block or file checksums. This requires a fixed checksum seed on the remote rsync, hence the need for --fixed-csumseed. I've included this feature in a pre-built rsync for cygwin that I include on the SourceForge BackupPC downloads. 1. Yes, you may contribute a patch. I favor the idea of being able to supply a checksum seed. 2. Lets get the option name down to a more reasonable length. --checksum-seed should be sufficient. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.1cvs dumps core when preserve_hard_links and backup
On Tue, Feb 10, 2004 at 01:40:48PM +0900, Shinichi Maruyama wrote:
I tried rsync 2.6.1cvs on my FreeBSD 4.X boxes.
It dumps core when preserve_hard_links and backup.
And SUPPORT_HARD_LINKS is true.
rsync -avH --delete --backup --backup-dir=/some/backup/dir \
srchost::dir/ dir/
The core says...
Program terminated with signal 11, Segmentation fault.
#0 0x80536c0 in keep_backup (fname=0x808f060 rsync.core) at backup.c:266
266 if (!kept preserve_hard_links file-link_u.links
(gdb) where
#0 0x80536c0 in keep_backup (fname=0x808f060 rsync.core) at backup.c:266
#1 0x80537c9 in make_backup (fname=0x808f060 rsync.core) at backup.c:297
#2 0x804c003 in delete_files (flist=0x8096100) at receiver.c:112
#3 0x805028c in do_recv (f_in=3, f_out=3, flist=0x8096100, local_name=0x0)
at main.c:434
#4 0x805093c in client_run (f_in=3, f_out=3, pid=-1, argc=1, argv=0x8095004)
at main.c:666
(gdb) p *file
$1 = {modtime = 1076386251, length = 577536, mode = 33152, u = {rdev = 0, sum = 0x0,
link = 0x0}, link_u = {idev = 0x80d1734, links = 0x80d1734},
basename = 0x80d1744 rsync.core, dirname = 0x0, basedir = 0x0, uid = 0, gid = 0,
flags = 0 '\000'}
In file backup.c.
266 if (!kept preserve_hard_links file-link_u.links
267 hlink_list[file-F_HLINDEX] != file) {
268 if (verbose 1)
269 rprintf(FINFO, %s is a hard link\n, f_name(file));
270 }
file-link_u.links (and also file-F_HLINDEX) has some values,
but no meanings because it's only an union with file-link_u.idev.
It may be
if (!kept preserve_hard_links file-u.link) {
or some ?
Yep, that is the receiver and it doesn't have access to that
data since Wayne changed hardlink handling to cope with new
links to existing files.
Thoughts Wayne?
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: memory reduction
On Thu, Feb 05, 2004 at 09:27:51PM -0800, jw schultz wrote: As those of you who watch CVS will be aware Wayne has been making progress in reducing memory requirements of rsync. Much of what he has done has been the product of discussions between he and myself that started a month ago with John Van Essen. Most recently Wayne has changed how the file_struct and its associated data are allocated, eliminating the string areas. Most of these changes have been small and relatively low impact although combining the allocation of the file_struct with the strings does impact the memory management of file_struct. Attached is a patch that implements the next step. It alters flist memory management and introduces a MM pool layer that reduces malloc overhead and allows destructors to actually release memory to the OS. The patch adds a couple of new files so use patch -p1 and rerun ./configure after patching. The patch has now been committed. If you use cvs update be sure to rerun ./configure. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Tue Feb 10 03:23:38 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv30417 Modified Files: Makefile.in NEWS backup.c batch.c flist.c hlink.c proto.h receiver.c rsync.h Log Message: Make idev, hlink and file_struct + strings use allocation pools. Revisions: Makefile.in 1.96 = 1.97 http://www.samba.org/cgi-bin/cvsweb/rsync/Makefile.in.diff?r1=1.96r2=1.97 NEWS1.151 = 1.152 http://www.samba.org/cgi-bin/cvsweb/rsync/NEWS.diff?r1=1.151r2=1.152 backup.c1.21 = 1.22 http://www.samba.org/cgi-bin/cvsweb/rsync/backup.c.diff?r1=1.21r2=1.22 batch.c 1.29 = 1.30 http://www.samba.org/cgi-bin/cvsweb/rsync/batch.c.diff?r1=1.29r2=1.30 flist.c 1.198 = 1.199 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c.diff?r1=1.198r2=1.199 hlink.c 1.36 = 1.37 http://www.samba.org/cgi-bin/cvsweb/rsync/hlink.c.diff?r1=1.36r2=1.37 proto.h 1.182 = 1.183 http://www.samba.org/cgi-bin/cvsweb/rsync/proto.h.diff?r1=1.182r2=1.183 receiver.c 1.70 = 1.71 http://www.samba.org/cgi-bin/cvsweb/rsync/receiver.c.diff?r1=1.70r2=1.71 rsync.h 1.182 = 1.183 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.h.diff?r1=1.182r2=1.183 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/lib
Date: Tue Feb 10 03:26:41 2004 Author: jw Update of /data/cvs/rsync/lib In directory dp.samba.org:/tmp/cvs-serv31354/lib Added Files: pool_alloc.3 pool_alloc.c pool_alloc.h Log Message: Added allocation pool code. Revisions: pool_alloc.3NONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsync/lib/pool_alloc.3?rev=1.1 pool_alloc.cNONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsync/lib/pool_alloc.c?rev=1.1 pool_alloc.hNONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsync/lib/pool_alloc.h?rev=1.1 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Feb 10 03:54:47 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv3541 Modified Files: NEWS flist.c main.c rsync.h Log Message: Dropped support for protocol versions less than 20 (2.3.0 released 15 Mar 1999) and activated warnings for protocols less than 25 (2.5.0 released 23 Aug 2001) Revisions: NEWS1.152 = 1.153 http://www.samba.org/cgi-bin/cvsweb/rsync/NEWS.diff?r1=1.152r2=1.153 flist.c 1.199 = 1.200 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c.diff?r1=1.199r2=1.200 main.c 1.191 = 1.192 http://www.samba.org/cgi-bin/cvsweb/rsync/main.c.diff?r1=1.191r2=1.192 rsync.h 1.183 = 1.184 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.h.diff?r1=1.183r2=1.184 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
Re: Full backup
On Fri, Feb 06, 2004 at 09:49:16AM +0100, Thorsten Schacht wrote: Hey guys, I'd like to take a full backup of our email server. Is it possible to clone the current server (postfix, spamassassin, qpopper...) to another clean system to have it ready if the current one fails? Yes it is possible, sort-of. Rsync could be employed in syncing the configurations. Dealing with the queues and mailstore are completely different matters and should be viewed as a clustering issue. Distinctions should also be made between gateways, routers, filters and repositories and other types of email servers. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: memory reduction
On Thu, Feb 05, 2004 at 09:27:51PM -0800, jw schultz wrote:
As those of you who watch CVS will be aware Wayne has been
making progress in reducing memory requirements of rsync.
Much of what he has done has been the product of discussions
between he and myself that started a month ago with John Van
Essen.
Most recently Wayne has changed how the file_struct and its
associated data are allocated, eliminating the string areas.
Most of these changes have been small and relatively low
impact although combining the allocation of the file_struct
with the strings does impact the memory management of
file_struct.
Attached is a patch that implements the next step. It
alters flist memory management and introduces a MM pool
layer that reduces malloc overhead and allows destructors to
actually release memory to the OS.
The patch adds a couple of new files so use patch -p1
and rerun ./configure after patching.
Here is an update of the patch to cope with patch conflicts
re CVS HEAD.
diff -rupNP --exclude-from cvs/.ignore cvs/Makefile.in pool2/Makefile.in
--- cvs/Makefile.in Tue Feb 3 16:34:24 2004
+++ pool2/Makefile.in Tue Feb 3 16:33:00 2004
@@ -27,7 +27,7 @@ [EMAIL PROTECTED]@
HEADERS=byteorder.h config.h errcode.h proto.h rsync.h
LIBOBJ=lib/wildmatch.o lib/compat.o lib/snprintf.o lib/mdfour.o \
- lib/permstring.o @LIBOBJS@
+ lib/permstring.o lib/pool_alloc.o @LIBOBJS@
ZLIBOBJ=zlib/deflate.o zlib/infblock.o zlib/infcodes.o zlib/inffast.o \
zlib/inflate.o zlib/inftrees.o zlib/infutil.o zlib/trees.o \
zlib/zutil.o zlib/adler32.o
diff -rupNP --exclude-from cvs/.ignore cvs/backup.c pool2/backup.c
--- cvs/backup.cWed Feb 4 03:49:36 2004
+++ pool2/backup.c Wed Feb 4 22:11:26 2004
@@ -189,6 +189,7 @@ static int keep_backup(char *fname)
backup_dir[--backup_dir_len] = '\0';
if (verbose 0)
rprintf(FINFO, backup_dir is %s\n, backup_dir);
+
initialised = 1;
}
@@ -199,7 +200,7 @@ static int keep_backup(char *fname)
if (do_stat(fname, st)) return 1;
#endif
- file = make_file(fname, NO_EXCLUDES);
+ file = make_file(fname, NULL, NO_EXCLUDES);
/* the file could have disappeared */
if (!file) return 1;
@@ -282,7 +283,7 @@ static int keep_backup(char *fname)
}
}
set_perms(keep_name, file, NULL, 0);
- free_file(file, FREE_STRUCT);
+ free(file);
if (verbose 1)
rprintf(FINFO, keep_backup %s - %s\n, fname, keep_name);
diff -rupNP --exclude-from cvs/.ignore cvs/batch.c pool2/batch.c
--- cvs/batch.c Fri Feb 6 01:54:56 2004
+++ pool2/batch.c Fri Feb 6 02:00:58 2004
@@ -136,9 +136,7 @@ struct file_list *create_flist_from_batc
exit_cleanup(1);
}
- batch_flist = new(struct file_list);
- if (!batch_flist)
- out_of_memory(create_flist_from_batch);
+ batch_flist = flist_new(WITH_HLINK, create_flist_from_batch);
save_read = stats.total_read;
save_pv = protocol_version;
@@ -150,9 +148,9 @@ struct file_list *create_flist_from_batc
for (i = 0; (flags = read_byte(f)) != 0; i++) {
if (protocol_version = 28 (flags XMIT_EXTENDED_FLAGS))
flags |= read_byte(f) 8;
- receive_file_entry(batch_flist-files[i], flags, f);
+ receive_file_entry(batch_flist-files[i], flags, batch_flist, f);
}
- receive_file_entry(NULL, 0, 0); /* Signal that we're done. */
+ receive_file_entry(NULL, 0, NULL, 0); /* Signal that we're done. */
protocol_version = save_pv;
stats.total_read = save_read;
diff -rupNP --exclude-from cvs/.ignore cvs/flist.c pool2/flist.c
--- cvs/flist.c Fri Feb 6 01:55:41 2004
+++ pool2/flist.c Fri Feb 6 02:00:59 2004
@@ -76,7 +76,6 @@ static unsigned int min_file_struct_len;
static void clean_flist(struct file_list *flist, int strip_root, int no_dups);
static void output_flist(struct file_list *flist);
-
void init_flist(void)
{
struct file_struct f;
@@ -507,7 +506,8 @@ void send_file_entry(struct file_struct
-void receive_file_entry(struct file_struct **fptr, unsigned short flags, int f)
+void receive_file_entry(struct file_struct **fptr, unsigned short flags,
+struct file_list *flist, int f)
{
static time_t modtime;
static mode_t mode;
@@ -624,12 +624,12 @@ void receive_file_entry(struct file_stru
idev_len = 0;
sum_len = always_checksum S_ISREG(mode) ? MD4_SUM_LENGTH : 0;
- file_struct_len = idev_len? sizeof file[0] : min_file_struct_len;
+ file_struct_len = min_file_struct_len;
alloc_len = file_struct_len + dirname_len + basename_len
- + linkname_len + sum_len + idev_len;
- if (!(bp = new_array(char, alloc_len)))
- out_of_memory(receive_file_entry
Re: Rsync doesn't appear to handle large files
On Fri, Feb 06, 2004 at 02:04:45PM -0800, Buwalda, Mark W wrote: I just started using rsync version 2.5.6 on sun solaris 8. In trying to mirror a large ufs filesystem, I've noticed that Rsync does not copy large files, specifically any file greater Than 2 GB. All other files in the filesystem were mirrored properly. Is this a bug/limitation of rsync on solaris, or is there A workaround for this ? Rsync works fine with huge files (4GB). It may be that your, only 3 months obsolete, binary of rsync was built incorrectly. It might also be that the destination filesystem was built or mounted without large file support. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync_failed
On Mon, Feb 02, 2004 at 02:27:29PM +, [EMAIL PROTECTED] wrote: Hello I came about the following phenomeon when the server ( dell poweredge 2650 ) is rsyncing to the same and remote server. the connection was close and copying the file didn't finish yet. Network Topology: Dell2650 L2SW---L3SW---Dell2650 L2SW: Procurve4108gl L3SW: BigIron15000 when the unexpected failed come about, the ping of end-to-end is no problem, and telnet is O.K when I look at the output message of netstat -i the statistics of errors and collision packet was pointing to 0 so it looks like no problem at all. I'll put the log # ./bin/PD_rsync.sh rsync: error writing 4096 unbuffered bytes - exiting: Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(463) rsync: writefd_unbuffered failed to write 4092 bytes: phase unknown: Broken pipe rsync: connection unexpectedly closed (13452651072 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(515) rsync error: received SIGUSR1 or SIGINT (code 20) at main.c(867) what kind of things can you guess ? A Broken pipe means the process at the other end of the pipe terminated. This could be because of an error or it could be because the session got dropped. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Fri Feb 6 10:00:33 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv25233 Modified Files: batch.c flist.c proto.h rsync.h Log Message: Start flist with a more reasonable size, grow it linearly once it reaches a largish size (16 million files) and make it actually match the comments so it may be used to initialize flists of known size (create_flist_from_batch()). Revisions: batch.c 1.28 = 1.29 http://www.samba.org/cgi-bin/cvsweb/rsync/batch.c.diff?r1=1.28r2=1.29 flist.c 1.197 = 1.198 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c.diff?r1=1.197r2=1.198 proto.h 1.181 = 1.182 http://www.samba.org/cgi-bin/cvsweb/rsync/proto.h.diff?r1=1.181r2=1.182 rsync.h 1.180 = 1.181 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.h.diff?r1=1.180r2=1.181 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
Re: rsync is not copying upper case files
On Thu, Feb 05, 2004 at 02:10:39PM +0100, [EMAIL PROTECTED] wrote: I am running rsync on MacOS 10.3 and I want to transfer files to a linux machine, which has linux partitions and fat32 partitions. Transfering the files to the linux partition works without problems, but if I transfer them to the fat32 partition, files are ignored if all characters in the file name are UPPER CASE. - The files which rsync should transfer do not exist in the target directory. Thus its not a problem with the time resolution of the fat32 system. - The files do not appear as lower case, they are simply not there at all. - If a file name contains one lower case character or special character, then it is copied without problems. Rsync treats filenames as null terminated, / delemited, blobs. It is up to the OS to handle any case or illegal characters. This sounds like a FAT32 bug but there may be a mount option involved. Eliminate OSX from the problem space and contact the Linux FAT32 maintainers. Hi, I have eliminated OSX from the problem space since the same error occures from linux to linux. The only difference is that the linux rsync gives me the (probably unrelated) error message: failed to set permissions on test-sync : Operation not permitted failed to set permissions on test-sync : Operation not permitted rsync error: partial transfer (code 23) at main.c(578) This error message does not occur when transfering to a linux partition, but only when transfering to a fat32 partition. But I could encircle the problem from another side: The problem only occures if I use the --delete-after option in the following command: rsync -aL --delete --delete-after -e ssh test-sync [EMAIL PROTECTED]:/mnt/hda8/backup If I don't use the --delete-after option, then the files are copied properly from linux to linux partition and to the fat32 partition. If I use the --delete-after, then UPPER CASE file, when transfered to the fat32 partition are not copied and even deleted, if they already exist. So, I think it is due to the fact that an all UPPER CASE file name is converted to an all lower case file name on the fat32 partition, which then, since the original file name was different, is instantly deleted when --delete-after was chosen. So my guess is that this could cause the problem. Good guess. Rsync requires a filesystem to not modify file names. I can understand the intention in translating allcaps filenames to lower case while preserving mixed case filenames but there should be a mount option to turn that off. The second part of my advice still applies: contact the Linux FAT32 maintainers. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
memory reduction
As those of you who watch CVS will be aware Wayne has been
making progress in reducing memory requirements of rsync.
Much of what he has done has been the product of discussions
between he and myself that started a month ago with John Van
Essen.
Most recently Wayne has changed how the file_struct and its
associated data are allocated, eliminating the string areas.
Most of these changes have been small and relatively low
impact although combining the allocation of the file_struct
with the strings does impact the memory management of
file_struct.
Attached is a patch that implements the next step. It
alters flist memory management and introduces a MM pool
layer that reduces malloc overhead and allows destructors to
actually release memory to the OS.
The patch adds a couple of new files so use patch -p1
and rerun ./configure after patching.
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
diff -rupNP --exclude-from cvs/.ignore cvs/Makefile.in pool2/Makefile.in
--- cvs/Makefile.in Tue Feb 3 16:34:24 2004
+++ pool2/Makefile.in Tue Feb 3 16:33:00 2004
@@ -27,7 +27,7 @@ [EMAIL PROTECTED]@
HEADERS=byteorder.h config.h errcode.h proto.h rsync.h
LIBOBJ=lib/wildmatch.o lib/compat.o lib/snprintf.o lib/mdfour.o \
- lib/permstring.o @LIBOBJS@
+ lib/permstring.o lib/pool_alloc.o @LIBOBJS@
ZLIBOBJ=zlib/deflate.o zlib/infblock.o zlib/infcodes.o zlib/inffast.o \
zlib/inflate.o zlib/inftrees.o zlib/infutil.o zlib/trees.o \
zlib/zutil.o zlib/adler32.o
diff -rupNP --exclude-from cvs/.ignore cvs/backup.c pool2/backup.c
--- cvs/backup.cWed Feb 4 03:49:36 2004
+++ pool2/backup.c Wed Feb 4 22:11:26 2004
@@ -189,6 +189,7 @@ static int keep_backup(char *fname)
backup_dir[--backup_dir_len] = '\0';
if (verbose 0)
rprintf(FINFO, backup_dir is %s\n, backup_dir);
+
initialised = 1;
}
@@ -199,7 +200,7 @@ static int keep_backup(char *fname)
if (do_stat(fname, st)) return 1;
#endif
- file = make_file(fname, NO_EXCLUDES);
+ file = make_file(fname, NULL, NO_EXCLUDES);
/* the file could have disappeared */
if (!file) return 1;
@@ -282,7 +283,7 @@ static int keep_backup(char *fname)
}
}
set_perms(keep_name, file, NULL, 0);
- free_file(file, FREE_STRUCT);
+ free(file);
if (verbose 1)
rprintf(FINFO, keep_backup %s - %s\n, fname, keep_name);
diff -rupNP --exclude-from cvs/.ignore cvs/batch.c pool2/batch.c
--- cvs/batch.c Thu Feb 5 20:47:31 2004
+++ pool2/batch.c Thu Feb 5 20:51:14 2004
@@ -136,9 +136,7 @@ struct file_list *create_flist_from_batc
exit_cleanup(1);
}
- batch_flist = new(struct file_list);
- if (!batch_flist)
- out_of_memory(create_flist_from_batch);
+ batch_flist = flist_new(WITH_HLINK, create_flist_from_batch);
save_read = stats.total_read;
save_pv = protocol_version;
@@ -153,9 +151,9 @@ struct file_list *create_flist_from_batc
for (i = 0; (flags = read_byte(f)) != 0; i++) {
if (protocol_version = 28 (flags XMIT_EXTENDED_FLAGS))
flags |= read_byte(f) 8;
- receive_file_entry(batch_flist-files[i], flags, f);
+ receive_file_entry(batch_flist-files[i], flags, batch_flist, f);
}
- receive_file_entry(NULL, 0, 0); /* Signal that we're done. */
+ receive_file_entry(NULL, 0, NULL, 0); /* Signal that we're done. */
protocol_version = save_pv;
stats.total_read = save_read;
diff -rupNP --exclude-from cvs/.ignore cvs/flist.c pool2/flist.c
--- cvs/flist.c Wed Feb 4 17:39:32 2004
+++ pool2/flist.c Thu Feb 5 18:41:21 2004
@@ -76,13 +76,12 @@ static unsigned int min_file_struct_len;
static void clean_flist(struct file_list *flist, int strip_root, int no_dups);
static void output_flist(struct file_list *flist);
-
void init_flist(void)
{
-struct file_struct f;
+ struct file_struct f;
-/* Figure out how big the file_struct is without trailing padding */
-min_file_struct_len = ((char*)f.flags - (char*)f) + sizeof f.flags;
+ /* Figure out how big the file_struct is without trailing padding */
+ min_file_struct_len = ((char*)f.flags - (char*)f) + sizeof f.flags;
}
@@ -499,7 +498,8 @@ void send_file_entry(struct file_struct
-void receive_file_entry(struct file_struct **fptr, unsigned short flags, int f)
+void receive_file_entry(struct file_struct **fptr, unsigned short flags,
+struct file_list *flist, int f)
{
static time_t modtime;
static mode_t mode;
@@ -616,12 +616,12 @@ void receive_file_entry(struct file_stru
idev_len = 0;
CVS update: rsync
Date: Fri Feb 6 04:51:09 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv2345 Modified Files: batch.c Log Message: Correct sizeof usage and other whitespace. Revisions: batch.c 1.27 = 1.28 http://www.samba.org/cgi-bin/cvsweb/rsync/batch.c.diff?r1=1.27r2=1.28 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
Re: ACL and Quota
On Wed, Feb 04, 2004 at 09:32:58AM -0800, Norman Zhang wrote: Hi, I have been grabbing mirrors using -aupg switches. Is there a way rsync to copy the ACLs and quota restrictions for particular folders too? At this time rsync does not support ACLs or Extended Attributes. To copy them you will need to wrap rsync. A patch was posted a few months ago that added some ACL support. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: excludes
On Wed, Feb 04, 2004 at 04:33:20PM -0800, tim wrote: Kevin Moschallski wrote: Hi i tried to use rsync to backup my remote server.I want to exclude folder like /var /proc ... .I use --exlude-from=/etc/rsync/excludes the file looks like this: - */var/* - */proc/* etc. My problem is that i run vservers on this machine and i don't want that the folder like /proc in this vservers are backuped how can i exclude all paths wich contain such foldernames? When using rsync to maintain incremental backups significant space can be saved on the backup server by excluding directories that have non-critical or transient data such as /tmp, /proc, /mnt, /cdrom, object and backup files, and caches. The problem is getting rsync to include the directory name but exclude its contents, a condition not clearly covered in rsync(1), EXCLUDE PATTERNS. That isn't difficult nor do i find it to be poorly explained in the current manpage. To exclude the contents of a directory such as /tmp but still make the directory just exclude /tmp/* There is no need to explicitly include it unless another pattern would match. If you want to do the same for all directories named cache wherever they are in the tree exclude **/cache/* A trailing ** has excactly the same effect as a trailing * I use a separate backup server and pull from LAN clients but any model should work. To test, add '-n -vv' to your rsync command and grep for exclude and include statements. Feedback: rsync at nsr500 dot net. Remote client example: rsync -nvva --stats --delete --exclude-from=file -e ssh \ client:/ /rsync_dir/client/ | egrep '^excl|^incl' The following rsync exclude file works on my RH 7.3 (2.4.23) and RH 6.2 (2.2.25) based systems for rsync 2.5.4. You may want to edit to suit your specific needs. Most of your + lines could be dropped. I left the coments in place. ### BEGIN rsync exclude file ### directory entries, no directory contents ### tmp, anywhere in tree ### + tmp/ - **/tmp/** ### transient, from root (/) ### + /proc/ - /proc/** + /mnt/ - /mnt/** + /cdrom/ - /cdrom/** + /tmpfs/ - /tmpfs/** ### cache, anywhere in tree ### # mozilla + Cache/ - **/Cache/** # ccache.samba.org + ccache/ - **/ccache/** + .ccache/ - **/.ccache/** # others + cache/ - **/cache/** + xover-cache/ - **/xover-cache/** ### obj ### # kernel - usr/src/**.o # special library .o (may be RH specific?) + usr/*/lib/**.o + usr/lib/**.o # all others - *.o ### backup ### - *~ ### END rsync exclude file ### BEGIN rsync exclude file ### directory entries, no directory contents ### tmp, anywhere in tree ### - **/tmp/* ### transient, from root (/) ### - /proc/* - /mnt/* - /cdrom/* - /tmpfs/* ### cache, anywhere in tree ### # mozilla - **/Cache/* # ccache.samba.org - **/ccache/* - **/.ccache/* # others - **/cache/* - **/xover-cache/* ### obj ### # special library .o (may be RH specific?) + usr/*/lib/**.o + usr/lib/**.o # all others - *.o ### backup ### - *~ ### END rsync exclude file -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HELP!!!
On Tue, Feb 03, 2004 at 12:51:32AM -0800, preetha thomas wrote: Hello and Greetings!! I am Preetha Thomas.I am a student and trying to use RSYNC for the download of the PDB FTP archives.I have been successful.I posted a question last wekk about updating my pdb archive.But i never heard from anyone on the listand my problem goes unsolved. SO please do help wiht my problem...I wanna know if RSYNCprovides any script so that I can update my downloaded PDB FTP archive with the new entries that keep pouring in every weekOr do I have to write a script myself.. please do reply at the earliest. Thankyou in advance Before posting again, read: http://www.catb.org/~esr/faqs/smart-questions.html -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync is not copying upper case files
On Tue, Feb 03, 2004 at 02:59:02PM +0100, [EMAIL PROTECTED] wrote: Hi, I am running rsync on MacOS 10.3 and I want to transfer files to a linux machine, which has linux partitions and fat32 partitions. Transfering the files to the linux partition works without problems, but if I transfer them to the fat32 partition, files are ignored if all characters in the file name are UPPER CASE. - The files which rsync should transfer do not exist in the target directory. Thus its not a problem with the time resolution of the fat32 system. - The files do not appear as lower case, they are simply not there at all. - If a file name contains one lower case character or special character, then it is copied without problems. Rsync treats filenames as null terminated, / delemited, blobs. It is up to the OS to handle any case or illegal characters. This sounds like a FAT32 bug but there may be a mount option involved. Eliminate OSX from the problem space and contact the Linux FAT32 maintainers. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.0 causing incredible load on Linux 2.4.x?
On Mon, Feb 02, 2004 at 03:00:11PM -0800, Wayne Davison wrote: On Mon, Feb 02, 2004 at 02:01:15PM -0800, Clint Byrum wrote: Oh, and one other thing... whats up with this: -rwxr-xr-x1 root root 1294978 Jan 30 12:54 /usr/local/rsync-2.6.0/bin/rsync -rwxr-xr-x1 root root 216785 Dec 3 20:42 /usr/bin/rsync Run ldd on those and check to see if 2.6.0 was linked with static or dynamic libraries. They're both stripped binaries in case thats what you were thinking. The size you cite for 2.6.0 is very near to the size of an unstripped rsync on my system, so I'd be sure to double-check that. And roughly double my sizes for near 2.6.0 (CVS head 28-dec) rsync: -rwxr-xr-x1 jw jw 180964 Feb 2 15:08 rsync.stripped -rwxr-xr-x1 jw jw 592537 Feb 2 15:08 rsync.unstripped -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.0 causing incredible load on Linux 2.4.x?
On Mon, Feb 02, 2004 at 05:35:17PM -0800, Clint Byrum wrote: On Mon, 2004-02-02 at 14:51, jw schultz wrote: On Mon, Feb 02, 2004 at 02:01:15PM -0800, Clint Byrum wrote: Hi everyone. Has anyone experienced rsync 2.6.0 causing huge amounts of system load? Especially on Linux 2.4? snip There aren't too many things that would account for this. The only one i can think of is an exclude pattern that causes pathalogical behavior with the new pattern matching code. 2.6.0 also defaults to ssh and if you have ssh doing compression by default that could affect it. You don't show your command-line so nothing else comes to mind. rsync -Wa --exclude=*tmp* --exclude=SYSLOCKS* --delete --delete-after /localsource $box::dest/dir That looks like pretty normal exclude stuff to me. I'm copying the entire dataset over to a throw-away box to test the idea, since I would imagine part of the problem would be the file list itself. That will take me a while though. Very normal. I can't think of any changes in rsync that would make that much adverse difference. The only thing left is coincidence: something else pushing you over the edge while rsync was stressing the system. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Tue Feb 3 03:42:49 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv23997 Modified Files: flist.c receiver.c sender.c Log Message: Corrected sizeof usage: sizeof obj sizeof (type) Revisions: flist.c 1.189 = 1.190 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c.diff?r1=1.189r2=1.190 receiver.c 1.69 = 1.70 http://www.samba.org/cgi-bin/cvsweb/rsync/receiver.c.diff?r1=1.69r2=1.70 sender.c1.36 = 1.37 http://www.samba.org/cgi-bin/cvsweb/rsync/sender.c.diff?r1=1.36r2=1.37 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
Re: innetgr revised netgroup patch against 2.6.0
On Sun, Feb 01, 2004 at 11:25:22PM +1100, Travis Freeland wrote:
innetgr.. much easier..
Much cleaner. But the code could be tightened further.
I've provided an example. Note also formatting changes.
Also, we use tabs, not spaces for indent. If that is your
mailer either fix the config or use text/plain (not x-diff)
attachments.
Had a look for the user section but didn't find it in my 15 seconds of
looking..
Do the user, user+host and the autoconf and I'll consider
this for CVS.
--- access.c2003-07-30 16:12:27.0 +1000
+++ ../rsync-2.6.0-Linux/access.c 2004-02-01 23:21:12.0 +1100
@@ -22,10 +22,21 @@
*/
#include rsync.h
+#include netdb.h
static int match_hostname(char *host, char *tok)
{
+ char *netgroup;
+
+ if(strlen(tok)1){
+ if(tok[0]=='@'){
+ netgroup=tok+1;
+ if(innetgr(netgroup, host, NULL, NULL)){
+ return(1);
+ }
+ }
+ }
if (!host || !*host) return 0;
return wildmatch(tok, host);
}
--- access.c2003-07-30 16:12:27.0 +1000
+++ ../rsync-2.6.0-Linux/access.c 2004-02-01 23:21:12.0 +1100
@@ -22,10 +22,21 @@
*/
#include rsync.h
+#include netdb.h
static int match_hostname(char *host, char *tok)
{
+ if (tok[0] == '@' tok[1]) {
+ if (innetgr(tok[1], host, NULL, NULL))
+ return(1);
+ }
if (!host || !*host) return 0;
return wildmatch(tok, host);
}
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] --links-depth for rsync
On Sat, Jan 24, 2004 at 11:09:46PM +0200, Antti Tapaninen wrote: Hello, about a year ago I ran into situation where there's a metadirectory containing directories and symlinks to files. There was a need to mirror the contents of files and directories gathered via symlinks to this metadirectory, regular mirroring of the tree wouldn't do any good. The attached patch gives the user ability to define how many symbolic links rsync should follow before actually adding it to the file list. Any code behaviour without the new parameter is unchanged, I've used the previous versions of the patch with rsync 2.5.x for the last year and might just as well share the patch. Consider merging the patch for the next release, please reply to me directly for additional feedback or issues. Cheers, -Antti Thanks for updating the patch for those who might have need of it. I don't anticipate committing this at this time. flist.c is in flux and 1.169 is in the midst of that so you might want to rediff against rsync 2.6.1 when that is released. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --daemon for clients
On Thu, Jan 29, 2004 at 04:58:45PM +0100, Stefan Hoefer wrote: Hello I have the following situation: one perl script opens a TCP connection to a remote server running another perl script. The file descriptors on both sides are mapped to STDIN and STDOUT file descriptors. After some talking, I would like to start rsync using the exec call on both sides of the connection, with both rsync partners using the existing connection (inherited on STDIN and STDOUT). On the server side, this is no problem (I can use the --daemon switch). Is there any way rsync can perform this on the client side, too? Thanks for any help I have a vague recollection that of there being a patch that would allow rsync to connect to open file descriptors. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: netgroups patch for rsync 2.6.0
On Sat, Jan 31, 2004 at 11:39:11PM +1100, Travis Freeland wrote: G'day, I figured some others might benefit from this patch. It allows you to use the samba style @netgroup names in hosts allow and hosts deny. I've tested it on solaris and linux with no apparent problems.. I'll get rid of the static string in a couple of days when I have to worry about trying to get this into prod .. but with any luck someone else may have done that for me.. :) Very nice idea and good of you to produce a patch. With improvements this would have a good chance of inclusion. This would have to be conditional on an #ifdef from autoconf regarding the existance of netgroups. The code could be tightened a bit. To start with, the sscanf is unnecessary. I've not done any netgroup programming and lack manpages but it seems that iterating over getnetgrent() is the wrong approach, especially since netgroup uses recursive data, and something like a single call to innetgr() would be more appropriate. It seems to me that if we are to add netgroups it should also apply to auth users as well. And given the selectivity of netgroups the user authorization should permit the by-host restrictions. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: different behavior for --backup-dir relative/path vs --backup-dir /full/path
On Thu, Jan 29, 2004 at 11:40:50AM -0800, [EMAIL PROTECTED] wrote: Why does using --delete -b --backup-dir relative/path puts the deleted files in the target folder, while using the full path works as expected? What you describe is exactly as expected. Like --link-dest and --compare-dest, relative paths for --backup-dir are relative to the destination directory. Why this is so becomes obvious if you imagine doing a push to a remote server. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.6, No buffer space available error...
On Fri, Jan 30, 2004 at 11:31:42AM +0100, Kate Ward wrote: On Wed, Sep 03, 2003 at 03:49:12PM -0400, Matt Miller wrote: I have seen this error in many postings, but the solutions seem not to apply in this case. This error happens during a small file transfer (gif image) and after 75% of a 165G rsync job (the building file list portion is complete.) Most solutions I saw were related to not having enough RAM or running out of disk space. Check out the machine specs below the error. rsync: writefd_unbuffered failed to write 4830 bytes: phase unknown: No buffer space available rsync error: error in rsync protocol data stream (code 12) at io.c(515) rsync: writefd_unbuffered failed to write 69 bytes: phase unknown: Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(515) uptime reports 11:17pm up 8:43, 2 users, load average: 1.65, 1.40, 1.33 IBM x330 server 2x1.4GHz PIII 4G RAM RedHat 7.2 (2.4.7-10smp -- required for veritas) rsync 2.5.6 local vxfs source and destination (different filesystems) and destination has 800G of free space (ideally the source will be on another machine, but I made it local to simplify troubleshooting... same problem with an NFS source) rsync command used: rsync -Wav --stats /sourcedir /destdir /opt/rsync.log 21 and syslog reports: Aug 31 04:07:57 moss kernel: ENOMEM in do_get_write_access, retrying. Aug 31 04:16:05 moss kernel: journal_write_metadata_buffer: ENOMEM at get_unused_buffer_head, trying again. anyone seen this? Not an rsync problem. This is a kernel issue. You seem to be out of ZONE_DMA memory. I don't know whether you should be reporting this to RH or Veritas or IBM but when you do be sure to send them /proc/slabinfo from the moment of error. One thing you might try is booting with mem=900M Running with more than 900M n 1G memory on 2.4 kernels is problematic. I am experiencing this same problem with rsync 2.5.6 and 2.5.7. The difference for me is I am running Solaris 8, not Linux. I am using the same procedure to rsync ten different servers to a shared NFS storage, but one of those ten servers comes up with the following error (very similar to above). rsync error: timeout in data send/receive (code 30) at io.c(103) rsync: writefd_unbuffered failed to write 69 bytes: phase unknown: Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(515) Just wanted to mention it was not Linux specific. Yours is not the same problem. No buffer space available means he ran out of memory. But that isn't what your error message says. You have a broken pipe which means that the process on the other end of the pipe died. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Can't Figure out why rsync job stops
On Thu, Jan 29, 2004 at 03:36:51PM -0600, Bryan Sauser wrote: I connecting to two offsite over servers, that are connected over dedicated T1 lines. I'm using the same script on both servers. One runs fine, but the other starts, gets the file list and processes a few folders. Then it will hang for about 5 minutes before sptting out the following errors: receiving file list ... 16756 files to consider dslagel/ dslagel/DRIVERS/ dslagel/DRIVERS/ATP_PKG/ dslagel/DRIVERS/ATP_PKG/Desktop/ dslagel/DRIVERS/ATP_PKG/System32/ dslagel/DRIVERS/ATP_PKG/Temp/ dslagel/DRIVERS/ATP_PKG/Windows/ dslagel/DRIVERS/ATP_PKG/Windows/PCHEALTH/ dslagel/DRIVERS/ATP_PKG/Windows/PCHEALTH/HELPCTR/ dslagel/DRIVERS/ATP_PKG/Windows/PCHEALTH/HELPCTR/BATCH/ dslagel/DRIVERS/ATP_PKG/Windows/help/ dslagel/DRIVERS/ATP_PKG/chms/ dslagel/DRIVERS/ATP_PKG/icons/ dslagel/DRIVERS/ATP_PKG/images/ dslagel/DRIVERS/ATP_PKG/images/large/ dslagel/DRIVERS/ATP_PKG/images/small/ dslagel/DRIVERS/W2K/ dslagel/DRIVERS/W2K/DISPLAY/ dslagel/DRIVERS/W2K/DISPLAY/BIN/ dslagel/DRIVERS/W2K/DISPLAY/PHILDEC/ dslagel/DRIVERS/W2K/DISPLAY/PHILDEC/WDM_2K/ dslagel/DRIVERS/W2K/DISPLAY/R62K/ dslagel/DRIVERS/W2K/DISPLAY/R62K/ATIDRIVE/ rsync: writefd_unbuffered failed to write 4092 bytes: phase unknown: Connection timed out rsync: connection unexpectedly closed (294104 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(666) rsync error: received SIGUSR1 or SIGINT (code 20) at main.c(886) I have tried making the job smaller. Changing to different folder with different file, but It always acts like this. I am running rsync version 2.6.0 protocol version 27 on all of the servers. My to offsite servers, and local backup server. Here is the OS information for the servers. Local: Linux version 2.4.18-14smp ([EMAIL PROTECTED]) (gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)) Working Offsite: Linux version 2.4.20-18.7 ([EMAIL PROTECTED]) (gcc version 2.96 2731 (Red Hat Linux 7.3 2.96-113)) Problem Offsite: Linux version 2.4.20-18.9 ([EMAIL PROTECTED]) (gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) Here is a copy of the script that I'm running. /usr/local/bin/rsync -avz --stats --progress rsync://[EMAIL PROTECTED]:873:/mercybackupstwo/dslagel /unmirror/mercy I am looking for pointer in the rights direction. I can't seem to find any more specific info on these errors out there. This looks a great deal like a dropped session of the sort one would see over a firewall or VPN. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Change in reporting for --dry-run in 2.6.x
On Wed, Jan 28, 2004 at 12:38:11PM -0500, Alberto Accomazzi wrote: I just noticed that there is an extra blank line in the output generated by rsync when the --dry-run (-n) flag is used. This seems to have started with 2.6.0. Is this desired? The reason why I'm asking is because I use scripts that parse the output from rsync and little modifications in verbosity can make or break things easily. Yes it is desired. It also got a mention in NEWS. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Server crashed using rsync
On Wed, Jan 28, 2004 at 02:12:02PM -0500, Eugen Luca wrote: I'm trying to make a backup using this command rsync -auvH /home/ /bak --delete --bwlimit=1000 --status server load has been increased so much and the server crashed, as well has gone out of memory My Server is a Dual Xeon 2.0 GHz with 2GB of Memory + 1GB Swap. Could be that there are too many files, about 5.000.000, to be backed up The file list size is part of your problem particularly for local transfer. At approximately 100 bytes per file you are near the limits of per processs allocatable space. Add 72 bytes per file (non CVS HEAD) for the -H on the generator and it gets worse. Now factor in the fact that for local transfer the file list (but not hlink list) is built twice (not shared) and you are pushing/exceeding 1GB RSS just for rsync. If this causes the OS to crash you have an unrelated problem that rsync i just revealing. ? The way the files are structured make very difficult to create several backup with less files. Is there any one else facing issues like this ? Is there any workaround on this ? Many people have faced this issue. I'd suggest perusing the archives. There are enhancements in CVS that reduce the increased memory requirements of -H. How it may be worked around will depend on your fileset. You have already indicated that your structure makes breaking up job difficult. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: File rename detection?
On Thu, Jan 29, 2004 at 11:42:12AM +1100, Donovan Baarda wrote: On Thu, 2004-01-29 at 06:27, jw schultz wrote: On Wed, Jan 28, 2004 at 09:06:52PM +0300, ??? wrote: Hello! As I was found rsync do not detect file renaming. If I just copy my backup.0.tgz (many Mbytes in size having it's md5) to backup.1.tgz (which will be equial in size and md5) it will be the same file in fact... Rsync will delete old file (backup.0.tgz) on the receiving side and download new one (backup.1.tgz). I do not think there are any difficulties to detect this situation and follow the natural way: just rename the file on the receiving side. Am I right? Reliably detecting renamed files in a manner that rsync can act on it is very difficult. I believe rsync currently sends whole-file md4sums along with the signature. At least when using the -c option, it _could_ use the md4sums to identify identical but moved files. Actually implementing it is another story :-) Whole file checksums are sent for all files only if -c is specified, slowing down the whole ball of wax. That is the purpose of -c (having checksums for file comparison). The key would be to change completely the delete code which currently runs either before or after the update stuff. Instead, before updates are done loops would have to be run correlate deleted and new files. Rename detection wouldn't necessarily require checksums. If a new file had the same timestamp, ownership and size as a deleted one (even without checksums) it would most likely be the same file. Such correlating files should be used as the basis file for an update not just a straight rename. That way a false positive would result in transferring the whole file as it is done now. I'm not sure i'd trust just the whole-file Adler32 and MD4 checksums to be birthday safe for a blind rename. Frankly, for the small payoff i'm not that eager to see this done near term. In most cases it is reasonable to adjust file naming schemes to use less ephemeral names thereby avoiding the problem altogether. For now, where that isn't possible and the file sizes make rsync's current behaviour too expensive i'd suggest looking at a pre-rsync pass with another tool to identify renamed files. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Differentiating debug messages from both sides
On Tue, Jan 27, 2004 at 12:31:05AM -0800, Wayne Davison wrote:
Some of the debug messages that rsync outputs (when verbose = 2) can
occur on both sides of the connection. This makes it hard to know which
program is saying what. Some debug messages deal with this by
outputting a [PID] string at the start of the message. Unfortunately,
the startup message that tells us which pid is which is only output when
verbose = 3, so there's a bit of a problem if we use this idiom for
messages such as expanding file_list ... and excluding file ...
(which get output at verbose = 2 and can occur on both sides).
I'd like to propose a change that prefixes dual-sided messages with [s],
[r], or [g] for the sender, receiver, and generator, respectively. This
makes them easier to read, and it works at any verbose level. Here's a
patch. Let me know what you think.
Good idea. I think i was contemplating such a thing a year
or so ago but you've actually done something about it.
The only thing i'm not sure about is using three characters
to present a one character abreviation.
..wayne..
--- exclude.c 20 Jan 2004 04:53:59 - 1.58
+++ exclude.c 27 Jan 2004 07:36:35 -
@@ -27,6 +27,7 @@
#include rsync.h
extern int verbose;
+extern char *who_am_i;
struct exclude_struct **exclude_list;
struct exclude_struct **local_exclude_list;
@@ -102,7 +103,7 @@ void free_exclude_list(struct exclude_st
struct exclude_struct **list = *listp;
if (verbose 2)
- rprintf(FINFO,clearing exclude list\n);
+ rprintf(FINFO, [%c] clearing exclude list\n, *who_am_i);
if (!list)
return;
@@ -203,12 +204,14 @@ static void report_exclude_result(char c
* then it is stripped out by make_exclude. So as a special
* case we add it back in here. */
- if (verbose = 2)
- rprintf(FINFO, %s %s %s because of pattern %s%s\n,
+ if (verbose = 2) {
+ rprintf(FINFO, [%c] %s %s %s because of pattern %s%s\n,
+ *who_am_i,
ent-include ? including : excluding,
name_is_dir ? directory : file,
name, ent-pattern,
ent-directory ? / : );
+ }
}
@@ -250,7 +253,8 @@ void add_exclude(struct exclude_struct *
out_of_memory(add_exclude);
if (verbose 2) {
- rprintf(FINFO,add_exclude(%s,%s)\n,pattern,
+ rprintf(FINFO, [%c] add_exclude(%s,%s)\n,
+ *who_am_i, pattern,
include ? include : exclude);
}
--- flist.c 27 Jan 2004 01:47:41 - 1.174
+++ flist.c 27 Jan 2004 07:36:30 -
@@ -46,6 +46,7 @@ extern int recurse;
extern char curr_dir[MAXPATHLEN];
extern char *files_from;
extern int filesfrom_fd;
+extern char *who_am_i;
extern int one_file_system;
extern int make_backups;
@@ -342,7 +343,8 @@ static void flist_expand(struct file_lis
}
if (verbose = 2) {
- rprintf(FINFO, expand file_list to %.0f bytes, did%s move\n,
+ rprintf(FINFO, [%c] expand file_list to %.0f bytes, did%s
move\n,
+ *who_am_i,
(double)sizeof(flist-files[0])
* flist-malloced,
(new_ptr == flist-files) ? not : );
@@ -781,8 +783,10 @@ struct file_struct *make_file(char *fnam
skip_excludes:
- if (verbose 2)
- rprintf(FINFO, make_file(%s,*,%d)\n, fname, exclude_level);
+ if (verbose 2) {
+ rprintf(FINFO, [%c] make_file(%s,*,%d)\n,
+ *who_am_i, fname, exclude_level);
+ }
file = new(struct file_struct);
if (!file)
@@ -1416,8 +1420,8 @@ static void clean_flist(struct file_list
return;
for (i = 0; i flist-count; i++) {
- rprintf(FINFO, [%ld] i=%d %s %s mode=0%o len=%.0f\n,
- (long) getpid(), i,
+ rprintf(FINFO, [%c] i=%d %s %s mode=0%o len=%.0f\n,
+ *who_am_i, i,
NS(flist-files[i]-dirname),
NS(flist-files[i]-basename),
(int) flist-files[i]-mode,
--- main.c27 Jan 2004 08:05:10 - 1.185
+++ main.c27 Jan 2004 07:59:07 -
@@ -56,6 +56,8 @@ extern char *remote_filesfrom_file;
extern char *rsync_path;
extern char *shell_cmd;
extern struct file_list *batch_flist;
+extern char *who_am_i;
+
/* there's probably never more than at most 2 outstanding child processes,
* but set it higher just in case.
@@ -198,7 +200,7 @@ static void show_malloc_stats(void)
getpid(),
am_server ? server : ,
am_daemon ? daemon : ,
- am_sender ? sender : receiver);
+
Re: rsync error using ssh : @ERROR: access denied to server.domain.com from unknown (0.0.0.0) {Scanned By MailScanner}
On Tue, Jan 27, 2004 at 04:31:53PM -0800, AI Connex wrote:
I use rsync to mirror several servers.
I run RH7.3
My rsyncd.conf file is:
motd file = /etc/rsync.d/rsync.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
hosts allow = 10.1.2.200 10.1.2.201
hosts deny = 0.0.0.0/0.0.0.0
use chroot = yes
max connections = 3
#syslog facility =
[website]
path = /var/www/website
comment = Connex Live WWW Server
uid = nobody
gid = nobody
read only = no
list = yes
auth users = someone,root
secrets file = /etc/rsync.d/rsync.secrets
I use the --rsh=ssh option to use a ssh protocol
A typical script contains:
#!/bin/ash
PATH=/usr/local/bin:/bin:/usr/bin
### Setting user
USER=root
echo Synchronizing Website
#echo
rsync --rsh=ssh \
--password-file=/root/.rsyncpwd \
--compress --recursive --times --perms --links --owner --group \
--include web_order* --include web_user.* --include
web_user_c* --include web_user_h* \
--include web_user_l* --include web_org* --include web_in* --include
web_quote* \
--include quick_connect.* \
--exclude * \
10.1.2.190::website /var/www/website
Everything worked perfectly.
I am now getting the error
@ERROR: access denied to server.domain.com from unknown (0.0.0.0)
The question is, what changed?
If I changed the script so I do not use a ssh shell, everything works fine.
I have hunted the web for a solution, but no joy.
Please advise how I correct the problem.
As near as i can tell it never should have worked because
a local connection, via ssh, would never be allowed access with hosts
allow clause unless perhaps one of the hosts listed were
0.0.0.0
I've attached an UNTESTED patch (against CVS HEAD but should
be applicable to some older versions) that disables hosts
[allow|deny] for rsync over ssh so that the same config file
may be used for both ssh and direct socket connections.
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
? ..clientserver.patch
Index: clientserver.c
===
RCS file: /data/cvs/rsync/clientserver.c,v
retrieving revision 1.115
diff -u -p -r1.115 clientserver.c
--- clientserver.c 27 Jan 2004 07:57:12 - 1.115
+++ clientserver.c 28 Jan 2004 01:02:45 -
@@ -226,17 +226,16 @@ static int rsync_module(int f_in, int f_
int ret;
char *request=NULL;
- if (!allow_access(addr, host, lp_hosts_allow(i), lp_hosts_deny(i))) {
+ if (am_daemon am_server) {
+ rprintf(FINFO, rsync allowed access on module %s from %s (%s)\n,
+ name, host, addr);
+ }
+ else if (!allow_access(addr, host, lp_hosts_allow(i), lp_hosts_deny(i))) {
rprintf(FERROR,rsync denied on module %s from %s (%s)\n,
name, host, addr);
io_printf(f_out, @ERROR: access denied to %s from %s (%s)\n,
name, host, addr);
return -1;
- }
-
- if (am_daemon am_server) {
- rprintf(FINFO, rsync allowed access on module %s from %s (%s)\n,
- name, host, addr);
}
if (!claim_connection(lp_lock_file(i), lp_max_connections(i))) {
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error using ssh : @ERROR: access denied toserver.domain.com from unknown (0.0.0.0) {Scanned By MailScanner}
On Tue, Jan 27, 2004 at 09:55:41PM -0500, [EMAIL PROTECTED] wrote: I've encountered a similar situation, and tracked it down. It seems that if the shell for your user is set to bash2 versions 2.0 - 2.05.0, it causes your IP to appear as 0.0.0.0 . However, this has been fixed in 2.05b.0, and also works as normal in every other shell I've tested (zsh, csh, bash v1.x, ash). When I use SSH keys, I use a forced command, with a from address, but I also use the rsync hosts allow/deny so that the keys can be used to access some shares from some machines, and some from others. the 2.05b version can be found at http://www.gnu.org/directory/GNU/bash.html . Hope that helps. Sure enough i've two versions of bash here and 2.05.0 unexports all the SSH environment variables but 2.05b.0 passes them. As near as i can tell it never should have worked because a local connection, via ssh, would never be allowed access with hosts allow clause unless perhaps one of the hosts listed were 0.0.0.0 I've attached an UNTESTED patch (against CVS HEAD but should be applicable to some older versions) that disables hosts [allow|deny] for rsync over ssh so that the same config file may be used for both ssh and direct socket connections. That patch is no good. If you are stuck with bash trashing your environment variables and want to share the rsyncd.conf file with a regular daemon while using hosts allow you can use a forced command or put 0.0.0.0 in the hosts allow list. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: patch for linux capabilities
On Mon, Jan 26, 2004 at 01:36:23PM -0600, Michael Glasgow wrote:
I was wondering if it might be possible for an rsync developer to
look over the attached patch (tested on Linux 2.4.24 against the
rsync-2.6.0 release), and offer suggestions on how I could improve it.
Basically I want to use Linux finer grained capabilities to retain
only CAP_SYS_CHROOT CAP_DAC_READ_SEARCH when rsync drops root
privs. That way I can take whole system backups as a (mostly)
unprivileged user.
Probably it should be some kind of config file option or something
rather than a simple compile-time option. Also it would be nice
to drop root much earlier if possible, but where?
Any suggestions are much appreciated!
Start by putting the capability drop code into a seperate
function so that it doesn't complicate the existing code.
Then add a config file option to enable it on a
per-module basis (like use chroot and uid).
Also, i'd drop the capabilities after chroot() and not keep
the cap_sys_chroot.
You might want to evaluate different capability sets
depending on read only or other options and drop
capabilities after doing the chroot.
On a wider note, i'd rather this stay a patch for awhile and
become less Linux specific.
Content-Description: rsync.patch
diff -urN rsync-2.6.0.orig/clientserver.c rsync-2.6.0/clientserver.c
--- rsync-2.6.0.orig/clientserver.c 2003-09-10 23:00:19.0 -0500
+++ rsync-2.6.0/clientserver.c2004-01-26 11:27:53.0 -0600
@@ -25,6 +25,18 @@
* rsyncd.
**/
+#ifdef HAVE_LINUX_CAPS
+#ifdef _POSIX_SOURCE
+#undef _POSIX_SOURCE
+#include sys/prctl.h
+#include sys/capability.h
+#define _POSIX_SOURCE
+#else
+#include sys/prctl.h
+#include sys/capability.h
+#endif
+#endif
+
#include rsync.h
extern int module_id;
@@ -217,6 +229,10 @@
int start_glob=0;
int ret;
char *request=NULL;
+#ifdef HAVE_LINUX_CAPS
+ cap_t cp;
+ cap_value_t newcaps[2] = { CAP_SYS_CHROOT, CAP_DAC_READ_SEARCH };
+#endif
extern int am_sender;
extern int am_server;
extern int am_daemon;
@@ -373,12 +389,46 @@
}
#endif
+#ifdef HAVE_LINUX_CAPS
+ if (setreuid(uid, 0)) {
+ rsyserr(FERROR, errno, setreuid(%d,0) failed, (int) uid);
+ io_printf(f_out, @ERROR: setreuid failed\n);
+ return -1;
+ }
+ if( prctl(PR_SET_KEEPCAPS, 1) 0 ) {
+ rsyserr(FERROR, errno, prctl failed);
+ io_printf(f_out, @ERROR: prctl failed\n);
+ return -1;
+ }
+ if( (cp = cap_init()) == NULL ) {
+ rsyserr(FERROR, errno, cap_init failed);
+ io_printf(f_out, @ERROR: cap_init failed\n);
+ return -1;
+ }
+ if( cap_set_flag(cp, CAP_PERMITTED, 2, newcaps, CAP_SET) 0 ||
+ cap_set_flag(cp, CAP_INHERITABLE, 2, newcaps, CAP_SET) 0 ) {
+ rsyserr(FERROR, errno, cap_set_flag failed);
+ io_printf(f_out, @ERROR: cap_set_flag failed\n);
+ return -1;
+ }
+ if( cap_set_proc(cp) 0 ) {
+ rsyserr(FERROR, errno, cap_set_proc failed);
+ io_printf(f_out, @ERROR: cap_set_proc failed\n);
+ return -1;
+ }
+#endif
+
if (setuid(uid)) {
rsyserr(FERROR, errno, setuid %d failed, (int) uid);
io_printf(f_out, @ERROR: setuid failed\n);
return -1;
}
+#ifdef HAVE_LINUX_CAPS
+ cap_set_flag(cp, CAP_EFFECTIVE, 2, newcaps, CAP_SET);
+ cap_set_proc(cp);
+#endif
+
am_root = (getuid() == 0);
}
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problems with --exclude
On Mon, Jan 26, 2004 at 09:21:39PM -0600, Brian Camp wrote: I'm trying to get rsync to exclude the directory /home/www/users/ftp/pub/ from being copied in the command line below and have not been successful. I've tried many combinations, such as dropping the /'s and adding *'s, without any luck. /home/www/users/ftp/pub/ is not part of a symlink, but it is linked to several times within /home. The local and remote machines are both running rsync 2.5.7 under openbsd 3.4. /usr/local/bin/rsync -zqrlptgo --delete --exclude /home/www/users/ftp/pub/ [EMAIL PROTECTED]:/home/ /home/backup/gorgimera/home/ What am I doing wrong? Any help would be *greatly* appreciated. To quote tha manpage (EXCLUDE PATTERNS) The filenames matched against the exclude/include patterns are relative to the destination directory, or top direc tory, so patterns should not include the path elements of the source or destination directories. The only way in which a pattern will match the absolute path of a file or directory is if the source path is the root directory. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --link-dest not working with rsync daemon?
On Mon, Jan 26, 2004 at 09:57:32PM -0500, Alberto Accomazzi wrote: Let me take this opportunity to thank you personally for taking on the task of pushing out the latest rsync release and for your and jw's continuing work on this. I know a lot of people have contributed patches and ideas to rsync but it does take a few good men to pull it all together. I think you guys are doing a top-notch job. So I think I speak for all the readers on the list when I say we are all *very* grateful for all you've done and are doing. Wayne has been doing the heavy lifting but the other developers have been active too. I would be remiss to not recognise the critical contribution of you and other users who have given us good bug reports, sent in patches or even suggested improvements. Those of us who know the code mostly have it working good enough for ourselves. And while I'm still on the line: do you have any ideas about upcoming releases? There was some discussion prior to releasing 2.6.0 about what should be tackled next, but I'm not sure there was consensus beyond what became 2.6.0. Soon, i hope. 2.6.1 is looking like a performance release. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --link-dest not working with rsync daemon?
On Mon, Jan 26, 2004 at 10:27:16PM -0800, [EMAIL PROTECTED] wrote: jw schultz writes: Soon, i hope. 2.6.1 is looking like a performance release. Can we get the craigb-perf patch in (sorry I haven't looked in CVS - maybe it is there)? Thanks to Wayne for porting the patch to 2.6.0. Wayne committed it to CVS January 3rd. It is central to my emphasis of the performance aspect of the next release. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Tue Jan 27 05:00:43 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv645 Modified Files: rsync.yo Log Message: EXCLUDE PATTERNS is not just about syntax. Revisions: rsync.yo1.145 = 1.146 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.yo.diff?r1=1.145r2=1.146 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
Re: scan for first existing hard-link file
On Sun, Jan 25, 2004 at 02:05:21AM -0800, Wayne Davison wrote: Here's a patch that makes rsync try to find an existing file in a group of hard-linked files so that it doesn't create the first one in the group from scratch if a later file could be used instead. Details: I decided to avoid having the code do an extra scan down the list when we encounter the lead file in the list. This is because it would be bad to have to do the same scan in the receiver that the generator just performed, especially since there's no guarantee that it will get identical results (if a file pops up at the wrong moment). My solution just keeps moving the master file in the group down the list, causing it to be processed in turn as we go through the normal flist scan. This ensures that we use the right basis file in the receiver, and keeps the code simple. The only complicating factor was that the hard-link post-processing pass was being done by the receiver, while the generator is the one that keeps track of the updated master. To deal with this, I moved the hard-link post-processing loop and the final touch-up of the directory permissions into the final work that the generator does after it gets the end of phase 2 indicator from the receiver. Some simple changes to the hard-link data structures was needed. I got rid of the head pointer, replacing it with an index into the hlink_list array. This lets us update this first item pointer to point to the current master. I then made the single-linked list of hard-linked items circular, and added a flag to mark the last item in the original list (so we know when to give up our search and just ask for the file to be created). Nice. Took a few minutes to grok that you made generator skip the file until you got a hit or it was the last in the original list. Having moved the hardlink creation to the generator i don't think the receiver does anything with hlink_list or related data. This means we can, later, free that memory in receiver after the fork, reducing the memory footprint a bit more and avoiding the COW of the hlink_list that you have just introduced. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: is there a bug with one-file-system?
On Sun, Jan 25, 2004 at 02:58:54PM -0600, Thomas Stivers wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I am trying to use rsync to backup a filesystem to another computer on a local network, but when I use rsync -arx / some.host:backup rsync still tries to copy everything from all mounted filesystems. Is there a bug, or is there a better way to do what I want than this? Thanks. Nope, looks fine. Perhaps it is your OS or an old version of rsync? -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Sun Jan 25 22:11:09 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv8033 Modified Files: flist.c Log Message: Tightened up code in skip_filesystem() Revisions: flist.c 1.169 = 1.170 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c.diff?r1=1.169r2=1.170 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
Re: rsync.yo doc patch
On Fri, Jan 23, 2004 at 01:43:41AM -0800, Wayne Davison wrote: On Thu, Jan 22, 2004 at 06:50:35PM -0800, jw schultz wrote: I'd leave it as is except to replace the paranthetic with the text regarding --numeric-ids in the --owner section Unfortunately the text in the --owner section is wrong -- the --numeric-ids option is not implied by anything. If the chroot area contains its own etc/passwd and etc/group files, the names will be mapped even when chroot is enabled. I've updated the sections on --owner, --group, --numeric-ids, and the use chroot setting. See how you like it. You are right, it isn't implied; it is compelled unless the configuration allows name and ID lookups. The only objection i have to your --owner, and --group text is The preservation is done primarily by name. It is only primarily by name when --numeric-ids isn't selected and passwd and group lookup is possible. I say passwd and group lookup is possible and not that /etc/passwd and /etc/group files are readable because the precise behaviour is dependant on the libraries and most often /etc/nsswitch.conf (if readable). On FreeBSD it is pwd.db that has to be readable. People using NIS or, i assume, LDAP don't need passwd and group files so don't need to keep them up-to-date. As i think about it i'm inclined to suggest we put only a reference to --numeric-ids in the --owner and --group sections and put a more detailed description in the --numeric-ids section. The more detailed description in --numeric-ids (as a central point for the rsync.1 manpage) could cover the issue of names that aren't on both ends and files owned by IDs not in the database, much as it does already. It also should perhaps not go into chroot issues but merely reference the chroot section of rsyncd.conf.5 with a mention that rsync daemons might not be configured to support ID preservation by name. The chroot section of rsync.conf.5 could cover the issue of database availability for name-ID lookups. As i think on it if we refer to the issue as being of identification database availability instead of specific files in /etc the text gets much simpler, more portable, and might be included on both manpages. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync.yo doc patch
On Fri, Jan 23, 2004 at 12:21:13PM -0500, Aaron S. Hawley wrote: On Fri, 23 Jan 2004, jw schultz scribbled: On Thu, Jan 22, 2004 at 07:00:17PM -0500, Aaron S. Hawley wrote: Patch includes fixes to man page including: - Typos/Spelling - Clarity - Special characters. - Moved around text explaining example. Let's keep the examples in the example, description order. this seems to have been taken up later in the thread and is not included in the patch below. There appear to be whitespace-only changes. If changing the location of line breaks in the source do that as a seperate patch so labeled. Then the actual text changes can be considered. Here are residual fixes from my initial patch for rsync.yo against version 1.144 (I noticed 1.145 was just checked-in. the patch still worked for me.) Whitespace-only changes were minimized (couldn't get diff's -w and -b options working to my liking, any suggestions?) Much better. again: Make sure I didn't confuse the exclude list/file paragraph (line ~969 after applying the patch), I haven't used the feature and the details of that paragraph are a tad unclear. responses to J.W. Schultz below. patch at very bottom of message and at: http://www.uvm.edu/~ashawley/rsync/rsync.yo-1.144-ashawley.diff-u /a Parens or quotes but not both, please. I find the : or :: set off by spaces to be perfectly clear, a bit easier to read and have had some cases (not here) where there was some doubt whether the surrounding characters intended as quotes might not be part of the string. But if people want : or (:) or even : i can live with it. I went with (:), seemed to be the choice of a few other man pages. Most of the time you now use () it is correct because context makes it a parenthetic. However, having done it that way means you are not using quotes. So when you have literals that are not parenthetic the parenthasees are out of place. Quite frankly the more i look at them the more i dislike your changes adding descriptive terms. Filesystems is correct usage. It might not be in your spelling dictionary but is correct and semantically distinct from file systems. Even if it weren't for the semantic difference this closed compound sould be consistant with the Chicago Manual of Style (7.84) Don't have a copy of Chicago Manual of Style handy, but the google test gave me file systems. Those fixes are removed from the initial patch. Again, This is correct usage. user name: George Washington username: gwashin Rsync is not going to extract user names from the comment field and try to match them. agreed. not in patch. Pulling that out of the parenthetic is good but the choice of term is one of: stdin standard input stdin (standard input) The last one only the first time stdin is used. went with last in patch. - *?[ then expression matching is applied using the shell filename + *?[ then expression matching is applied using the shell filename Adding quotes here confuses the issue. The one thing that might be done is to make the wildcard character list in boldface as a literal expression. Good, although i doubt it will be readily discernable on-screen. in patch. speaking of which: Index: rsync.yo === RCS file: /cvsroot/rsync/rsync.yo,v retrieving revision 1.144 diff -u -r1.144 rsync.yo --- rsync.yo 23 Jan 2004 09:34:04 - 1.144 +++ rsync.yo 23 Jan 2004 16:19:31 - @@ -48,35 +48,35 @@ itemize( it() for copying local files. This is invoked when neither - source nor destination path contains a : separator + source nor destination path contains a single colon (:) separator I'll use this one as an example. You specify the character by name and then show it. When you do both the second will be a parenthetic expression, either set off by () (parenthasees), isolated by , (commas) or em dash(es) which would really confuse things because em dashes should not be adjoined by spaces and are not available for display in an in the ASCII terminal or emulator. Personally, i find the repeated use of this form to be tiresome. I'd rather not take this set of changes where all you are doing is adding a text description of the : or :: every time they are used. If the first time : (colon) and :: (double colon) were used we had the text description, it might be OK because that helps the surpising number of people who don't know what to call the symbol--i've heard it called double-dot and my father once had to spend half an hour on the phone describing where the character was on the keyboard and that it differed from the semi-colon, with a secretary who didn't know the word and had to type D: on a dos prompt. it() for copying from the local machine to a remote machine using a remote shell program as the transport
Re: [jerry.seutter@pason.com: Is rsync supposed to resume?]
On Fri, Jan 23, 2004 at 10:23:57AM -0700, Jerry Seutter wrote: Retrying with the _correct_ address this time. *blush* - Forwarded message from Jerry Seutter [EMAIL PROTECTED] - Date: Fri, 23 Jan 2004 10:01:47 -0700 From: Jerry Seutter [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Is rsync supposed to resume? User-Agent: Mutt/1.2.5.1i Hi, I recently got caught out by the behaviour in rsync and am not sure if it is a bug or not. I was running rsync like: rsync -avz -e ssh --progress source destination The source was a directory structure. Partway through the transfer I stopped it with a Ctrl-c. Later I ran the same command again and it completed. Cut out irrelevant confusion here I did a checksum on the files in the source and destination directories and the checksums on one of the files was different. I think it was on the file that I stopped rsync on. From what I read on the manpage, I thought rsync was supposed to create a copy of the the source on the destination. Is this not the case? After testing a bit I found that adding the --checksum option provided the behaviour I needed. Can someone offer some enlightenment? rsync version on source: rsync version 2.5.6 protocol version 26 rsync version on destination: rsync version 2.5.4 protocol version 26 Most likely you hit a temporal window on the modification time. This was probably not related to the interruption. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Group Permissions?
On Fri, Jan 23, 2004 at 09:09:44AM -0800, Wayne Davison wrote: On Fri, Jan 23, 2004 at 11:49:56AM -0500, [EMAIL PROTECTED] wrote: Does this mean that rsync requires that the destination directory be owned by the user calling rsync and that it is not sufficient for the user to have write permissions to that directory? It depends on what you're trying to do and what OS you're running. If you're just putting files into a directory, you don't need to own it. However, changes to the directory information (such as a mode change) requires that rsync have permissions to make the change, and that typically means owning the directory (or being root). Similar errors could be reported for files when the content doesn't change (i.e. the file isn't being recopied). Keep in mind that you'd get the same kind of error from cp -p if you don't own the file, so this is not an rsync-specific thing. It would be possible to be specific if you told us what OS and filesystem types are involved although specific info may be lacking for some combinations. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Equivalent between rsync -aHr and cp -al
On Thu, Jan 22, 2004 at 06:19:06PM +0900, Charles Nadeau wrote: Hi, Just a quick question. I am looking for a replacement for the command cp -al which has problems dealing with a massive amount of files (http://mail.gnu.org/archive/html/bug-coreutils/2003-12/msg00028.html). Could I use rsync -aHr to do _exactly_ the same thing as cp -al? No. The -H (--hard-links) option is only to replicate internal linkages not make external ones. That is, if a/b is linked to a/c and you do rsync -aH a/ z z/b will be linked with z/c but not to a. Rsync's --link-dest option could be twisted to accomplish that but i'd be flabbergasted if any currently released version of rsync used less memory than cp. I'd suggest trying cpio's pass-through mode cd $src; find . | cpio -pl $dest on the chance that cpio might be a little more efficient about link tuples than cp. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync's Speed
On Thu, Jan 22, 2004 at 01:36:05PM -0700, Steve Sills wrote: Well, it is running under SSH, that could slow it down i guess Very easy to tell. If ssh is burning user-mode cpu time the encryption is a factor. It is very unlikely the encryption is having any affect on throughput. Unless you are on very old hardware the network is slower than the CPU and for rsync the disk latency likely makes it slower than the 10MBps network. Steve Sills Platnum Computers, President http://www.platnum.com [EMAIL PROTECTED] - Original Message - From: Streubert, Christoph [EMAIL PROTECTED] To: Chuck Wolber [EMAIL PROTECTED]; Steve Sills [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Thursday, January 22, 2004 12:52 PM Subject: RE: Rsync's Speed I do not know the details of your syntax - however in case you use it - encryption adds a lot of overhead to the transfer process. After I turned off encryption (internal network with uncritical data) speeds increased dramatically. C -Original Message- From: Chuck Wolber [mailto:[EMAIL PROTECTED] Sent: Thursday, January 22, 2004 10:30 AM To: Steve Sills Cc: [EMAIL PROTECTED] Subject: Re: Rsync's Speed During my initial download for my home directory backup, it took rsync over 6 hours to do the initial backup, but I can FTP the stuff in about 30 Mins. Is Rsync usualy this slow? I have compression turned on, and its across a 100 MB/S network, Anyone had this problem before? In this case, I believe compression is actually slowing you down. -Chuck -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit. - FDR -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync.yo doc patch
On Thu, Jan 22, 2004 at 05:41:12PM -0800, Wayne Davison wrote: I've attempted to snag the least controversial changes out of the patch and checked them in. I also changed file system into filesystem in one place, made your suggested allow-support change, and changed id to ID (since we're not talking about the id and the ego). I'll comment more in another email. It isn't worth its own checking but there are at least two cases where the abreviation etc. is misspunctuated. It is an abreviation of etcetera and should be marked as such. This does not apply to etc as a directory name. Just a thought but with your change of id to ID (which i agree with) it might be a good idea to change IO to I/O to reduce the doubletake. Or that could just be the dyslexia talking. More comments below. ..wayne.. --- rsync.yo 15 Jan 2004 17:45:53 - 1.139 +++ rsync.yo 23 Jan 2004 01:32:55 - @@ -77,7 +77,7 @@ using a remote shell program as the transport, using rsync server on the remote machine. This is invoked when the destination path contains a :: separator and the - --rsh=COMMMAND option is also provided. + --rsh=COMMAND option is also provided. it() for listing files on a remote machine. This is done the same way as rsync transfers except that you leave off the @@ -111,7 +111,7 @@ You use rsync in the same way you use rcp. You must specify a source and a destination, one of which may be remote. -Perhaps the best way to explain the syntax is some examples: +Perhaps the best way to explain the syntax is with some examples: quote(rsync *.c foo:src/) @@ -162,8 +162,8 @@ You may establish the connection via a web proxy by setting the environment variable RSYNC_PROXY to a hostname:port pair pointing to -your web proxy. Note that your web proxy's configuration must allow -proxying to port 873. +your web proxy. Note that your web proxy's configuration must support +proxy connections to port 873. Using rsync in this way is the same as using it with a remote shell except that: @@ -217,7 +217,7 @@ manpagesection(RUNNING AN RSYNC SERVER) -An rsync server is configured using a config file. Please see the +An rsync server is configured using a configuration file. Please see the rsyncd.conf(5) man page for more information. By default the configuration file is called /etc/rsyncd.conf, unless rsync is running over a remote shell program and is not running as root; in that case, the default name @@ -496,7 +496,7 @@ is not used and the whole file is sent as-is instead. The transfer may be faster if this option is used when the bandwidth between the source and target machines is higher than the bandwidth to disk (especially when the -disk is actually a networked file system). This is the default when both +disk is actually a networked filesystem). This is the default when both the source and target are on the local machine. dit(bf(--no-whole-file)) Turn off --whole-file, for use when it is the @@ -519,7 +519,7 @@ dit(bf(-g, --group)) This option causes rsync to set the group of the destination file to be the same as the source file. If the receiving program is not running as the super-user, only groups that the -receiver is a member of will be preserved (by group name, not group id +receiver is a member of will be preserved (by group name, not group ID number). The parenthetic should be dropped. It will be by ID if --numeric-ids is specified or a daemon is running chrooted. Explaining that here would, i think, only confuse things. Also, receiving user might be a better term. dit(bf(-D, --devices)) This option causes rsync to transfer character and @@ -602,7 +602,7 @@ default, but you may prefer to use rsh on a local network. If this option is used with bf([EMAIL PROTECTED]::module/path), then the -remote shell em(COMMMAND) will be used to run an rsync server on the +remote shell em(COMMAND) will be used to run an rsync server on the remote host, and all data will be transmitted through that remote shell connection, rather than through a direct socket connection to a running rsync server on the remote host. See the section CONNECTING @@ -748,16 +748,16 @@ information sent for matching data blocks. dit(bf(--numeric-ids)) With this option rsync will transfer numeric group -and user ids rather than using user and group names and mapping them +and user IDs rather than using user and group names and mapping them at both ends. -By default rsync will use the user name and group name to determine +By default rsync will use the username and groupname to determine what ownership to give files. The special uid 0 and the special group 0 are never mapped via user/group names even if the --numeric-ids option is not specified. If the source system is a daemon using chroot, or if a user or group -name does not exist on the destination
Re: rsync.yo doc patch
On Thu, Jan 22, 2004 at 06:27:49PM -0800, Wayne Davison wrote: On Thu, Jan 22, 2004 at 06:02:35PM -0800, jw schultz wrote: it might be a good idea to change IO to I/O to reduce the doubletake. Yes, I prefer I/O for input/output as well. dit(bf(-g, --group)) This option causes rsync to set the group of the destination file to be the same as the source file. If the receiving program is not running as the super-user, only groups that the receiver is a member of will be preserved (by group name, not group ID number). The parenthetic should be dropped. It will be by ID if --numeric-ids is specified or a daemon is running chrooted. Explaining that here would, i think, only confuse things. Also, receiving user might be a better term. How about this text? dit(bf(-g, --group)) This option causes rsync to set the group of the destination file to be the same as the source file, as much as the receiving user's permissions permits. The preservation is done primarily by name, but falls back to using the ID number if no matching name is found. See also the --numeric-ids option. It still seems awkard and may not be strictly true. I'd leave it as is except to replace the paranthetic with the text regarding --numeric-ids in the --owner section or with: Like --owner if the remote system is a daemon using chroot, the --numeric-ids option is implied because the remote system cannot get access to the groupnames from /etc/group. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: how about auth users without a password?
On Wed, Jan 21, 2004 at 07:26:50AM -0800, Larry Brasfield wrote: Jim Salter wrote: Wouldn't this (accomplishing security restrictions without need to enter a password, or to enter a password more than once) be a lot more easily accomplished by simply using SSH transport and public/private keys instead of using the daemon mode at all? Sorry to be unclear. I am using SSH transport. In fact, I do not have the daemon running or rsync's favorite port operating under inetd's province. An example might be clearer. I have CVS access setup so that clients use SSH to get in to the server, but then CVS configuration and Linux filesystem permissions (on the repositories) are used to control who can actually perform various operations on which repositories. This limitation is effected regardless of how somebody invokes CVS. Clients use SSH public/private keys, with the aid of a key agent if they like, to get past sshd with whatever limits their shell (from /etc/passwd) imposes. There is no password entry phase at all, and if they use a key agent, repeated access is very convenient. I hope to manage rsync access the same way. Clients would be forced to come in via SSH (because no other ports are open), and once in, the configuration of rsync will determine what they can do, precisely. This is just a hope at the moment because when I try to limit per-user access via rsyncd.conf, it still demands a password even though the user in question has already been authenticated to permit their SSH entry. Perhaps I'm being anal about this, but I intend to impose some modularity on server access. SSH does authentication and early screening of known users (and encrypts the channel), while the various services are responsible for fine-grained access control. I don't want rsync's authentication, just its per-user control. It sounds like what you want to do is run the rsync daemon over ssh. That way ssh would do user authentication and identification. Identification would be disabled in the rsyncd.conf files and no passwords would be requested by rsync (anonymous rsync). If you needed broader priveleges than were provided by running the daemons as the users themselves you could use the ssh ssh forced command facility with explicit selection of rsync config files. -- Larry Brasfield Jim Salter Larry Brasfield wrote: Hi, from a generally pleased new rsync user. I have setup a number of services to be accessible via SSH. For most of them, it has been possible to arrange that clients can use a key agent and ssh's level 2 protocol to gain access without the need of entering passwords more than once, at the start of a session (assuming their keys are not stored in the clear). Most of these services can be setup to restrict specific users to specific subsets of the potentially available access. With rsync, this appears to be feasible using the auth users configuration item in rsyncd.conf, but in my efforts so far, this always results in a password prompt. So, this is either a question or a suggestion. How can I use rsyncd.conf to limit module access to specific users (or groups, preferably) without inducing rsync to demand a password? If this is not presently possible, I suggest that a nice enhancement would be to make it possible via some device such as a '*' in the associated password entry. This might be limited to rsync invocations by a currently authenticated user (such as occurs with SSH access) and disallowed for the listen on rsync's port mode of operation. I would like to use SSH to authenticate users and grant access to the machine, leaving more specific rights management to the configuration of individual services. With rsync, these functions appear to be a bit more intertwined than they have to be. If people think this is a good idea, (especially the owners of rsync), I would be happy to revise the code to make it work. Let me know at larry nospacehere brasfield at m s n dot com and I will post the results to this list/thread after a week or so. -- Larry Brasfield -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: help!!
On Wed, Jan 21, 2004 at 02:35:38AM -0800, preetha thomas wrote: Greetings!! I am a college student doing my Masters Of Computer Applications.Towards the completion of my project I have undertaken a project in BIOINFORMATICS in the School of BIOTECHNOLOGY.Madurai Kamaraj University. I am trying to use rsync to download the protien structures from the FTP archieve of PDB.I have rsync installed in my system and I have the address of the remote server I want it downloaded from.I have the rsync.sh script to download the archive.But when I run the script (sh rsync.sh) it does not seem to run,neither does it show any error.I am totally lost.I wanna know if i have to do anything else for the successful running of the program.Kindly clear me on it at the earliest. Preetha Thomas There are so many things that may be wrong i couldn't begin to start. Perhaps one of your faculty advisors or an undergrad could help you. They would have some idea what the rsync.sh script is supposed to do and what might be wrong with your machine, the network or the server configuration. I'm sorry to see your university's computing masters programs are compairable with too many here in the US. I do wonder what a master of computer applications is supposed to be capable of if he cannot even debug simple scripts and netorking. Perhaps getting it to work is part of the exercise. You might consider this a wake-up call. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
FIXED: rsync hangs on cygwin
On Wed, Jan 21, 2004 at 05:24:19AM -0600, John Van Essen wrote: On Wed, 21 Jan 2004, Dmitry Melekhov [EMAIL PROTECTED] wrote: Jeremy Lin wrote: rsync hangs under Cygwin have been a longstanding issue. There was a message earlier today about upgrading to the 1.5.6-1 cygwin dll, which supposedly fixes the problem (and does, for me). Hello! Thank you! It works :-) Another happy Cygwin user. :) Since this has been such a long-standing, important bug and is finally now fixed, maybe the contents of that message announcing the 1.5.6-1 cygwin dll deserves placement on the rsync home page? I concur. To help googlers i have changed the subject and include this text: The rsync cygwin hang bug appears to be fixed. Users experiencing rsync hangs when on cygwin should run the cygwin dll version 1.5.6-1 or greater. This will avoid the problem variously as rsync process never dies or rsync does not complete, rsync does not exit, and the ones previously mentioned. If those having web pages related to rsync on cygwin link to a mail archive of this message that will also help raise it's profile in google and other search engines. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.0 - suspected memory leak bug
On Wed, Jan 21, 2004 at 03:35:37PM +, Kelly Garrett wrote: Wayne Davison wayned at samba.org writes: On Tue, Jan 20, 2004 at 06:16:34PM -0800, jw schultz wrote: As far as i can tell Under inetd each connection should get independant rsync process(es) which all exit so there would be no rsync processes running unless there is an active connection. Yes, that's my understanding as well. He also said that the receiving side is initiating the connection (and having the problem), so I'm assuming that he's not complaining about the daemon side. One other thing I thought of: Kelly, check for lingering rsync processe(s) in the process table. That's the only way I know that memory could continue to be allocated at the end of a transfer: if one of the processes didn't actually exit. ..wayne.. There are no residual rsync processes running after the transfer. I think you hit it on the head when you indicated that Linux would use available mem for cacheing that is messing up. Rsync seems to be exiting correctly, and I have tried it with several different versions of rsync. The way that the kernel is currently compiled/configured seems not to be releasing the diskcache memory back to other processes when needed, and the machine crashes once all available RAM and swap has been consumed by the cache during the rsync transfer. So, I do not beleive it is rsync's fault -- rsync merely exposed a flaw in the OS due to the massive size of the transfers we are doing with it. My appologies to the rsync team, it is an excellent tool and had I not been impressed with it I would not have bothered tracking down the problem. Does anyone know how to build a version of the kernel that either does no disk cacheing (we have very fast RAID processors and SCSI disks on the machine) or limit the amount of cache that the system will allocate for disk? Any help would be appreciated. This is actually a fairly common problem. From your description it sounds a bit more like the dcache than the page cache but either way it has appeared frequently on the Linux kernel mailing list (lkml). You might want to search the lkml archives for things like paging broken, out of memory, updatedb and swapiness. I seem to recall there being some tunable as well as patches. The 2.4.20 kernel is a bit old as far as the lkml guys are concerned but i'll bet there is something you can do short of upgrading to a vanilla kernel. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Moving from one server to another with rsync?
On Wed, Jan 21, 2004 at 10:58:03PM +0100, Spear wrote: Hi, I am currently in the process of migrating from RH 7.3 to RHEL ES 3.0 on my system. Of course there is a lot of iles that will need to change server (new server is ready). My question is of course what is the best way to do this? My thought is rsync with a function like this: 1: Running rsync once to get all the files over to my new system 2: Running rsync several times to keep the files updated, only updating/copying new/changed files to the new server 3: running rsync frequently to make sure everything is beeing copied. 4: Switching server. That will work although #1 can be faster with other tools. And here are my questions: 1: Am i going about this all wrong? *lol* No. 2: Can i keep the user/group permissions? Yes. 3: Is there a website where i can find information about this kind of move? Probably. There are websites for practically everything even urinals but sometimes it can take more work to find the website that matches your needs than to figure it out yourself. 4: Can i get all my http, mysql, sendmail etc copied this way? Yes. This i what i have done in advance: 1: Installed Webmin and synchronized users/groups on both servers Good. 2: Upgraded Apache from RH 7.3's 1.x to Apache 2.x.. OK. Upgrading the source machine to the new version saves converting the configs as part of the transfer. 3: Installed sendmail on the new server with all users and settings from RH 7.3 Fine. Whatever you are comfortable with. 4: Installed mySQL with the same settings as on my RH 7.3 system OK. 5: Found some information and ran the command: rsync --verbose --progress --stats --checksum --rsh=/usr/bin/ssh --recursiv e --times --perms --links /www/kunder/web/* MYHOST:www (This worked, however everything was copied into /root/www/* and not into /www which i thought i had set up. Furthermore the ownership, chmod and groupsettings for files was not copied. The /www/ is the home directory for my clients. I guess i will have the same problem when trying this with mySQL databases and sendmail etc). That is because the path on MYHOST you specified was relative. If /root is on the same filesystem as /www you can just mv it. Otherwise you will probably be better off removing /root/www and starting over. The failure to preserve ownership was because you didn't tell rsync to do so. The -a option is a handy shorthand for -rlptgoD which covers most of the options you will need. rsync -avH --rsh=/usr/bin/ssh /www/kunder/web/ MYHOST:/www/kunder/web Would have been what you want. Or perhaps rsync -avH --rsh=/usr/bin/ssh /www/ MYHOST:/www --recursive --times --perms --links are part of -a along with some more options you needed. --progress If you like noise and have BIG files, ok. Me, i'm satisfied with -v. --stats Useful sometimes but not really needed, it will give you an idea of how much time rsync saves in the later rsync runs to keep things up-to-date. --checksum Not needed and slows things down considerably. I tried mounting the /www system on my new server through SMB and running rsync, but this way i could not get the correct user/group permissions on my server. Ick. If you are going to use a network filesystem just copy with cp or cpio -p. For rsync you want rsync to handle the network communications (via ssh). Hope someone could help or point me in the right direction. Best regards, Terje Dahl Norway -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to avoid rsyncing incomplete uploads? (best practice on fileservers?)
On Tue, Jan 20, 2004 at 03:37:38PM +0100, Peter wrote: Hi, some big files are regularily updated on server A. Rsync should sync that with Server B. If rsync starts in the middle of an upload process, will it copy that incomplete file to Server B? First tests showed it will. How can this be avoided? Comments on rsync-usage with big files and very active users would be very appreciated! You have asked several questions all of which are answered in the archives. Happy spelunking. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.6.0 - suspected memory leak bug
On Tue, Jan 20, 2004 at 06:09:09PM -0800, Wayne Davison wrote: On Tue, Jan 20, 2004 at 02:45:47PM -0700, Garrett, Kelly wrote: After every rsync transfer there is a large amount of memory that is not freed up. Your report sounds like you're not talking about process size, but a free-memory report from something like top. If so, Linux uses unused memory as disk cache, so the more disk I/O that happens, the less free memory you'll see on your system. This isn't a bad thing, though, as this disk-cache memory will get used for process memory as needed. Also keep in mind that once a process terminates, there's no way it can continue to hold memory (unless there's a bug in the kernel). If you meant something else, please explain what you're measuring. In my tests rsync's memory size stays steady throughout the transfer (once the file list has been built). Shared memory between the forked processes on the receiving side does slowly become unshared, but that happened in prior rsync versions too (and we've got an internal change in CVS that should make this better for the next release). When running the daemon as a detached processes the forked processes exit, freeing that memrory. As far as i can tell Under inetd each connection should get independant rsync process(es) which all exit so there would be no rsync processes running unless there is an active connection. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Improving name-truncation detection
On Mon, Jan 19, 2004 at 10:17:30AM -0800, Wayne Davison wrote: I've got a patch that changes f_name_to() to return an unsigned int (like sme_tonprintf() and strlcpy() do) and adds checking to ensure that we didn't overflow the name before we try to use it: http://www.blorf.net/name-overflow.patch If anyone would care to check out the following patch before I commit it, please do. I don't like it on two counts. If we are going to vet the path name for overflow (a good idea) lets do it once, explicitly, as we receive it instead of having tests scattered throughout the code. In other words, test for strlen(file-dirname) + strlen(file-basename) = MAXPATHLEN - 2 in receive_file_entry(). When all you are doing is concatinating a couple of strings snprintf is overkill, especially in an infrastructure function. As it is now f_name_to() is about as good as it gets. We went through several iterations with get_tmpname() to keep it clean and efficient there is no need to regress here. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync and Tar
On Mon, Jan 19, 2004 at 03:56:40PM -0700, Steve Sills wrote: Hello, I am trying to setup a backup system. What I would like to do is have rsync create a tar file as it downloads (to save space) and just have it update that tar file. Can rsync create a archive on the fly, or does this have to be done after its finished its transfer? I would also like No. You might want to look at what rsync does or how rsync works. to have rsync report its transfer stat to a MYSQL database for billing purposes, on a per client basis. Can this be done? What you do with the stats in a post-processing script is up to you. Instead of reinventing the wheel you might what to take a look at dirvish and other existing free backup systems that use rsync to get the most out of bandwidth and disk space. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Improving name-truncation detection
On Mon, Jan 19, 2004 at 05:44:20PM -0800, Wayne Davison wrote: On Mon, Jan 19, 2004 at 02:39:56PM -0800, jw schultz wrote: If we are going to vet the path name for overflow (a good idea) lets do it once, explicitly, as we receive it instead of having tests scattered throughout the code. Yeah, good idea. When all you are doing is concatinating a couple of strings snprintf is overkill I forgot how slow snprintf() is (it's nearly three times slower than other methods of concatenation). There are a few places in the code that are already using it (such as the backup code), which we should change in light of this. A bigger problem, though, is that the current f_name_to() code doesn't handle an overflow properly: if bsize - off is negative, the value If f_name_to were a generalized function i'd agree. It isn't It concatenates a pair of fixed strings that can only produce a negative bsize - off if we don't test the values on receipt and have a malicious client. turns unsigned once it gets to the strlcpy() function. Ouch! Using successive strlcat() calls would be safe, but using a strlcat() call after a strlcpy() of just a couple hundred bytes has about a 50% speed penalty compared to following it with an offset strlcpy() call. Cache misses caused by rescanning the destination to find the terminator and length. (It takes a few hundred bytes before you are likely to see L1 misses.) That is why we use stacked strlcpy. I wrote a function that will let us concatenate values together safely without a speed penalty. Encapsulating it in a function ensures that we don't have to have a series of if (no overflow) x = strlcpy(...); calls every time we concatenate something. It also ensures that we do it the safe way everywhere. The function is called with any number of character strings, like this: total_len = stringjoin(dest, sizeof dest, s1, s2, s3, NULL); I'll append my util.c patch to this email. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Improving name-truncation detection
On Mon, Jan 19, 2004 at 06:00:32PM -0800, Wayne Davison wrote:
On Mon, Jan 19, 2004 at 05:44:20PM -0800, Wayne Davison wrote:
I'll append my util.c patch to this email.
Or perhaps to this one...
If you're going to do the strlen(src) and whatnot you might
as well just snag the strlcpy source and tweak it so you
only have to scan the data once. It isn't very long.
ftp://ftp.openbsd.org/pub/OpenBSD/src/lib/libc/string/strlcpy.c
If you are going to plug in this low level function, why not
do a version that does path joining so users don't have to
provide the /. Do we really have such a variety of
callers that need the overhead of varargs?
..wayne..
--- util.c2 Jan 2004 07:31:02 - 1.123
+++ util.c20 Jan 2004 01:14:34 -
@@ -553,6 +553,36 @@ void strlower(char *s)
}
}
+/* Join any number of strings together, putting them in dest. The return
+ * value is the length of all the strings, regardless of whether they fit in
+ * destsize or not. Just NULL-terminate the list of strings. This function
+ * is _much_ faster than a comparable snprintf() call. */
+size_t stringjoin(char *dest, size_t destsize, ...)
+{
+ va_list ap;
+ size_t len, ret = 0;
+ const char *src;
+
+ va_start(ap, destsize);
+ while (1) {
+ if (!(src = va_arg(ap, const char *)))
+ break;
+ len = strlen(src);
+ ret += len;
+ if (destsize 1) {
+ if (len = destsize)
+ len = destsize - 1;
+ memcpy(dest, src, len);
+ destsize -= len;
+ dest += len;
+ }
+ }
+ *dest = '\0';
+ va_end(ap);
+
+ return ret;
+}
+
void clean_fname(char *name)
{
char *p;
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Improving name-truncation detection
On Mon, Jan 19, 2004 at 07:09:29PM -0800, Wayne Davison wrote: On Mon, Jan 19, 2004 at 06:46:48PM -0800, jw schultz wrote: If you're going to do the strlen(src) and whatnot you might as well just snag the strlcpy source and tweak it so you only have to scan the data once. I snagged rsync's version of strlcpy() from the lib/compat.c file. I just tried the BSD version, and it's 2.33 times slower on my Linux box (the strlen() and memcpy() functions must be using the string-oriented processor functions, but the BSD version is not). The memcpy may outperform a loop if it uses the gcc builtin. As long as the string is short the strlen will just preload the L1, if it is long it will thrash the L1. One possiblility would be to not do the strlen on the first src arg which tends to be a dirname and will be pre-vetted for length. If you are going to plug in this low level function, why not do a version that does path joining so users don't have to provide the /. Just because I thought this was more generic, and there are several sections of code that join strings without including a /. However, the joining of two strings with a '/' in between is certainly the most common join, so it might be worth a dedicated function just for that idiom. I wasn't saying instead but also. Do we really have such a variety of callers that need the overhead of varargs? In my timings it wasn't much of a hit -- in 900,000 iterations it caused an extra ~.03 user-seconds of a total of ~1.30 user-seconds. FYI, the rsync code currently has string-joins that range from 2 - 4 items. ..wayne.. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete-sent-files (AKA --move-files)
On Sat, Jan 17, 2004 at 09:02:27AM -0800, Wayne Davison wrote: On Fri, Jan 16, 2004 at 10:24:02PM -0800, jw schultz wrote: Long term this would give us the ability to log at the right point, after it is actually been completed but done on the local end. Right now a push logs the completion when the sender has finished with the file even if the last bit fails. The code logs whatever the local side did. If the local side sent data, it logs that the send happened and what the stats were. If the local side received the file, it logs that instead. It doesn't currently log whether the file was successfully updated. Almost but not quite. If the generator creates a non-file node it logs it regardless of local or remote. What i'm suggesting is that the logging (stdout) can be improved to actaully indicate completion. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Buffer Overflow?
On Sat, Jan 17, 2004 at 10:31:54PM -0800, Chuck Wolber wrote: Just got this from our nightly backup rsync: overflow: flags=0x6e l1=99 l2=1952984691 lastname=var/www/manual/mod/mod_php4/de/function.get-exte ERROR: buffer overflow in receive_file_entry rsync error: error allocating core memory buffers (code 22) at util.c(238) Command exited with non-zero status 22 real 5.15 user 1.00 sys 0.88 Rsync version information: [EMAIL PROTECTED] root]# rsync --version rsync version 2.5.4 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 64-bit system inums, 64-bit internal inums Maybe it is time to upgrade. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Any (known) scaling issues?
On Fri, Jan 16, 2004 at 03:55:42PM -0800, Bret Foreman wrote: I'm considering using rsync in our data center but I'm worried about whether it will scale to the numbers and sizes we deal with. We would be moving up to a terabyte in a typical sync, consisting of about a million files. Our data mover machines are RedHat Linux Advanced Server 2.1 and all the sources and destinations are NFS mounts. The data is stored on big NFS file servers. The destination will typically be empty and rsync will have to copy everything. However, the copy operation takes many hours and often gets interrupted by an outage. In that case, the operator should be able to restart the process and it resumes where it left off. The current, less than desirable, method uses tar. In the event of an outage, everything needs to be copied again. I'm hoping rsync could avoid this and pick up where it left off. There are really two scaling problems here: 1) Number and size of files - What are the theoretical limits in rsycn? What are the demonstrated maxima? 2) Performance - The current tar-based method breaks the mount points down into (a few dozen) subdirectories and runs multiple tar processes. This does a much better job of keeping the GigE pipes full than a single process and allows the load to be spread over the 4 CPUs in the Linux box. Is there a better way to do this with rsync or would we do the same thing, generate one rsync call for each subdirectory? A major drawback of the subdirectory approach is that tuning to find the optimum number of copy processes is almost impossible. Is anyone looking at multithreading rsync to copy many files at once and get more CPU utilization from a multi-CPU machine? We're moving about 10 terabytes a week (and rising) so whatever we use has to keep those GigE pipes full. The numbers you site should be no problem for rsync. However, the scenario is one that rsync has no real advantage and several disadvantages. You are copying, not syncing so rsync will be slower. Your network is faster than the disks and rsync is designed for disks several times faster than the network. Rsync is even worse over NFS and you are doing NFS to NFS copies. All in all, i wouldn't use rsync. My inclination would be to use cpio -p with no -u. The one thing rsync gets you is checksumming and NFS over udp has a measurable data corruption rate but caches are likely to defeat rsync's checksums so a seperate checksum cycle would still be wanted. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete-sent-files (AKA --move-files)
On Fri, Jan 16, 2004 at 09:30:57PM -0800, Wayne Davison wrote: Yes, it's time once again to return to the subject of moving files. With the recent changes to the communications code between the receiver and the generator, there is now a non-clogging channel that we can use to signal the sender when a file has been successfully transferred, which allows us delete the original for all transferred files. I have in the past waffled on whether this feature needs to be in rsync. I'm currently back on the side that it's a nice thing to support. YMMV. Here's a new implementation of the feature that adds a generic message (MSG_SUCCESS) for the receiver to send back to the sender (through the generator). I made the generator simply forward this message on to the sender, but to do that it means that the generator must be in multi- plexed IO mode, which it used to be only when it was on the server side. My patch adds a new internal flag that lets the code request that the generator have messaging capability even when it is on the client side (the non-delete-specific variable makes the code more generic). The one not-so-nice thing about this setup is that the sender process gets the MSG_SUCCESS for a particular index when it is down in the I/O code, and that code doesn't know about the file list. I decided to make this code call a new function, successful_send(), which is back in sender.c. This function is the one that handles translating the index into a file name and deleting the source file (assuming that the delete_sent_files flag is on, which is currently the only time that MSG_SUCCESS gets sent). I also added a run-time flag to mark the items we sent off to the receiver, just to make sure that nothing funny is going on in the sequence of events (aside: the sender side has no copy-on-write issues to make us avoid tweaking the flags). So, feel free to take a look and see if you like what I've done. I'll leave aside the issue of whether we want a delete-after. What you have to underpin the delete-after is almost a per-file status being communicated. Long term this would give us the ability to log at the right point, after it is actually been completed but done on the local end. Right now a push logs the completion when the sender has finished with the file even if the last bit fails. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: 2.6.0 backwards compatibility
On Thu, Jan 15, 2004 at 01:08:34PM -0500, Piela, Ben wrote: Hello all, I saw on the Release notes that 2.6.0 is using protocol version 27 which is different than 2.5.7 and 2.5.6. I did a quick peak through the rest of the release notes and quickly browsed the archives on the list and I didn't see any notes on how well 2.6.0 works with older versions of rsync. At our site, we do have 2.5.7 and 2.5.6 installed. Are there any documented problems when using 2.6.0 with a down revved version on another system? The code is designed to allow backward compatability. When a newer version is communicating with an older the will use the highest common protocol version. We do limit the backwards compatability somewhat and will exit with an error if you try to connect between incompatible versions. Beginning with 2.6.0 there is also a warning if connecting to a version that may not be supported by the next release. In addition to the protocol errors there may be newer command-line options so if you are using a new client and try to use a command-line option not supported by an older server the server will exit with an error message. For the most part you don't need to worry about the protocol versions. On the other hand in addition to feature enhancements, over time we fix bugs and improve performance and if you do have problems only limited support is given to older versions. As for your situation 2.5.6 and 2.5.7 are not that old. Some distributions are still shipping much older versions. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Resolving problems in the generator-receiver pipes
On Thu, Jan 15, 2004 at 12:11:39PM -0800, Wayne Davison wrote: When I was working on the the hard-link change, I noticed that many of the hard-link verbose messages were getting lost. These messages get output very near the end of the transfer, and it turns out that the reason for the loss was that there are two pipes flowing from the generator and the receiver, and it was possible for the we're all done message to get received down the redo pipe without all of the messages getting sent down the error pipe. It's also a long- known bug that the redo pipe can clog (but only if a really large number of redo items build up). In looking at this code, I figured out that it would be much better to just dump the redo pipe and use the error pipe as a message pipe. Since it already uses the same transfer protocol as the multiplexed IO between other components, it turned out to be a very simple thing to add a REDO and a DONE message to the existing code. This gets rid of the redo pipe and all of its problems. At the same time I though it would be good to separate the existing message-sending code out of the log-file code. This is because one other non-logging message is already supported (the data stream in multiplexed IO) so the addition of two extra messages makes it a better fit for the io.c code than the log.c code (i.e. when the log code needs to send a message to another program, it calls the code in io.c to do it instead of doing it itself). Finally, the flushing code was tweaked to allow it to signal that the writing out of the receiver-generator pipe should be completely flushed (from the in-memory list of messages). This allows the final flushing to complete more efficiently than calling io_flush() in a loop. In my exuberance for having finally solved the redo-hang problem in a much better way than my previous suggested fixes for it, I went ahead and checked in my changes. However, feedback is still gratefully accepted. The changes can be found here: http://www.blorf.net/redo.patch What i looked at from the cvs digest looked like decent code. It was just lacking the overview you provided here. This sounds pretty good. Just needs some thorough testing. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Progress reporting: N more to check
On Mon, Jan 12, 2004 at 10:37:40PM -0800, Wayne Davison wrote: A recent posting here got me thinking about having the --progress output tell the user about how many files were left to go in the transfer. I submit the attached patch which outputs an extra suffix onto the progress line at the end of each file's transfer (so it only appears once per file, not on every status update). The output would look like this: [...] flist.c 35671 100%6.67MB/s0:00:00 (#3; 262 more to check) flist.o 87288 100%4.59MB/s0:00:00 (#4; 261 more to check) log.c 13812 100% 12.50MB/s0:00:00 (#5; 221 more to check) [...] The to check part is perhaps a little verbose, but it helps to alert the user that we're not talking about how many files are left to transfer (since we won't know that number until we check them) and the more number will often not go down to zero (if the final file in the list is up-to-date). Thoughts? Suggestions? Seems over-complicated to me. How about (%d of %d files)\n, cur_index, flist-count; For that you just need cur_index to come from i in recv_files() or send_file(); That gives index into flist as it is being walked. If you wanted to have something more complicated i'd go with using bytecounts of transferred plus skipped as a percentage of total so when complete it would be 100% -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Progress reporting: N more to check
If you wanted to have something more complicated i'd go with using bytecounts of transferred plus skipped as a percentage of total so when complete it would be 100% Skip this part. Short of some ugly code only the generator process can knows the bytecounts of skipped data but it is the sender or receiver that reports progress. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Progress reporting: N more to check
On Tue, Jan 13, 2004 at 01:59:18AM -0800, Wayne Davison wrote: On Mon, Jan 12, 2004 at 11:46:05PM -0800, jw schultz wrote: How about (%d of %d files)\n, cur_index, flist-count; I tried that, but I didn't like it. The first number in my patch tells you the current count of how many files were updated, not the offset within the list, which I think is more useful to know (yeah --stats gives that count transferred at the end, but this gives you a running count and the total at the end without all the other stats). It also has a count-down that approaches 0 in the second part, which is friendlier for large numbers of files. Being able to see at a glance in a really large transfer that there are just 3000 files to go, and only 394 files had needed to be updated so far would be cool. The simpler x-of-y stat can convey the amount to go via subtraction (e.g. subtract 29837 from 32725), plus a constant reminder of the total file count (which is only visible near the start of the transfer in my scheme). Without knowing the size of flist the number of files to check doesn't tell me where i am in the list statically which is what i would want to know. I also would want it to be such that i could know this from an single entry. No back-scroll of a 52,197 lines to find the total file count. If you want a rolling count of files updated that's fine. I just thought it was more complicated than was needed. I'd still like to know the position in the file list either as a percentage or %d of %d. Whether that is counting down or up i don't care. I don't think the two are mutually exclusive. So, I personally like my suggestion better than a (file %d of %d) display. Dunno if I'm in the minority on that or not, though. You cut the text of your format but here's a few suggestions (sync 1034, 31.5% more) A little vague but is the shortest. (sync 1034 of 1450, 2963 more) | | | | | flist-count - flist_index | flist_index files_transfered Relatively clear with all three numbers changing every entry. Most should be able to figure out what the three numbers mean as it scrolls. The first number increments by one while the second jumps as files are skipped. The third gives you a jumping countdown. (sync 1034, 1450 of 4413) | | | | | flist-count | flist_index files_transfered My own preference, The first number increments by one, the second jumps (as in previous example) to converge on the last number which is fixed. All of which would meet both requirements. The use of # as a prefix usually indicates an index number or address, not a count. (FYI, I think I've talked myself out of the extra more to check Agreed. verbage. It should be enough to just say more and mention in the manpage that it means more to check and not more to transfer.) All the numbers should be explained in the manpage. If you'd like to put an explanation of the --stats numbers as well that might be good too. Since i don't much care i've never bothered to research them but it seems to be a fairly frequent question on the list. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Unexpected timeout in rsync
On Tue, Jan 13, 2004 at 06:50:28PM +, John Hunt wrote: Hi I am having trouble with intermittent timeouts when backing up a largish set of files from /usr using rsync. Approx 125,000 files, 2GB. Backup is to same machine, from a partition on hda to one on hdb. I am using ribs 2.1 (slightly modified to ignore rsync error 23), run with nice -n 5 via cron. (for ribs visit http://rustyparts.com/phpBB/viewforum.php?f=9 ) Typical error report is: rsync error: timeout in data send/receive (code 30) at io.c(103) rsync: connection unexpectedly closed (2613404 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) rsync: connection unexpectedly closed (2613404 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) rsync return value = 12 The rsync command being generated by ribs is: /usr/bin/rsync -artpl --delete --delete-excluded --stats --timeout=600 --exclude=/usr/local/* -e ssh [EMAIL PROTECTED]:/usr /backups/hdb/system/usr/weekly.0 /tmp/ribs-system-weekly_b4tInu 21 I've just run this from the command line (at default priority) and it took only 7 minutes to complete. Increasing the timeout from 180 (ribs default) to I assume you mean fail. 600 has not improved matters. On the other hand a 6GB set of files, mostly jpegs, appears to be handled reliably. As are all my other ribs backups via cron, generally a few 100MB. Mandrake 9.2, rsync 2.5.6, open ssh 3.6.1p2. Any suggestions? It sounds like a something is causing a hang. I'd start by adding -v to see when it is hanging. Perhaps -vv. You might want to fsck the filesystem. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Performance problem
On Mon, Jan 12, 2004 at 06:22:04PM -0500, Andrew Boyce-Lewis wrote: Hi, I am running rsync version 2.5.7 (stock distro on redhat linux, ES and 9) to rsync a directory with ~300k files in it from a machine on a 10Mbit internet tap to a machine with a 100Mbit internet tap. The problem is that I am only getting about 500Kbps during the transfer. I have tested link speed by scping a file between the two systems, performance was approximately 7.5Mbit/sec. Ping time between the two locations is about 2.5 ms (only 20 miles apart) This is obviously a problem. I have also tried rsyncing just a few files It is? Rsync and scp do two completely different things and their performance is not comparable. (say 400) to rule out problems related to huge file lists, however I had the same problem. Is this a known problem? Am I missing something really obvious? I have tried testing drive performance in each machine and have found it to be acceptable. But how is the filesystem performance at doing what rsync does to it? Raw sequential throughput is irrelevant. The line that I am using to do the rsync is the following: rsync -avvu --exclude-from=/root/sync_list.exclude --delete -e ssh --stats /data/ dest:/data/ Try --whole-file; if that improves things you are disk bound. Drop to just one -v; -vv is for debugging and adversely affects network buffering. It probably isn't your exclude file but a large exclude file would add overhead. 2.5.7 is a security release with minimal changes from 2.5.6 which is almost a year old. There have been a lot of changes since then most of which are in 2.6.0. Since 2.6.0 there have been a number of changes related to performance including one that dramatically improves the network performance. All of this you would have known if you had read the list archives or checked NEWS. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: chroot and pwd.db file not found ...
On Thu, Jan 08, 2004 at 01:11:43PM -0500, Bill Moran wrote: This has been asked before, but my question is somewhat different. On FreeBSD, the message: rsyncd[520]: /etc/pwd.db: No such file or directory is obviously caused by a chrooted rsyncd being unable to access /etc/pwd.db to get at the password database to convert UIDs to names and back. Workorounds of putting /etc/pwd.db in the chrooted directory have been offered, and that's one solution. However, my questions are these: 1) Does it hurt anything for rsyncd to not be able to resolve uids? It doesn't seem to affect the function of the system. 2) If it's not hurting anything, is there a way to stop these messages? Thanks in advance for any suggestions. The messages are being generated by the library, not rsync. I'd start by trying the --numeric-ids option. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: start a batch after rsync
On Fri, Jan 09, 2004 at 11:38:20AM +0100, Andreas Moroder wrote: Hello, I would like to replicate one file from one machine to a second one with rsync. After the file is copied ( rsync in read mode ), if the file has changed i must start a small batch, if it has not change I must do nothing. What's the easiest way to do this ? What constitutes easiest will depend on a number of factors. Until you have thought of at least three ways of doing it you don't know the problem well enough. None of the first four that come to mind are difficult. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problems with rsync with dirctories with spaces in the name
On Fri, Jan 09, 2004 at 10:55:38PM +0800, Nunuz Yourbiz wrote: Hi guys I have read read the why's and whyfores and howto's and whento's about posting for help, but I'm not sure whether this has already been addressed - plenty other Win/spaces-in-dir-names issues have been. I was running rsync to copy stuff off of a Windows disk onto a Windows disk on another machine via Linux - Mdk 9.1 rsync - (rsync-2.5.7-0.1.92mdk) , when I first noticed the problem. I have reproduced it below. The command I used was rsync -avz -e ssh /mnt/D/Program\ Files/Activision/WizardsWarriors/* [EMAIL PROTECTED]:/mnt/downloads/Program\ Files/Activision This command gave me the following in the /mnt/downloads directory [EMAIL PROTECTED] richard1]$ ll /mnt/downloads/ total 5344753 drwxrwxr-x 29 richard1 richard1 4184 Jan 9 21:54 Program/ It seems that the rsync (or the shell or whatever) loses everything after the whitespace (even though it has a \) Is this a documented bug? If so is it a problem with rsync or the shell? I did try with scp and got the following [EMAIL PROTECTED] richard1]$ scp -r /mnt/D/Program\ Files/Activision/WizardsWarriors/* [EMAIL PROTECTED]:/mnt/downloads/Program\ Files/Activision scp: ambiguous target So I suppose it might be a shell problem It is a shell problem of sorts. Your source path should be OK but you would be better off leaving off the trailing * so the shell won't glob. The real problem is the destination path. The destination path has a space in it but you only escaped it once. That protects it from the local shell but not the remote shell. Rsync is going to invoke ssh with the destination path on the command-line which will cause ssh to invoke your login shell on the remote end with that command-line as arguments. The space needs to be escaped or quoted in a way that will get to the remote end. That either means nested quotes or stacked backslashes. Hmm, just a thought here for the other developers: Perhaps rsync should check the source/dest portions of remote argument list for whitespace and if there but no quotes or backslashes insert backslashes or surrounding quotes. That way a path could be specified for local or remote in the same way even if it has spaces in it. Having a test for quotes or backslashes would provide backwards compatability. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Fri Jan 9 13:55:56 2004 Author: jw Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv17778 Modified Files: flist.c Log Message: Finished the last_* renameing and fixed a cast.. Revisions: flist.c 1.154 = 1.155 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c.diff?r1=1.154r2=1.155 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/testsuite
Date: Fri Jan 9 14:02:00 2004 Author: jw Update of /data/cvs/rsync/testsuite In directory dp.samba.org:/tmp/cvs-serv18994/testsuite Modified Files: devices.test Log Message: Added a few more devices to the devices.test to hopefully test same, same high and different device numbers. Revisions: devices.test1.3 = 1.4 http://www.samba.org/cgi-bin/cvsweb/rsync/testsuite/devices.test.diff?r1=1.3r2=1.4 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
Re: [patch] making rsync less verbose
On Tue, Jan 06, 2004 at 06:51:41PM +0100, Dick Streefland wrote: Unless you suppress all output with the -q option, rsync will always print an initial building/receiving file list ... done line. In my opinion, this is a bit superfluous. When I want to see a progress indication, I can use the --progress option. Another issue is the 3-line transfer speed and speedup factor report at the end. So every rsync invocation produces at least four lines of output. Attached are two patches to reduce the verbosity of rsync. The first one removes the initial line. The second patch adds a new option -s/--speedup to control the generation of the speedup report, which is now disabled by default. What do you think? Would you consider something like this for future versions of rsync? Very good providing a patch to go with the suggestion! I agree that the building/receiving file list...done messages are often unwanted but we've already hashed out the issue of proliferation of verbosity options and don't want to go there. I have something in the pipeline for keyword based control of verbosity partially to avoid this sort of proliferation of verbosity control options. Probably see it in 2.6.2. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync performance question
On Thu, Jan 08, 2004 at 01:05:25PM -0500, Rick Frerichs wrote: Hello, I seem to be having a performance problem with rsync. I have done some testing of rsync and ftp. If I do a transfer (either way) with ftp, I get about 500 Kbytes/sec. Using rsync to do the same transfer (either way) I only get about 50 Kbytes/sec. I am only testing straight file copies. There is a firewall which I believe is configured properly. The ethernet interface is running at 10 Mbit full duplex. Does anyone have an idea of where I should be looking to solve this problem? A mirror :) Seriously, rsync increases the load on CPU, memory and disk for the sake of reducing load on the network for synchronising files and directory trees. Rsync is less efficient than most copy utilities at straight copying and it's performance as a copy utility should not be considered when evaluating it as a synchronisation utility. The performance hit you are getting seems greater than i would normally expect. It may be rsync is pushing you over a threshold into a swap storm. Perhaps you are using some odd options (-c, -T) that slow things down dramatically. A 10MbFD network sounds very odd, FD would be switched or on a crossover 10Mb would be obsolete hardware indicating that possibly you have a very slow CPU, small memory and slow disks. As you provide no information regarding system configuration, versions or command-line options i can only speak in generalities. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: TODO hardlink performance optimizations
On Wed, Jan 07, 2004 at 01:04:34AM -0600, John Van Essen wrote:
On Tue, 6 Jan 2004 15:11:54 -0800, jw schultz [EMAIL PROTECTED] wrote:
On Tue, Jan 06, 2004 at 02:04:19AM -0600, John Van Essen wrote:
On Mon, 5 Jan 2004, jw schultz [EMAIL PROTECTED] wrote:
[snip]
union links {
struct idev {
INO64_T inode;
DEV64_T dev;
}
struct hlinks {
intcount;
file_struct**link_list;
}
}
[snip]
Now that you have opened the door for re-using the inode and
dev area in file_struct, I can make my initial suggestion that
requires two 'additional' pointers in file_struct. I didn't
suggest it before because I didn't want to add to the struct.
I propose to reuse the DEV amd INODE areas to store two new
file_struct pointers (the 2nd part of the union). These would
be set during the post-qsort phase during init_hard_links.
The first pointer links together the file_structs for each
hardlink group in a list in the same order as in the qsort
so they can be walked by your proposed link-dest method.
The second pointer points to the head (first file_struct of
a hardlink group).
For each file_struct that is modified in this way a flag bit
needs to be set indicating so.
No it doesn't. After we walk the qsorted hlink_list all
file_structs-links pointers either point to a hlinks or
are NULL. If there are no links file_structs-links is
NULL.
If you qsort the entire list without first filtering out any
entries (e.g. !IS_REG), then yes, that would be true.
Either NULL it during the hlink_list walk or don't populate
file_struct-links while building the file list (saving a
malloc) for unlinkable files.
Then, if the file_struct address equals its head pointer, then
you are at a head of a hardlink list which can be processed
using the new method using link-dest that you outlined earlier.
So you are proposing a singly linked list:
struct hlinks {
file_struct*head;
file_struct*next;
}
Yes.
That would work too. With the pointers into the array
you compare the first in the array with yourself instead of
head. I preferred knowing in advance how many links there
were.
But there is no need to know exactly how many links, is there?
Not at this time.
I'm still not clear on what exactly will happen during that new
processing method. I assume that if the head file does not exist,
it will find any existant file and hardlink it to the head file,
and if not found, will transfer the head file. Is that correct?
Not exactly. Like my earlier post if the head (your term)
doesn't exist (existence including compare_dest) we iterate
over the link set and the first that does exist is used for
fnamecmp. But that is phase two or three.
Let me put it another way... Using your proposed method, after
the 'head' file is processed, will it now exist on the receiver
so that its siblings can be hardlinked to it?
Yes.
Processing of non-head files (file_struct address not equal to
head pointer) can be skipped since they are hardlinked later.
The final walk-through that creates the hardlinks for the
non-head files can walk the qsorted list and use the head
pointer as the target for the hardlinks of the non-nead files.
Actually, if we could do the hardlinks of non-head files as
we encounter them while walking the file list and used your
singly linked lists we could free the hlink_list after
traversal. The problem is that we would need to be sure the
head had completed processing so the hardlink would have to
be created in receiver and that gets ugly.
Right. You've just explained the here-to-fore unknown reason
why that sibling hlinking was being done in a separate, final
phase. If we keep it that way, I'd like to see a comment added
to explain what you just explained.
But here's an idea, still using my proposed head/next struct...
- Make *next a circular list (the last points to the first).
- Leave *head NULL initially.
- When a file of a hlink group is examined, and *head is NULL,
then it is the first one of that group to be processed, and
it becomes the head. All the *head pointers are set to its
file_struct address.
- Subsequent processing of siblings can immediately hardlink
them to the head file.
The drawback is that this will invoke CopyOnWrite (as dicussed
earlier in this thread). To avoid that, *head would have to
point outside to a group head pointer, which would then be set.
So you'd need an array of pointers of the same size as the
number of distinct hardlink groups. Say! We already have the
sorted hlink_list. We could just point to the first element
of the group (setting it to NULL initially) after creating
the circularly
Re: TODO hardlink performance optimizations
On Wed, Jan 07, 2004 at 01:33:43AM -0600, John Van Essen wrote: On Tue, 6 Jan 2004 22:33:06 -0800, Wayne Davison [EMAIL PROTECTED] wrote: I'd suggest also changing the last line of the function: -return file_compare(f1, f2); +return file_compare(f1p, f2p); This is because the old way asks the compiler to take the address of f1 and f2, thus forcing them to become real stack variables. Changing the code to use the passed-in f1p and f2p allows the compiler to leave both f1 and f2 as registers (if possible). Good point, but I have an even better suggestion, now that I finally understand the nuts and bolts of all the hlink.c code. The file_compare() is invoked when the dev and inode values match in order to present a consistent sorting order during the sort. There is no compelling reason to have the hlink list be sorted alphabetically. It just has to sort consistently. So the final comparison can be done on the addresses of the file_structs, since they are not moved around and will remain constant: return ( ( f1 f2 ) ? -1 : ( f1 f2 ) ); (Unsure if the code is right, but you get my drift.) For filesets with many hardlinks, this will use less CPU time. There may well be good reason for having the link sets subsorted consistantly with the file list. See my notes regarding COW, fork and the modification of the link info. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem with many files in rsync server directory ?
On Wed, Jan 07, 2004 at 10:26:16AM -, Jon Hirst wrote: Having had a night to sleep on this I think rsync's limit on filename globbing needs pointing out more clearly. I think we need: 1) An entry in the FAQ (Done) 2) A better error message from rsync when it exceeds the limit. Saying: rsync: read error: Connection reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(201) doesn't help many users. Not even programmers with 20 years experience, like me ;-) 3) How about adding a file called LIMITS to the source distribution that tells system administrators and users of the limits that are built in to rsync, and how they can be changed. 4) Or maybe even - horror or horrors - some comments in the source file rsync.h. 5) It trumpeted from the mountain tops (and maybe in the documentation somewhere) that using * to get all files in a directory is stupid or ignorant. a) * and ? are globbed by the shell unless quoted and may produce unexpected behaviour. b) There are limits to the size of command-lines. c) filenames with spaces glob badly. d) The only time the argument globbing is done by rsync is on the daemon, all other times it is done one shell or another. I've lost track of the number of times someone has complained on this list because blah/blah/* didn't behave as he expected and the problem went away when he dropped the unnecessary wildcard. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Copying hard-linked tree structure
On Wed, Jan 07, 2004 at 03:18:40PM -0600, Max Kipness wrote: I have a tree structure on one server similar to the following: /Current /01-04-2003 /01-03-2003 etc... /Current holds the most recent rsynced data, and the date directories are created with cp -al on a daily basis so they are hard-linked. I'm going back 60 days. The question is how can I move this entire structure to a new server and preserve the links from the date directories to the /Current directory? Well, I ended up rsyncing the root directory to the new server with the -H option and it seemed to work. I have 30 directories for 30 days of rotating backups. However, I had a dir called /Current that had 12Gbs and then all the /date directories had 120mb, 60mb, etc...the daily changes that occurred. Well now the directory called /01-01-2004 has 12Gb and /Current has like 100mb. I guess /01-01-2004 went first do to sorting. It has to do with the tool you are using to measure them. Anyway to change /Current back as the real directory? Or does it even matter? What do you man real. With hardlinks all links for an inode are equal. I'm using du --max-depth=1 -h on the root dir. The actual file(s) has to be stored in some directory, right? And then the hard links point to this directory. Well they are all pointing to /01-01-2004 instead of /Current. Only symlinks point to another directory entry. All hardlinks are equal. The way you are using it du is simply using the directory order to pick which paths to descend first. ls -f should list the directory in the same order that du does. On the source system the directory order will be semi-random if you have been creating and deleting entries for awhile. On the destination they will be in lexical order because that was the creation order by rsync and you haven't mixed that up yet. Try this, mv 01-01-2004 01-01-2004-a-long-name mv 01-01-2004-a-long-name 01-01-2004 Now 01-01-2004 will likely not be the first on the list from ls -f and another direcotory will likely be held responsible for the space by du. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
