Re: delta copies
Not necessarily. Depending on how pg_dump works, it could be that small changes to the database are resulting in unnecessarily large changes to the dump. Make sure you are using the uncompressed format because most compression algorithms defeat the delta-transfer algorithm almost completely. Then you might take a look at two consecutive dumps and check whether records common to both appear in the same order in each dump. (If pg_dump is dumping the records in a different order each time, that would also defeat the delta-transfer algorithm because no block of several consecutive records could be matched.) I was wondering if it would be possible to add a switch (probably coupled to -v(+) ) that would report the number of matched blocks per file. Maybe even with the offset of the block. That of course wouldn't help much in a production environment, but it could help while setting up the whole backup strategy. This would have helped the original poster finding out if rsync is working correctly. bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: delta copies
On 9/24/07, Fabian Cenedese [EMAIL PROTECTED] wrote: I was wondering if it would be possible to add a switch (probably coupled to -v(+) ) that would report the number of matched blocks per file. Maybe even with the offset of the block. Rsync already lists all matched blocks by their offsets and lengths if given -vvv. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: delta copies
At 07:51 24.09.2007 -0400, Matt McCutchen wrote: On 9/24/07, Fabian Cenedese [EMAIL PROTECTED] wrote: I was wondering if it would be possible to add a switch (probably coupled to -v(+) ) that would report the number of matched blocks per file. Maybe even with the offset of the block. Rsync already lists all matched blocks by their offsets and lengths if given -vvv. Right, sorry about that. I didn't find it because the manual doesn't state what info is given for more than 2 v's, only used for debugging :) bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--delete not working
Hi, I've set up a rsync deamon with the following config, on the remote side. [deploiement] path = /data/one_tree use chroot = yes max connections=2 lock file = /var/lock/rsyncd read only = no uid = nobody gid = nogroup hosts allow = someIP from the source host I do : rsync -rtlv --exclude='.svn/**' --exclude=.svn --delete preprod/ rsync://remote-host/deploiement/preprod/ the --dry-run gives me some file to delete. but when I run the command without --dry-run, nothing is deleted? Regards, Sylvain. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete not working = +filtering disable delete!
Hi, Sylvain Viart a écrit : rsync -rtlv --exclude='.svn/**' --exclude=.svn --delete preprod/ rsync://remote-host/deploiement/preprod/ the --dry-run gives me some file to delete. but when I run the command without --dry-run, nothing is deleted? I think I've it. The remote folder in the deletion list contains .svn subfolder from previous rsync without filtering. remote-host:/data/one_tree/preprod# find . -name .svn -type d | xargs rm -rf hehe, much better. why rsync doesn't complain about it can't remove no empty folder of something? Regards, Sylvain. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete not working
On 9/24/07, Sylvain Viart [EMAIL PROTECTED] wrote: from the source host I do : rsync -rtlv --exclude='.svn/**' --exclude=.svn --delete preprod/ rsync://remote-host/deploiement/preprod/ the --dry-run gives me some file to delete. but when I run the command without --dry-run, nothing is deleted? I bet the discrepancy has to do with your exclude filter. The --exclude=.svn tells the receiver never to delete a file named .svn, so it does not delete such a file's parent directories either, even if they are extraneous. Deletions of parent directories are neither itemized nor performed on a real run, but unfortunately, rsync 2.6.9 and earlier incorrectly itemize them on a dry run. For more information see: https://bugzilla.samba.org/show_bug.cgi?id=3825 If this isn't your situation, give me some more information and I'll try and figure out what the problem is. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Mapped Drive
I am having trouble running rsync over a mapped drive. Basically it only copies whole files. I use the -rvcS switches. Any suggestions? TIA _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Mapped Drive
On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: I am having trouble running rsync over a mapped drive. Basically it only copies whole files. I use the –rvcS switches. Any suggestions? The delta-transfer algorithm reduces only the communication between the sending and receiving rsync processes; each process reads and writes entire files in whatever filesystem it is using (local or networked). If you want to benefit from delta transfers, one of the rsync processes has to actually run on the system hosting the mapped drive. You can accomplish this either by remote shell or by setting up and accessing an rsync daemon on that system. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Mapped Drive
On Mon 24 Sep 2007, Stephen Zemlicka wrote: I am having trouble running rsync over a mapped drive. Basically it only copies whole files. I use the -rvcS switches. Any suggestions? From the manpage: -W, --whole-file With this option the incremental rsync algorithm is not used and the whole file is sent as-is instead. The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the disk is actually a networked filesystem). This is the default when both the source and des- tination are specified as local paths. This is because there's no advantage to updating only parts instead of the whole file. If rsync finds it needs to update a file, it now only has to read the source, write the destination (via a temp file so that the destination filename either points to the old version or the new version; not an incomplete new version. If you want that, use --inplace). If it did it incrementally, then rsync would need to read the source, read the old destination file, compare, and while writing the new temp file read from either the source or the old destination. This is more IO, and hence less efficient. Rsync's incremental algorithm is for optimizing network traffic at the expense of more local disk IO, in the assumption that the network is slower than local disk IO. If you're doing a local transfer, it optimizes the disk IO... Using a mapped drive appears to be a local disk. If you can transfer directly between the server from which the drive is mapped instead of going through the mapping, then that's preferable (and will decrease network traffic...). Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Mapped Drive
On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: What exactly is involved in the remote shell? Can this be done on a windows to windows backup? Do you have a link handy otherwise google to the rescue. Thank you very much. You set up an ssh server on the machine with the mapped drive, install an ssh client on the machine running rsync, and direct rsync to the mapped drive using a path of the form host:/cygdrive/C/path/to/mapped/drive/ (note the *single* colon). cwRsync ( http://itefix.no/cwrsync/ ) is a nice packaging containing rsync, an ssh client and server, and some glue to get the ssh server running in Windows; I recommend it. Note: you might run into the infamous remote-rsync-under-Cygwin hang bug, which is being tracked here: https://bugzilla.samba.org/show_bug.cgi?id=2208 Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
The reason I was trying this was because all I can do is map a drive. As of yet, I cannot modify the server like that. I tried the --inplace and it still did the whole file. Also, this will eventually be over the internet but right now I'm doing it locally (faster testing). I may take a peek at rdiff. Any experience with that? _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, September 24, 2007 12:24 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: What exactly is involved in the remote shell? Can this be done on a windows to windows backup? Do you have a link handy otherwise google to the rescue. Thank you very much. You set up an ssh server on the machine with the mapped drive, install an ssh client on the machine running rsync, and direct rsync to the mapped drive using a path of the form host:/cygdrive/C/path/to/mapped/drive/ (note the *single* colon). cwRsync ( http://itefix.no/cwrsync/ ) is a nice packaging containing rsync, an ssh client and server, and some glue to get the ssh server running in Windows; I recommend it. Note: you might run into the infamous remote-rsync-under-Cygwin hang bug, which is being tracked here: https://bugzilla.samba.org/show_bug.cgi?id=2208 Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Mapped Drive
On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: The reason I was trying this was because all I can do is map a drive. As of yet, I cannot modify the server like that. If the only access you have to the remote directory is to read or write it via a mapped drive, you can't hope to do any better than copying the whole file. All approaches to delta transfers rely on performing some kind of special computation on the remote machine. BTW, are you pushing files to the mapped drive or pulling files from it? Knowing that would make future discussion clearer. I may take a peek at rdiff. Any experience with that? rdiff is essentially a command-line interface to the three stages of the rsync delta-transfer algorithm. It won't work any better than rsync in this setting. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
I'm pushing. I thought rdiff could install just the changes to a dir and then when you restore, it applies the changes from the dir you specify. Is that incorrect? If rdiff does do that, I could write a gui or wrapper for it to make restores more user friendly. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, September 24, 2007 12:46 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: The reason I was trying this was because all I can do is map a drive. As of yet, I cannot modify the server like that. If the only access you have to the remote directory is to read or write it via a mapped drive, you can't hope to do any better than copying the whole file. All approaches to delta transfers rely on performing some kind of special computation on the remote machine. BTW, are you pushing files to the mapped drive or pulling files from it? Knowing that would make future discussion clearer. I may take a peek at rdiff. Any experience with that? rdiff is essentially a command-line interface to the three stages of the rsync delta-transfer algorithm. It won't work any better than rsync in this setting. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
using rsync on selinux
I'm trying to use rsync to synchronize customer application files across multiple WEB servers. The actual copying of the files works fine. The problem is that it changes the selinux security context of some of the files. Since there are over 2000 files for each customer, it becomes a major issue to correct the security context after the fact as not all have the same context. Searching the WEB, I have found references to a -X option to rsync, but this option does not exist in the RHEL4 version of rsync. I have an open ticket with RedHat, but no resolution yet. Is there rsync source code somewhere, compatible with my OS, I could compile that would fix my problem? My OS is RHEL4 2.6.9-55.0.2.ELsmp The rsync command I am using is as follows: cd /local/base/directory rsync -avz localdirectory rsync://remotehost/modulename Richard Riley System Administrator Email: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Procuri Inc. www.procuri.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
I think I got it (with the help of a friend). You can, in fact, rsync to a mapped drive efficiently. You must include --no-whole-file. My 100MB file only transferred a few MB using that method and it opened up with the changes just fine. Now it works just like I want. It's not the quickest but definitely quicker than uploading hundreds of MB for each database. Thanks you for the assistance. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, September 24, 2007 12:46 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: The reason I was trying this was because all I can do is map a drive. As of yet, I cannot modify the server like that. If the only access you have to the remote directory is to read or write it via a mapped drive, you can't hope to do any better than copying the whole file. All approaches to delta transfers rely on performing some kind of special computation on the remote machine. BTW, are you pushing files to the mapped drive or pulling files from it? Knowing that would make future discussion clearer. I may take a peek at rdiff. Any experience with that? rdiff is essentially a command-line interface to the three stages of the rsync delta-transfer algorithm. It won't work any better than rsync in this setting. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: using rsync on selinux
On 9/24/07, Richard Riley [EMAIL PROTECTED] wrote: Searching the WEB, I have found references to a -X option to rsync, but this option does not exist in the RHEL4 version of rsync. Is there rsync source code somewhere, compatible with my OS, I could compile that would fix my problem? Yes. The latest stable release of rsync is 2.6.9, and it comes with a patch that adds support for -X ( patches/xattrs.diff ). The source package is at: http://rsync.samba.org/ftp/rsync/rsync-2.6.9.tar.gz Or you might like Fedora 7's RPM of an rsync 2.6.9 binary patched with ACL and extended attribute support: http://download.fedora.redhat.com/pub/fedora/linux/releases/7/Fedora/i386/os/Fedora/rsync-2.6.9-2.fc7.i386.rpm Note that -X works only if the rsyncs on both ends of the copy support it. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
