Strong encryption
We need to encrypt files before transferring them to the destination Host for security reasons. Encryption must be strong: IDEA, 3DES or similar. One way would be the integration of PGP into rsync. Is there any PGP integration into rsync available ? Thank´s Clemens -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Strong encryption
On Wed, Jun 05, 2002 at 11:42:12AM +0200, C.Zimmermann wrote: We need to encrypt files before transferring them to the destination Host for security reasons. Encryption must be strong: IDEA, 3DES or similar. One way would be the integration of PGP into rsync. Is there any PGP integration into rsync available ? Thank´s Clemens You are a little vague here. Do you mean you need the transmission encrypted or that you want the files to be stored encrypted on the destination? If you want the transmission encrypted use rsync over ssh or you can look into encrypted tunnels and VPN stuff. If you want them stored on the destination encrypted you will need to keep them encrypted on the source. Rsync won't be able to compare an encrypted (cyphertext) file with an unencrypted (plaintext) one. For rsync to support encryption it would need to be able to decrypt your files and that would be very insecure. If you keep the files encrypted on the source and destination you might as well use the rsync -w option because for encryption to be effective it truly does need to randomize the data rendering the special efficencies of rsync moot. Additionally, for the class of cyphers you mention to be secure they need long keys that change frequently and are best generated randomly and then encrypted using even more secure cyphers. This randomness assures that no file will be encrypted to the same cyphertext twice. Sorry if i bowled you over on the cyphertext files issues. Hopefully all you needed was a pointer to secure transmission. If not, a coherent description of why the need for encryption may help us find a way to meet your needs. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Strong encryption
On Wed, Jun 05, 2002 at 12:21:18PM +0200, C.Zimmermann wrote: If you want them stored on the destination encrypted you Yes, that´s it. The owner of the source files will be sure, that no one else can read his files on the destination host. I thought, rsync only looks at the modification date of a file and decides whether to backup this file or not. In this case, the backup could be stored encrypted. Rsync can handle encrypted files just fine. It just treats them as ordinary binary files. If the owner of the files encrypts them on the source they will be encrypted on the destination. As you have said rsync normally just looks at the modification date for deciding whether to update the destination (unless you use the -c option) But, unless the -w option is used rsync will use some rather clever (nod to rsync developers) methods to transfer only the changed the parts of changed files. It is this feature that gives rsync its speed. My comments below are to advise you that that clever feature is nullified by encrypted files. In fact for encrypted files rsync would be sub-optimal. If most/all of the changed files are encrypted i would use the -w option. Enjoy Bye Clemens will need to keep them encrypted on the source. Rsync won't be able to compare an encrypted (cyphertext) file with an unencrypted (plaintext) one. For rsync to support encryption it would need to be able to decrypt your files and that would be very insecure. If you keep the files encrypted on the source and destination you might as well use the rsync -w option because for encryption to be effective it truly does need to randomize the data rendering the special efficencies of rsync moot. Additionally, for the class of cyphers you mention to be secure they need long keys that change frequently and are best generated randomly and then encrypted using even more secure cyphers. This randomness assures that no file will be encrypted to the same cyphertext twice. Sorry if i bowled you over on the cyphertext files issues. Hopefully all you needed was a pointer to secure transmission. If not, a coherent description of why the need for encryption may help us find a way to meet your needs. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Strong encryption
Is there any PGP integration into rsync available ? No, but there's OpenSSH... quite what you're searching for =) http://www.openssh.org/ -- Lapo 'Raist' Luchini [EMAIL PROTECTED] (PGP X.509 keys available) http://www.lapo.it (ICQ UIN: 529796) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: AW: Strong encryption
I thought, rsync only looks at the modification date of a file and decides whether to backup this file or not. By default, it does not, in fact it's REAL usefullness it's that uit examines the content of the file to send just what is needed, differently from other mirroring software. -- Lapo 'Raist' Luchini [EMAIL PROTECTED] (PGP X.509 keys available) http://www.lapo.it (ICQ UIN: 529796) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Strong encryption
On Wed, Jun 05, 2002 at 03:33:23AM -0700, 'jw schultz' wrote: As you have said rsync normally just looks at the modification date And the file size. This check, to the best of my knowledge, cannot be turned off. - Adrian -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Strong encryption
On Wed, Jun 05, 2002 at 06:45:44PM +0800, Adrian Ho wrote: On Wed, Jun 05, 2002 at 03:33:23AM -0700, 'jw schultz' wrote: As you have said rsync normally just looks at the modification date And the file size. This check, to the best of my knowledge, cannot be turned off. Yes. I remembered that just after sending. A significant oversimplification on my part. Sorry. There is --size-only, --ignore-times, --modify-window and --checksum that affect the detection of changed files. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
AW: AW: Strong encryption
I thought, rsync only looks at the modification date of a file and decides whether to backup this file or not. By default, it does not, in fact it's REAL usefullness it's that uit examines the content of the file to send just what is needed, differently from other mirroring software. But beside this rsync offers the mechanism for incremental mirroring on file bases over the network, so I think it makes sence to enhance rsync with the encryption option on destination files. Its better than tar -cvf - / | pgp | ssh user@destinationhost Clemens -- Lapo 'Raist' Luchini [EMAIL PROTECTED] (PGP X.509 keys available) http://www.lapo.it (ICQ UIN: 529796) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Strong encryption
On Wed, Jun 05, 2002 at 01:22:28PM +0200, C.Zimmermann wrote: I thought, rsync only looks at the modification date of a file and decides whether to backup this file or not. By default, it does not, in fact it's REAL usefullness it's that uit examines the content of the file to send just what is needed, differently from other mirroring software. But beside this rsync offers the mechanism for incremental mirroring on file bases over the network, so I think it makes sence to enhance rsync with the encryption option on destination files. Reread my earliest response on this thread. An encryption option would require large changes to the protocol and be a real nightmare as far as keys are concerned. If you want encryption, keep the files on the source encrypted. Its better than tar -cvf - / | pgp | ssh user@destinationhost That is like saying a courier service is better than drayage (cargo hauling). -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: http://rsync.samba.org/cvs.html gives 404 Not Found
On Fri, May 31, 2002 at 05:55:14PM +0400, [EMAIL PROTECTED] wrote: Hello, http://rsync.samba.org/cvs.html gives 404 Not Found. I do not know if it is ok or not ;) Looks like another casualty of the web site rearrangement. One that works is http://www.samba.org/samba/cvs.html I see the URL you gave is referred to at http://rsync.samba.org/download.html. How should it be fixed, Martin? I found I have write access to the rsync web directory but a symlink to ../samba/cvs.html just changes the error from Not Found to Permission Denied. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Problem with --relative and symlinks
There appears to be a problem with the handling of symlinks within source directories by the --relative (-R) option. I am using rsync 2.5.5 on an SGI Origin 200 running IRIX 6.5.11f. Example: copying the contents of /usr/local/etc/inhouse/db/ from wolf to fox where /usr/local is a directory on wolf, but /usr/local is a link to /usr2/local on fox. Here's the dry-run output (some anonymizing editing has been done): --- 129 (wolf)root /usr/local/bin/rsync -a -R -v --delete -n -e /usr/local/bin/ssh --rsync-path=/usr/local/bin/rsync /usr/local/etc/inhouse/db/ fox.cisti.nrc.ca: building file list ... done deleting usr/local/etc/inhouse/db/bak/20020505130006.tar.gz deleting usr/local/etc/inhouse/db/bak/20020505010005.tar.gz /usr/local/etc/inhouse/db/bak/20020605010006.tar.gz /usr/local/etc/inhouse/db/bak/20020605130005.tar.gz /usr/local/etc/inhouse/db/maillog /usr/local/etc/inhouse/db/pri-req.log /usr/local/etc/inhouse/db/registry /usr/local/etc/inhouse/db/waitlist wrote 6474 bytes read 44 bytes 686.11 bytes/sec total size is 137212442 speedup is 21051.31 --- Here's the (abbreviated) output of the actual update: --- 131 (wolf)root /usr/local/bin/rsync -a -R -v --delete -e /usr/local/bin/ssh --rsync-path=/usr/local/bin/rsync /usr/local/etc/inhouse/db/ fox.cisti.nrc.ca: building file list ... done deleting usr/local/etc/inhouse/db/bak/20020505130006.tar.gz deleting usr/local/etc/inhouse/db/bak/20020505010005.tar.gz usr/ usr/local/ usr/local/etc/ usr/local/etc/inhouse/ usr/local/etc/inhouse/db/ /usr/local/etc/inhouse/db/Backup /usr/local/etc/inhouse/db/Makefile /usr/local/etc/inhouse/db/areas.txt usr/local/etc/inhouse/db/bak/ /usr/local/etc/inhouse/db/bak/20020506010006.tar.gz /usr/local/etc/inhouse/db/bak/20020506130005.tar.gz /usr/local/etc/inhouse/db/bak/20020507010006.tar.gz /usr/local/etc/inhouse/db/bak/20020507130006.tar.gz /usr/local/etc/inhouse/db/bak/20020605010006.tar.gz /usr/local/etc/inhouse/db/bak/20020605130005.tar.gz /usr/local/etc/inhouse/db/branch /usr/local/etc/inhouse/db/courses /usr/local/etc/inhouse/db/maillog /usr/local/etc/inhouse/db/pri-req.log /usr/local/etc/inhouse/db/registry /usr/local/etc/inhouse/db/waitlist wrote 137245608 bytes read 4164 bytes 469229.99 bytes/sec total size is 137212442 speedup is 1.00 --- The deletions are performed properly, but the /usr/local link is replaced by a /usr/local directory and then the complete contents of /usr/local/etc/inhouse/db/ is copied to this new location. My work-around is to not use -R and to fully specify the destination directory: --- 136 (wolf)root /usr/local/bin/rsync -a -v --delete -e /usr/local/bin/ssh --rsync-path=/usr/local/bin/rsync /usr/local/etc/inhouse/db/ fox.cisti.nrc.ca:/usr2/local/etc/inhouse/db/ building file list ... done bak/ bak/20020605010006.tar.gz bak/20020605130005.tar.gz maillog pri-req.log registry waitlist wrote 2551675 bytes read 52142 bytes 226418.87 bytes/sec total size is 137212442 speedup is 52.70 --- (The deletions don't show up because they were previously done.) Th work-around is okay for a single source directory, but it does not scale properly for multiple sources. --relative needs to be fixed so that it does not alter the source directory structure. - Greg Kresko National Research Council of Canada [EMAIL PROTECTED] (I am not subscribed to the rsync list) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Acls support in rsync
Hi, I already wrote about this subject last month but didn't get attention... I've found a way to transfert acls with a script and applying them. This process has already been described in this mailing list (http://lists.samba.org/pipermail/rsync/2000-March/001992.html) but there's a way to do it in a more efficient way using some getfacl options. You can dump acls of a full directory with this command : getfacl -R --skip-base --no-effect --absolute-names --access /somedir/ -R : recursive --skip-base : do not process files with standard rights --no-effect : no comments --absolute-names : get full file name --access : display file access control list, not default access control list The advantage of this command is that you get all but only effective acls, so standard access rights are still processed by rsync, and you only transfert supplementary access rights. using some sed magic (don't flame, i'm not a Regex master...): getfacl -R --skip-base --no-effect --absolute-names --access * | \ sed 's/# file: \(.*$\)/\1 /' | grep -v '^#' | tr \n , | \ sed s/ ,/ /g | sed 's/,,/\/g' | sed 's/\(.*\) user::\(.*\)/user::\2 \1/ you get a line for each acl looking like this : user::rwx,user:joe:rwx,group::rwx,group:users:rwx,mask::rwx,other::r-x myfile Processing this command on +250 000 files took less than 2 minutes on a fresh rebooted (no cache involved) xfs file system (linux 2.4.18). Running 1 hour later, it will generally run in less than 5 seconds. You just have to pipe in a file in the involved directory on the master before running rsync and apply acls on the other side after rsync completed. To apply, use this script : #!/bin/bash ### This script work only if you don't have and / in your filenames ### I hope for you... :-) cat $1 | while read acl file do myfile=$file echo Applying acl in $file... /usr/bin/setfacl -s $acl $myfile done Running time depend of the number of acls in your directory. :-) I don't know if this method work well on other file systems, but it do on xfs and is really efficient. I hope this will help acls users, so please let me know if you get success with this method. I still want to create a getfacl patch for rsync, so that acls are transfered with a shorter syntax (u::rwx,u:101:rwx,g::rwx,g:100:rwx,m::rwx,o::r-x myfile), and so you don't have to script anymore :-) So please dear rsync developpers, could you please give me some advice about what to modifiy in rsync to add acls to standard access rights synchronisation. Thank you in advance. Olivier Tarnus -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: General speed question
On Monday 03 June 2002 07:03, David Arnold wrote: We have some speed/performance issues: We have a 100M fullduplex private network setup to handle rsync transfers to our mirror server with a command like: time rsync -e ssh -avzl --delete --rsync-path=/usr/local/bin/rsync \ --exclude .netscape/cache/ --delete-excluded \ bigserver:/staff1 /mirror/bigserver Here are your speed problems: 1- If you're really on a private network, you should run rsync as a daemon, and avoid ssh. This will save many cpu cycles and some bandwidth. 2- With 100 Mbits dedicated to rsync, you shouldn't use compression (-z). This will save many cpu cycles too. :-) Take a look but i'm quite sure that your problems are actually on cpu side and not on network side... I'm actually testing rsync on shared 100Mbit switched network and it take less than 1 minute to synchronize 55G in more than 250 000 files. Olivier -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Acls support in rsync
On Wed, Jun 05, 2002 at 10:57:22PM +0200, Olivier Tarnus wrote: Hi, I already wrote about this subject last month but didn't get attention... I've found a way to transfert acls with a script and applying them. This process has already been described in this mailing list (http://lists.samba.org/pipermail/rsync/2000-March/001992.html) but there's a way to do it in a more efficient way using some getfacl options. You can dump acls of a full directory with this command : getfacl -R --skip-base --no-effect --absolute-names --access /somedir/ -R : recursive --skip-base : do not process files with standard rights --no-effect : no comments --absolute-names : get full file name --access : display file access control list, not default access control list The advantage of this command is that you get all but only effective acls, so standard access rights are still processed by rsync, and you only transfert supplementary access rights. using some sed magic (don't flame, i'm not a Regex master...): getfacl -R --skip-base --no-effect --absolute-names --access * | \ sed 's/# file: \(.*$\)/\1 /' | grep -v '^#' | tr \n , | \ sed s/ ,/ /g | sed 's/,,/\/g' | sed 's/\(.*\) user::\(.*\)/user::\2 \1/ you get a line for each acl looking like this : user::rwx,user:joe:rwx,group::rwx,group:users:rwx,mask::rwx,other::r-x myfile Processing this command on +250 000 files took less than 2 minutes on a fresh rebooted (no cache involved) xfs file system (linux 2.4.18). Running 1 hour later, it will generally run in less than 5 seconds. You just have to pipe in a file in the involved directory on the master before running rsync and apply acls on the other side after rsync completed. To apply, use this script : #!/bin/bash ### This script work only if you don't have and / in your filenames ### I hope for you... :-) cat $1 | while read acl file do myfile=$file echo Applying acl in $file... /usr/bin/setfacl -s $acl $myfile done Running time depend of the number of acls in your directory. :-) I don't know if this method work well on other file systems, but it do on xfs and is really efficient. I hope this will help acls users, so please let me know if you get success with this method. I still want to create a getfacl patch for rsync, so that acls are transfered with a shorter syntax (u::rwx,u:101:rwx,g::rwx,g:100:rwx,m::rwx,o::r-x myfile), and so you don't have to script anymore :-) So please dear rsync developpers, could you please give me some advice about what to modifiy in rsync to add acls to standard access rights synchronisation. Thank you in advance. Olivier Tarnus Thanks for this research. As one using rsync for backups it is useful info. I'm sure the regular rsync developers will have their own thoughts on this but i believe that adding ACL support to rsync will require a protocol extension. For the transfer protocol I'm inclined to think that ACLs should probably be handled as a subset of Extended File Attributes (EFA) as has been discussed on lkml. EFAs are of interest on many platforms especially Macintosh where they correlate to the file resource fork. One of the main issues of dealing with ACLs is that we really want to handle them in a way that allows proper translation between implementations. This is an issue for backups where admins want to rsync Windows boxes (shudder) to Unix/Linux. Can i grant/deny access to a non-existent account or group? If not i would want to be able to apply the --numeric-ids option. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Strong encryption
On Wed, Jun 05, 2002 at 03:33:23AM -0700, 'jw schultz' wrote: On Wed, Jun 05, 2002 at 12:21:18PM +0200, C.Zimmermann wrote: If you want them stored on the destination encrypted you Yes, that?s it. The owner of the source files will be sure, that no one else can read his files on the destination host. I thought, rsync only looks at the modification date of a file and decides whether to backup this file or not. In this case, the backup could be stored encrypted. Rsync can handle encrypted files just fine. It just treats them as ordinary binary files. If the owner of the files encrypts them on the source they will be encrypted on the destination. As you have said rsync normally just looks at the modification date for deciding whether to update the destination (unless you use the -c option) But, unless the -w option is used rsync will use some rather clever (nod to rsync developers) methods to transfer only the changed the parts of changed files. It is this feature that gives rsync its speed. My comments below are to advise you that that clever feature is nullified by encrypted files. In fact for encrypted files rsync would be sub-optimal. If most/all of the changed files are encrypted i would use the -w option. Perhaps he wants a gpg-rsyncable patch to gpg? Seriously, such a thing would be possible. After the long thread in which I made a dick of myself discussing how gzip-rsyncable couldn't possibly work, only to have someone explain how it does, I now know how it could be done; encript the data in chunks, where the chunk boundaries are determined by the same heuristic gzip-rsyncable uses. I think I could even whip up a working Python wrapper around gpg in a day or so that could do the job... but I'm too buisy looking for paid work right now to do it so I'll leave it as an excersise for others :-) -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key -- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync as a backup solution
Be aware that with any live backup solution your backup directory may be corrupt in subtle ways. We've been running rsync for years now to back up Windows files stored on our Samba server, and we often find files on the backup destination that have the identical date and time stamp as the source but different content. I suspect that, since Unix only records file times to the second (and Windows only to every two seconds), a file was being changed while rsync was reading it. Rsync copied the data from the file and, in the same second, the user overwrote part of the file with new data. The result - two files that have the same size, date, and time but different data. Unless you call rsync with the --ignore-times option next time, this error will go undetected by subsequent rsync runs. -- -IAN! Ian! D. Allen Ottawa, Ontario, Canada [EMAIL PROTECTED] Home Page on the Ottawa FreeNet: http://www.ncf.ca/~aa610/ College professor at: http://www.algonquincollege.com/~alleni/ Board Member, TeleCommunities CANADA http://www.tc.ca/ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
