Storage compression patch for Rsync (unfinished)
Hi, i am using Rsync for making backups of a MySQL database. The MySQL files can be compressed about 1:10 and i want to make use of this fact. Rsync currently doesn't support saving files in a compressed state. I personally think this should be a feature for the filesystem (in the sense of synchronised files) but currently there is no such filesystem for Linux available. Here my idea: We will have two new options: -X : this will specify a compress programm (e.g. gzip, bzip...) - the default compressor is gzip -Z : this will activate storage file compression. If -Z is enabled. every name (files, directories, links, ...) get's an extension called .rsc. If we have a true file, there is a header section and a data section. The header section will store the followin attributes: - magic number - unpacked size - packed size - compress programm (e.g. gzip, bzip2, ...) - magic number After the header section we will have the compressed file using the programm the user gave us with -X Every action in rsync will work - we will some exceptions: 1) Every file objects has the extension .rsc. 2) Doing simple checks (size, etc.) on files. the filesize needs evaluation for the .rsc header. 3) The local file needs to be decompressed when it is accessed for reading. 4) The local file needs to be compressed after it was modified or created. A header section needs to be added. 5) The file stats (atime/ctime/mtime) will be applied to the .rsc file. In normal way. Problems/ideas: 1) On Unix this will allow us only files with names 255 - strlen(.rsc) ... but this might be a very very rare case we will disable compression for this single file. 2) Rsync will need a new option for decompressing and stating the .rsc file tree. (single file, recursive) We should also offer options for validating .rsc files and converting a tree to a .rsc filetree. I am sending some compressor patches. I am very new to the rsync source, so here a list of what i did: options.c - added -X and -Z options (-Z is passed thru a server wenn using [EMAIL PROTECTED]:/directory) flist.c: extension .rsc is added to every file/directory (in -Z mode) rsync.c: finish_transfer() now does the compression when in -Z mode before stating the file. That means the compressed file has the same stat as the uncompressed file. receiver.c: I added two new functions: - storage_decompress: this will decompress an .rsc file to a tmp file, e.g. for calculating sums (note: a delete function is missing!) - storage_decompress_update_stats: this will update a given stat structure with the decompressed filesize of the rsc file. Currently transfering new files and compressing works. But the receiver doesn't make use of the stats that storage_decompress_update_stats. I don't know if i am calling it at the right place. I also don't know if the sum is allways calculated for a file. If this is the case we need to store the md4 sum in the .rsc header. -- Bye, Harald Email: [EMAIL PROTECTED] ICQ: #15582696 A cool os: www.linux.org PGP Finger-print: C2 8F 7B 55 7B 9B 8C 7E 48 35 48 21 8A DF 01 66 rsync-compress-2.5.5.patch.gz Description: GNU Zip compressed data
Re: Storage compression patch for Rsync (unfinished)
On Wed, Jan 15, 2003 at 11:50:27AM +0100, Harald Fielker wrote: Hi, i am using Rsync for making backups of a MySQL database. The MySQL files can be compressed about 1:10 and i want to make use of this fact. Rsync currently doesn't support saving files in a compressed state. I personally think this should be a feature for the filesystem (in the sense of synchronised files) but currently there is no such filesystem for Linux available. e2compr is not dead. See http://www.alizt.com/ Here my idea: We will have two new options: -X : this will specify a compress programm (e.g. gzip, bzip...) - the default compressor is gzip -Z : this will activate storage file compression. Why two options? Just specify the compressor and that enables compression. If -Z is enabled. every name (files, directories, links, ...) get's an extension called .rsc. And .rsc stands for what, rsync? Even windows has overcome the three letter extension limit. If we have a true file, there is a header section and a data section. The header section will store the followin attributes: - magic number - unpacked size - packed size - compress programm (e.g. gzip, bzip2, ...) - magic number So you add yet another compressed file format. There's something the world is crying out for. After the header section we will have the compressed file using the programm the user gave us with -X Every action in rsync will work - we will some exceptions: 1) Every file objects has the extension .rsc. 2) Doing simple checks (size, etc.) on files. the filesize needs evaluation for the .rsc header. 3) The local file needs to be decompressed when it is accessed for reading. 4) The local file needs to be compressed after it was modified or created. A header section needs to be added. 5) The file stats (atime/ctime/mtime) will be applied to the .rsc file. In normal way. Problems/ideas: 1) On Unix this will allow us only files with names 255 - strlen(.rsc) ... but this might be a very very rare case we will disable compression for this single file. Rsync already has issues with tempfile names. This is shorter. 2) Rsync will need a new option for decompressing and stating the .rsc file tree. (single file, recursive) We should also offer options for validating .rsc files and converting a tree to a .rsc filetree. I am sending some compressor patches. I am very new to the rsync source, so here a list of what i did: options.c - added -X and -Z options (-Z is passed thru a server wenn using [EMAIL PROTECTED]:/directory) flist.c: extension .rsc is added to every file/directory (in -Z mode) rsync.c: finish_transfer() now does the compression when in -Z mode before stating the file. That means the compressed file has the same stat as the uncompressed file. receiver.c: I added two new functions: - storage_decompress: this will decompress an .rsc file to a tmp file, e.g. for calculating sums (note: a delete function is missing!) - storage_decompress_update_stats: this will update a given stat structure with the decompressed filesize of the rsc file. Currently transfering new files and compressing works. But the receiver doesn't make use of the stats that storage_decompress_update_stats. I don't know if i am calling it at the right place. I also don't know if the sum is allways calculated for a file. If this is the case we need to store the md4 sum in the .rsc header. While the idea of rsyncing with compression is mildly attractive i can't say i care for the new compression format. It would be better just to use the standard gzip or other format. If you are going to create a new file type you could at least discuss storing the blocksums in it so that the receiver wouldn't have to generate them. Finally, i didn't even look at your patch because it was not text/plain. Unless absolutly necessary patches should be either inline or text/plain attachments. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync hanging with openssh-2.9.9p2 as the transport
On Tue, Jan 14, 2003 at 05:29:06PM -0500, Andrew J. Schorr wrote: Hi, This is perhaps a stupid question. I apologize in advance if it's already been covered, but I'm stumped... I'm using rsync to backup some file systems to a remote host. The transport is openssh-2.9.9p2. The problem does not occur when the transport is rsh. I'm invoking rsync as follows: rsync -RlHptgoD --numeric-ids -e ssh -rzx --stats . remotehost:/nfs/mirror But it hangs when there are large numbers of files in the . filesystem. If I run with -vvv, I see lots of make_file entries as send_file_list starts sending the list of files to the remote host, but then it freezes after around 4500 to 6500 make_file messages. I see the same problem with versions 2.4.6, 2.5.5, and 2.5.6pre1. And I see it on linux 2.4.18 (red hat 8.0) and solaris 8/x86. I can get it to work by breaking it up into chunks of 1000 files or so. It might work with more, I haven't tested exhaustively. I have tried using the --blocking-io and --no-blocking-io options, but neither one solves the problem. I could provide more info about where it hangs, but I thought somebody might know the answer, since this is clearly related to the interaction with openssh (since there's no problem with rsh). Is there a trick to make rsync work nicely with openssh? I searched the archives and haven't found anything... Does upgrading to openssh 3 solve the problem? Unfortunately I don't think anybody is going to be able to tell you. I've not heard of anybody lately posting a similar problem. In the past hanging problems have been traced to many different sources. Rsync stresses network (and filesystem) implementations greatly, and combining it with ssh stresses things that much more. I think it's worth a try to use openssh 3.1 or 3.2 (I've not been happy with versions after 3.2). What's the network between the local and remote machines? Does the name /nfs/mirror imply that the files are not directly mounted on remote but are instead on an NFS server? That has been the cause of many problems in the past. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: specifying a list of files to transfer
On Tue, Jan 14, 2003 at 10:01:47PM -0600, Lee Eakin wrote: Yes, people do restrict args via ssh key restrictions. OK, I thank you both for enlightening me on the subject. My current patch applies the sanitize_path() function to all names read via the --files-from option, regardless of whether we're pushing or pulling. This means that all leading slashes are dropped from file names as well as all leading ../ prefixes, and that any infix dir/../ combos are removed. This ensures that we can't get above the root dir that was specified on the command-line. so any sanitize code could first make sure all pathnames begin with a valid module and then make sure the file or dir is really inside that module. This isn't needed since the module name is specified on the command-line and then all paths are relative to the directory that was specified in that module. For instance: rsync --files-from=foo remote::module/bar forces all pathnames read to be relative to the bar dir of the module. If no /bar path was specified, the paths would all be relative to the root-dir of the module. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Rsync over SSH v2 with strong authentication but not encrypted to get the highest speed on Rsync?
Also, if any of the clients are windows clients, there is a known problem that the small packet sizes used by the common rsync versions cause some kind of massive performance problem that appears attributed to ssh or rsync. Buffering patches dramatically speed up rsync. Jim Aaron Morris wrote: I do not believe there is a way to turn encryption off, however, you might be able to use blowfish (or DES, if possible) as a cipher instead of 3DES which is much slower (and the default for SSH2 in OpenSSH). Also, make sure you are not using compression in rsync or ssh, since it will increase CPU load as well. Boris Gegenheimer wrote: Hello I am trying to set up a backup server running Solaris 8 with rsync 2.5.5 and ipfilter the latest version. The problem i have is i have about 16 different interfaces that are secured via ipfilter , and i tried running rsync via rsh but ipfilter would not set up a keepstate with rsh which meant i had to open up and that is not acceptable. So what i tried then was via ssh and that worked fine but it was very slow and cpu heavy due to the encryption . What i am asking is there some way to configure ssh and rsync to use strong authentication but drop the encryption so that it works faster? Regards Boris -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Latest --files-from patch
If anyone is trying out my --files-from patch, please grab a new version from here: http://www.clari.net/~wayne/rsync-files-from.patch I fixed a potential bug in the pathname-reading code when pulling files. This version also sanitizes all read pathnames, as discussed elsewhere. Remember that this patch is still young, so use it for testing only. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: specifying a list of files to transfer
---begin quoted text--- From: Wayne Davison [EMAIL PROTECTED] Subject: Re: specifying a list of files to transfer Date: Wed, 15 Jan 2003 10:10:29 -0800 On Tue, Jan 14, 2003 at 10:01:47PM -0600, Lee Eakin wrote: Yes, people do restrict args via ssh key restrictions. OK, I thank you both for enlightening me on the subject. My current patch applies the sanitize_path() function to all names read via the --files-from option, regardless of whether we're pushing or pulling. This means that all leading slashes are dropped from file names as well as all leading ../ prefixes, and that any infix dir/../ combos are removed. This ensures that we can't get above the root dir that was specified on the command-line. That's awsome. Now as long as I want to allow access to the given portion of the file tree I can allow files-from. Now if I can only figure out a way to intercept the list when I need to be real picky about which individual files are accessed ... so any sanitize code could first make sure all pathnames begin with a valid module and then make sure the file or dir is really inside that module. This isn't needed since the module name is specified on the command-line and then all paths are relative to the directory that was specified in that module. For instance: rsync --files-from=foo remote::module/bar forces all pathnames read to be relative to the bar dir of the module. If no /bar path was specified, the paths would all be relative to the root-dir of the module. That's cool too, so no additional/special code to handle server-mode ;) I like this a lot, now to test ... ---end quoted text--- -Lee -- Lee Eakin - [EMAIL PROTECTED] Benchley's Law of Distinction: There are two kinds of people in the world, those who believe there are two kinds of people in the world and those who don't. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Win2k, rsync, and ssh on cygwin
If so, then the behavior is normal for rsync. What happens is that rsync sends a lot of small packets containing checksums, which is very expensive on cygwin/windows. If you want, I can send you patched sources (from Craig Barratt originally) [...] I got a good 10X throughput boost with Craig's patch, both on exchanging CRCs to figure out which bits are different and then on getting the changes through. AFAIK, the sooner it's checked and then checked in, the better. Thanks to Greger Cronquist, who kindly sent me patch. (I'm still excited. There's no longer enough time to make coffee during my mail and code backups. I barely have time to switch windows. W00t!) Regards, Garth. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: specifying a list of files to transfer
On Wed, Jan 15, 2003 at 02:48:05PM -0600, Lee Eakin wrote: Now if I can only figure out a way to intercept the list when I need to be real picky about which individual files are accessed ... This should be possible with a filter process. Here's how the new, slightly tweaked protocol works: 1. The normal startup exchange occurs up to the point just before where the (normal) file info (names + attributes) starts to flow from the sender to the receiver. 2a At this point IFF the sender is the remote process (i.e. we're pulling files), the receiver begins to send file names (separated by either newlines or nulls, as indicated by the --null option) over the socket (normally there is no data being sent to the sender during this stage). The end of the list is marked by an empty entry. (Note that the receiver begins receiving file info from the sender during this stage, so it must do both things at once without blocking.) If the recursive flag is set, the receiver may get more names back than it sends out. 2b Alternately, if the sender is the local process, the normal file info transfer happens (without anything new occurring over the socket). 3. The rest of the transfer proceeds as normal. So, if a filter understood the protocol enough to be able to pass through all the initial rsync data, it could actually look at all the names that go over the wire and allow/disallow/tweak them however it desired. (It's sad that this filter would then have to continue to relay all the data over the socket after its work was done, but that's the price you pay.) You'd just have to look for the --null option on the command-line to know if you're looking for a newline or a null EOL character, and stop scanning at the first empty name. Alternately, you could just disallow the --files-from option and not worry about authorizing the data. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync hanging with openssh-2.9.9p2 as the transport
On Wed, Jan 15, 2003 at 09:27:37AM -0600, Dave Dykstra wrote: Unfortunately I don't think anybody is going to be able to tell you. I've not heard of anybody lately posting a similar problem. In the past hanging problems have been traced to many different sources. Rsync stresses network (and filesystem) implementations greatly, and combining it with ssh stresses things that much more. I think it's worth a try to use openssh 3.1 or 3.2 (I've not been happy with versions after 3.2). What's the network between the local and remote machines? Does the name /nfs/mirror imply that the files are not directly mounted on remote but are instead on an NFS server? That has been the cause of many problems in the past. Actually, it turns out that my testing on this issue has been invalid since I've been using the -vvv flag to rsync to get a grip on where it's hanging. But I just discovered that the -vvv flag itself seems to cause rsync to hang. So all the testing I've done so far is useless, so I'm not sure whether this is related to openssh or the slow connection or what. I know that the problem exists in 2.5.5, but I need to do more testing without the -vvv flag to figure out what's causing it. Is it well-known that the -vvv flag can cause rsync to hang? Thanks, Andy -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync hanging with openssh-2.9.9p2 as the transport
On Wed, Jan 15, 2003 at 05:12:21PM -0500, Andrew J. Schorr wrote: On Wed, Jan 15, 2003 at 09:27:37AM -0600, Dave Dykstra wrote: Unfortunately I don't think anybody is going to be able to tell you. I've not heard of anybody lately posting a similar problem. In the past hanging problems have been traced to many different sources. Rsync stresses network (and filesystem) implementations greatly, and combining it with ssh stresses things that much more. I think it's worth a try to use openssh 3.1 or 3.2 (I've not been happy with versions after 3.2). What's the network between the local and remote machines? Does the name /nfs/mirror imply that the files are not directly mounted on remote but are instead on an NFS server? That has been the cause of many problems in the past. Actually, it turns out that my testing on this issue has been invalid since I've been using the -vvv flag to rsync to get a grip on where it's hanging. But I just discovered that the -vvv flag itself seems to cause rsync to hang. So all the testing I've done so far is useless, so I'm not sure whether this is related to openssh or the slow connection or what. I know that the problem exists in 2.5.5, but I need to do more testing without the -vvv flag to figure out what's causing it. Is it well-known that the -vvv flag can cause rsync to hang? That has been shown to cause hangs in the past, but I thought it was resolved. Maybe it hasn't; people don't tend to care so much to solve problems like that once they discover the workaround of removing the verbosity. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: signing tarballs
[replied to list] There was a discussion about this on the Samba list a while ago http://lists.samba.org/pipermail/samba-technical/2002-November/040931.html Briefly We should create a team signing key, with an lifetime of about a year. It has to be relatively short to allow for turnover in the people who have access to the key. The signing key must only be stored on secure machines, certainly *not* on samba.org.(If it was on samba.org, somebody who compromised that machine could also generate new signatures and it would be pointless.) The key should be signed by team members and other relevant people; we should also sign each others' keys. The key should be on the keyservers and on the web site. Unless you've already done so I'll create the key and send the private half to you and the public half to the website, keyservers, and list. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
ÎÒºÜÇî×ã²ÊÔ¤²â×ÊÁÏ2003ÄêµÚÒ»ÆÚ£¡£¡£¡
ÎÒºÜÇî2003ÄêµÚÒ»ÆÚ×ã²ÊÔ¤²â×ÊÁÏ 1¡¢²©ÂåÄáÑÇVSACÃ×À¼ »ðÌì´óÓУº´ÓØÔÃû¿´£¬Ö÷¶Ó»ñʤµÄ¿ÉÄÜÐԽϴ󣬵«¸¸Ä¸³ÖÊÀ£¬µ£ÐÄÖ÷¶ÓÄÜȡʤ£¬Ó¦»ðÉúÊÀÍÁ£¬¶ÔÖ÷¶ÓÓÐÀû¡£2س±äµÃµ½ÀëΪ»ð£¬±ÈºÍØÔË«·½ÄÑ·Öʤ¸º£¬¹Ù¹í³ÖÊÀ¶ÔÖ÷¶Ó²»Àû£¬Ó¦Ë®¿ËÊÀ»ð¶Ô¿Í¶ÓÓÐÀû£¬µ«ÈÕ½¨ÊÜÊÀ»ðËùº¦£¬¶ÔÖ÷¶ÓÓÐÀû¡£ÈÕ½¨ÖúӦˮ£¬¶Ô¿Í¶ÓÓÐÀû¡£ÍƼö£º01£¨3£© 2¡¢²¼À×Î÷ÑÇVSÀÆë°Â µØÀ׸´£ºÄ¾¿ËÍÁ¶Ô¿Í¶ÓÓÐÀû£¬Ó¦ÍÁ¿ËÊÀË®¶Ô¿Í¶ÓÓÐÀû£¬2س±äµÃµØÔóÁÙ£ºÍÁÉú½ð£¬¶Ô¿Í¶ÓÓÐÀû£¬ÈÕ½¨¿ËÊÀľ£¬»¯È¥¹Ù¹íµÄΣº¦£¬µ«Ê±½¨ÖúÊÀľ£¬²¢²»ÄÜÍêÈ«»¯È¥¹Ù¹íµÄΣº¦¡£Í¬Ê±£¬Ó¦Ë®ÉúÊÀľ£¬¶ÔÖ÷¶ÓÓÐÀû¡£ÍƼö£º1£¨0£© 3¡¢¿ÆĪVSÑÇÌØÀ¼´ó Ë®À×ÍÍ£ºË®Éúľ¶Ô¿Í¶ÓÓÐÀû£¬µ«ÊÀľ¿ËÓ¦ÍÁ¶ÔÖ÷¶ÓÓÐÀû¡£6س±äµÃ·çÀ×Ò棺±ÈºÍ·â£¬Ë«·½ÄÑ·Öʤ¸º£¬µ«Ó¦Ä¾¿ËÊÀÍÁ£¬¶Ô¿Í¶ÓÓÐÀû£¬Í¬Ê±ÈÕ½¨¿ËӦľ£¬¶Ô¿Í¶Ó²»Àû£¬µ«Ó¦Ä¾¿ËÊÀÍÁ£¬¶Ô¿Í¶ÓÓÐÀû¡£ÍƼö£º31£¨0µÄ¿ÉÄÜÐÔ½ÏС£© 4¡¢¹ú¼ÊÃ×À¼VSĦµÂÄÉ »ðµØ½ú£º»ðÉúÍÁ¶Ô¿Í¶ÓÓÐÀû£¬ÐֵܳÖÊÀ£¬Ö÷¶ÓºÄʧ¡£3س±äµÃ»ðɽÂ㺴ÓØÔÃû¿´£¬Ö÷¶ÓÓÌÈçÔÚÂÃ;£¬Æ®¸¡²»¶¨£¬¸¸Ä¸³ÖÊÀ¶øÉúÓ¦½ð£¬¶Ô¿Í¶ÓÓÐÀû£¬µ«ÈÕ½¨ÖúÊÀÍÁ¶øÍú£¬¶ÔÖ÷¶ÓÓÐÀû¡£ÍƼö£º3£¨1µÄ¿ÉÄÜÐÔ½ÏС£© 5¡¢ÓÈÎÄͼ˹VSÀ×¼ªÄÉ Ôó·ç´ó¹ý£º½ð¿Ëľ¶ÔÖ÷¶ÓÓÐÀû£¬µ«´ÓØÔÃû¿´£¬Ö÷¶ÓÄÑÒÔȡʤ¡£3س±äµÃÔóË®À§£º½ðÉúË®£¬¶Ô¿Í¶ÓÓÐÀû£¬´ÓØÔÃû¿´£¬Ö÷¶ÓȡʤÀ§ÄÑ¡£µ«Ó¦Ë®ÉúÊÀľ£¬¶ÔÖ÷¶ÓÓÐÀû£¬Ê±½¨ÉúÊÀľ£¬¶ÔÖ÷¶ÓÓÐÀû¡£ÍƼö£º1£¨3£© 6¡¢Æ¤ÑÇÇÙ²ìVSÅÁ¶ûÂê µØ·çÉý£ºÄ¾¿ËÍÁ£¬¶Ô¿Í¶ÓÓÐÀû£¬Æ޲ƳÖÊÀ£¬¶ÔÖ÷¶ÓÓÐÀû¡£5س±äµÃË®·ç¾®£ºË®Éúľ¶Ô¿Í¶ÓÓÐÀû£¬²Æ³ÖÊÀ£¬¶ø¿ËӦˮ£¬¶ÔÖ÷¶ÓÓÐÀû£¬µ«ÈÕ½¨Ê±½¨ÖúӦˮ¶øÍú£¬¶Ô¿Í¶ÓÓÐÀû¡£ÍƼö£º03 7¡¢ÂÞÂíVSÇÐÎÖ ·çÀ×Ò棺±ÈºÍØÔ£¬µ«²Æ³ÖÊÀ£¬¶ÔÖ÷¶ÓÓÐÀû£¬5س±äµÃɽÀ×ÒãºÄ¾¿ËÍÁ£¬¶Ô¿Í¶ÓÓÐÀû£¬µ«²Æ³ÖÊÀ¶ÔÖ÷¶ÓÓÐÀû£¬ÊÀÍÁ¿ËӦˮ£¬¶ÔÖ÷¶ÓÓÐÀû¡£µ«ÈÕ½¨ÖúӦˮ¶øÍú£¬Ë«·½ÊƾùÁ¦µÐ¡£ÍƼö£º13 8¡¢ÎÚµÏÄÚ˹VSÅå³¼Ö Ë®À×ÍÍ£ºË®Éúľ£¬¶Ô¿Í¶ÓÓÐÀû£¬ÊÀľ¿ËÓ¦ÍÁ£¬¶ÔÖ÷¶ÓÓÐÀû¡£2س±äµÃË®Ôó½Ú£º½ðÉúË®£¬¶ÔÖ÷¶ÓÓÐÀû£¬ÊÀ»ð¿ËÓ¦½ð£¬¶ÔÖ÷¶ÓÓÐÀû£¬µ«ÈÕ½¨ÖúÓ¦½ð¶øÍú¡£Í¬Ê±Ê±½¨º¦ÊÀ»ð£¬¶ÔÖ÷¶Ó²»Àû¡£ÍƼö£º13 9¡¢ÀÊ˹VSÉ«µ± Ë®»ð¼È¼Ã£ºË®¿Ë»ð¶ÔÖ÷¶ÓÓÐÀû£¬µ«ÐÖ³ÖÊÀ£¬¶ÔÖ÷¶Ó²»Àû£¬Ö÷¶Ó·Àʧ¡£5س±äµÃµØ»ðÃ÷ÒÄ£º´ÓØÔÃû¿´£¬Ö÷¶ÓȡʤÀ§ÄÑ¡£¹Ù¹í³ÖÊÀ£¬¶ÔÖ÷¶Ó²»Àû£¬µ«ÈÕ½¨º¦Ó¦Ä¾£¬¶ÔÖ÷¶ÓÓÐÀû¡£Ó¦Ä¾¿ËÊÀÍÁ£¬¶Ô¿Í¶ÓÓÐÀû¡£Ê±½¨ÖúӦľ¶øÍú¡£ÍƼö£º13£¨0¿ÉÄÜÐÔ½ÏС£© 10¡¢ÀÕ°¢¸¥¶ûVS˹ÌØÀ˹±¤ »ðɽÂ㺻ðÉúÍÁ£¬¶Ô¿Í¶ÓÓÐÀû£¬×ÓËï³ÖÊÀ£¬¿É±£Æ½°²£¬2س±äµÃ»ð·ç¶¦£º±íÊ¢´óµÄÇìºØ£¬¹Ù¹í³ÖÊÀ£¬¶ÔÖ÷¶Ó²»Àû£¬Ó¦ÍÁ¿ËÊÀË®£¬¶Ô¿Í¶ÓÓÐÀû£¬µ«ÈÕ½¨ÖúÊÀË®¶øÍú£¬µ«Ô½¨ÖúÓ¦ÍÁ¶øÍú¡£ÍƼö£º31£¨0¿ÉÄÜÐÔ½ÏС£© 11¡¢Ä¦ÄɸçVS°Í˹µÙÑÇ »ðɽÂ㺻ðÉúÍÁ£¬¶Ô¿Í¶ÓÓÐÀû£¬×ÓËï³ÖÊÀ£¬¿É±£Æ½°²£¬2س±äµÃ»ð·ç¶¦£º±íÊ¢´óµÄÇìºØ£¬¹Ù¹í³ÖÊÀ£¬¶ÔÖ÷¶Ó²»Àû£¬Ó¦ÍÁ¿ËÊÀË®£¬¶Ô¿Í¶ÓÓÐÀû£¬µ«ÈÕ½¨ÖúÊÀË®¶øÍú£¬µ«Ô½¨ÖúÓ¦ÍÁ¶øÍú¡£ÍƼö£º31£¨0¿ÉÄÜÐÔ½ÏС£© 12¡¢Äá˹VSÀï¶û Ìì·ç¥:½ð¿Ëľ£¬¶ÔÖ÷¶ÓÓÐÀû¡£2س±äµÃÌìɽ¶Ý£ºÍÁÉú½ð£¬¶ÔÖ÷¶ÓÓÐÀû¡£ÊÀ»ð¿ËÓ¦½ð£¬¶ÔÖ÷¶ÓÓÐÀû£¬µ«¹Ù¹í³ÖÊÀ¶ÔÖ÷¶Ó²»Àû¡£×ÓËï·¢¶¯¶ø˳¿Ë£¬¶ÔÖ÷¶ÓÓÐÀû¡£ÍƼö£º3£¨1£© 13¡¢²¨¶û¶àVSÃɱËÀû°£ ·çÌìСÐ󣺽ð¿Ëľ£¬¶Ô¿Í¶ÓÓÐÀû£¬Ó¦ÍÁ¿ËÊÀË®£¬¶Ô¿Í¶ÓÓÐÀû¡£4س±äµÃǬΪÌ죺±ÈºÍØÔ£¬Ë«·½ÄÑ·Öʤ¸º£¬µ«²Æ³ÖÊÀ£¬¶ÔÖ÷¶ÓÓÐÀû¡£ÍƼö£º3£¨1£© ±¾ÆÚÇ¿¶Ó¶àÊýÔÚÖ÷³¡£¬±¨ÀäµÄ¿ÉÄÜÐÔ²»ÊǺܴ󣬵«ÄѶÈÈ´²»Ð¡¡£ÒÔÉϸ´Ê½Ì«¶à£¬ÏÖËõΪÒÔϸ´Ê½£º03£¬10£¬310£¬3£¬3£¨1£©£¬03£¬1£¨3£©£¬13£¬310£¬30£¬31£¬3£¨1£©£¬3£¨1£©¡£´ó¼Ò¿É¸ù¾ÝÉÏÃæ×ÊÁϵĸÅÊö£¬´ÓÖÐÑ¡ÔñÊôÓÚ×Ô¼ºµÄÄÇÒ»×¢500Íò¡£±¾ÆÚ±¾ÈË×¼±¸Í¶£º0£¬1£¬31£¬3£¬31£¬03£¬31£¬31£¬13£¬30£¬13£¬13£¬3Öеļ¸×¢µ½50×¢²»µÈ¡£±¾ÈËûÓÐÔËÆø£¬Ï£Íû´ó¼Ò×Ô¼º´Ó×ÊÁÏÖÐÕÒ³öÊôÓÚÄãµÄÄÄÒ»×¢£¬²»Ò»¶¨¸ú×ÅÎÒͶ¡£±ÈÈçµÚÒ»³¡£¬ÓÐÐí¶àÈËÈÏΪÊÇ1£¬Æäʵ03µÄ¿ÉÄÜÐÔ×î´ó¡£µÚ¶þ³¡£¬ÀÆë°ÂÁ¬Ðø¶à³¡²»°Ü£¬Ò²ÐíÓÐÈËÈÏΪËü¸ÃÖÕ½áÁË¡£µ«±¾ÈËÈÏΪ1µÄ¿ÉÄÜÐÔ×î´ó¡£ÂÞÂíºÜÈÝÒ×±»ÇÐÎÖ±Æƽ¡£ÀÊ˹ºÍÉ«µ±£¬´ò³Éʲô½á¹û¶¼²»Îª¹ý¡£Äá˹VSÀï¶û1µÄ¿ÉÄܴܺó¡£Ô¤²âÊý¾ÝÖ»×÷²Î¿¼,Ͷע¾ö²ß»¹µÃ¿¿Äã×Ô¼º¶¨¶á¡£À¨ºÅÄÚµÄΪ³öÏֵĻú»á½ÏС£¬Ò»°ãС¶îͶע¿É²»Ó迼ÂÇ¡£±¾×ÊÁÏÿÆÚ50Ôª¡£ÈçÐ趩ÔÄÇëÁªÏµQQ£º18419276 Email:[EMAIL PROTECTED]) ĵµ¤¿¨ºÅ£º4580 6865 8224 7749 ¿ª»§ÐУº¹¤ÉÌÒøÐÐÖØÇìÊзÖÐÐÓªÒµ²¿ ÐÕÃû£ºÐìÔ¾¸£¡£ ÀÏ¿Í»§¿É7ÕÛÓÅ»Ý5ÆÚÒÔÉÏ£¬10ÆÚÒÔÉÏ65ÕÛ£¬ÓàϵÄÈ«¶©¿É6ÕÛÓŻݡ£Òѵ½ÆÚµÄÓû§£¬±¾ÆÚΪ×îºóÒ»ÆÚÊÔÓá£×îºó×£´ó¼ÒÐÂÄêµÚÒ»ÆÚÖÐ´ó½± -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: specifying a list of files to transfer
I like it (except for the work). I don't think I'm ready to code up a filter yet, but it is good to know the general theory in case I need to use it. Of course, if I keep thinking about it I'll probably run across a situation where it's use would greatly speed/simplify my life, so maybe I should go ahead and throw something together ;). Yes, the filter would need to sit in the middle for the duration of the xfer, but I am thinking of those situations where the tree scan is very costly so the little overhead and complexity of the filter would be worth it. -Lee ---begin quoted text--- From: Wayne Davison [EMAIL PROTECTED] Date: Wed, 15 Jan 2003 13:34:08 -0800 On Wed, Jan 15, 2003 at 02:48:05PM -0600, Lee Eakin wrote: Now if I can only figure out a way to intercept the list when I need to be real picky about which individual files are accessed ... This should be possible with a filter process. Here's how the new, slightly tweaked protocol works: 1. The normal startup exchange occurs up to the point just before where the (normal) file info (names + attributes) starts to flow from the sender to the receiver. 2a At this point IFF the sender is the remote process (i.e. we're pulling files), the receiver begins to send file names (separated by either newlines or nulls, as indicated by the --null option) over the socket (normally there is no data being sent to the sender during this stage). The end of the list is marked by an empty entry. (Note that the receiver begins receiving file info from the sender during this stage, so it must do both things at once without blocking.) If the recursive flag is set, the receiver may get more names back than it sends out. 2b Alternately, if the sender is the local process, the normal file info transfer happens (without anything new occurring over the socket). 3. The rest of the transfer proceeds as normal. So, if a filter understood the protocol enough to be able to pass through all the initial rsync data, it could actually look at all the names that go over the wire and allow/disallow/tweak them however it desired. (It's sad that this filter would then have to continue to relay all the data over the socket after its work was done, but that's the price you pay.) You'd just have to look for the --null option on the command-line to know if you're looking for a newline or a null EOL character, and stop scanning at the first empty name. Alternately, you could just disallow the --files-from option and not worry about authorizing the data. ---end quoted text--- -- Lee Eakin - [EMAIL PROTECTED] Lynch's Law: When the going gets tough, everybody leaves. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
rsync feature request: conditional compression
rsync is optimized to conserve network resources, sometimes at the expense of CPU resources. As long as CPU's are fast and networks are slow, rsync with compression does a great job. However, if I run rsync on a slow box (166MHz Ultra SPARC) over a fast network (100Mb/s), the compression kills the transfer rate. I turned compression off and I should be done rsyncing my files (60GB) overnight. I would like to request a feature I call conditional compression. What this would do would check the network buffer, and if it is mostly full, then rsync knows that the network is slow and it will compress data. If the network buffer is mostly empty, then rsync knows that the network is fast and doesn't waste time compressing data. I use the terms mostly full and mostly empty loosely. It could mean 85% full and 15% full; without testing, I don't know which values would work best. This algorithm would be self-adjusting to both network and CPU load. I haven't thought through the ramifications of the impact on the remote rsync process, but I suspect they will have some impact on the overall performance. Since I am writing to this list, I wanted to take a few words to thank you guys for a great piece of software. I have been using it extensively for the past year, and it solves a good number of problems I have. I have been using it for the past 6 months for making rotating snapshot backups of my home Linux server. In case you are interested in the implementation, click here: http://www.mikerubel.org/computers/rsync_snapshots/contributed/sean_herdejurgen Since I am not a member of this mailing list, please contact me directly via e-mail. Regards, Seann Herdejurgen [EMAIL PROTECTED] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Please test rsync-2.5.6pre1
On Tue, Jan 14, 2003 at 09:25:34PM -0600, Albert Chin wrote: Latest version of popt is 1.7: ftp://ftp.rpm.org/pub/rpm/dist/rpm-4.1.x/popt-1.7.tar.gz I just applied this version to my local copy. On FreeBSD with gcc-3.2.1 it produces a bunch of warnings about unused variables and a number of these: In file included from ../rsync/options.c:22: ../rsync/popt/popt.h:444: warning: type qualifiers ignored on function return type This article, http://gcc.gnu.org/ml/gcc-patches/2000-10/msg00814.html does suggest that these warnings are harmless though. The patch is available at http://www.catnook.com/patches/rsync-popt-1.7.diff -- Jos Backus _/ _/_/_/ Sunnyvale, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ jos at catnook.com_/_/ _/_/_/ require 'std/disclaimer' -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Newbie question - does it work with Sun Cobalt RaQ550?
On Wed, Jan 15, 2003 at 10:54:04AM +0900, Majorosi.net - Stéphane wrote: Hi, I'd like to know if RSYNC work on a Sun Cobalt RaQ550. Does someone trid it? If yes, please let me know if I can take contact with you. Stephane Since no one else has responded i will. I don't have a RaQ myself. A quick look at www.sun.com tells me that that it is a linux box running a 2.4 kernel on a PIII cpu and has a ssh and telnet daemons. There is no reason that this box couldn't support rsync. I'd suggest getting a shell prompt and looking around. Rsync might already be installed. If it isn't you should be able to build an executable either on the machine or on another with similar libs and install it yourself. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync feature request: conditional compression
On Wed, Jan 15, 2003 at 06:47:09PM -0600, Seann Herdejurgen wrote: rsync is optimized to conserve network resources, sometimes at the expense of CPU resources. As long as CPU's are fast and networks are slow, rsync with compression does a great job. However, if I run rsync on a slow box (166MHz Ultra SPARC) over a fast network (100Mb/s), the compression kills the transfer rate. I turned compression off and I should be done rsyncing my files (60GB) overnight. If your network is faster than CPU and the disk subsystem you will find --whole-file speeds things up even more. The rsync algorithm is great but it does generate more than twice the disk I/O of --whole-file (subject to caching). -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Why is rsyncd trying to access /etc/pwd.db?
I have rsync-2.5.5 humming along on FreeBSD 4.7-STABLE with no apparent problems. However, I notice when some clients connect to sync, I get this in /var/log/messages rsyncd[42843]: /etc/pwd.db: No such file or directory rsync is running as root with this basic rsync.conf: use chroot = yes read only = yes timeout = 600 Well of course /etc/pwd.db exists and could be readable, but more importantly, why is it trying to be accessed at all? It's an anonymous server, so people trying to use auth. have no benefit. Is there some specific client rsync flag that triggers that message? How can I correct it, and what does it mean? Thank you, = Hans Zaunere President, New York PHP http://nyphp.org [EMAIL PROTECTED] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync feature request: conditional compression
Thanks for the tip. I was thinking about my suggestion, and I'm guessing that the --bwlimit feature is implemented as an if statement like: if (data sent/time since last check bwlimit) then sleep There probably isn't a network buffer that is being checked. Maybe this can be changed to: if (data sent/time since last check bwlimit) then compress data The problem then becomes, if bwlimit isn't specified, how could rsync determine what it should be? Is there any way you can think of that would allow rsync to detect when it is on a fast network? (as compared to CPU or disk subsystem) I am not familiar with the inner workings of rsync. I have a gut feel that if you were to implement conditional compression that it would require asynchronous I/O. Does rsync use AIO? If a target file does not exist (for example, on the initial run of rsync), does it do --whole-file by default? -Seann jw schultz wrote: On Wed, Jan 15, 2003 at 06:47:09PM -0600, Seann Herdejurgen wrote: rsync is optimized to conserve network resources, sometimes at the expense of CPU resources. As long as CPU's are fast and networks are slow, rsync with compression does a great job. However, if I run rsync on a slow box (166MHz Ultra SPARC) over a fast network (100Mb/s), the compression kills the transfer rate. I turned compression off and I should be done rsyncing my files (60GB) overnight. If your network is faster than CPU and the disk subsystem you will find --whole-file speeds things up even more. The rsync algorithm is great but it does generate more than twice the disk I/O of --whole-file (subject to caching). -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: .rsync-/.rsync+ patch and --link-dest example
This patch also adds an example for using --link-dest to the man page: I don't know if that is needed. Lacking context the example has minimal meaning and compare-dest doesn't have an example A more complete context is available here: http://www.math.ualberta.ca/imaging/rlbackup/ rlbackup is a simple yet secure backup mechanism for generating linked backups over the network, with historical pruning, thanks to rsync and your very useful --link-dest option. -- John Bowman Associate Professor Department of Mathematical and Statistical Sciences University of Alberta Edmonton, Alberta Canada T6G 2G1 Office: CAB 521 Phone: +1 780 492 0532 FAX: +1 780 492 6826 http://www.math.ualberta.ca/~bowman Email: [EMAIL PROTECTED] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Rsync over SSH v2 with strong authentication but not encrypted to get the highest speed on Rsync?
No it is one Sun solaris 8 box trying to back up several aix boxes.But apparantly it is not possible to disable the encryption so I was hoping that I could use another -e like ftp or something like that thru rsync. But I cannot seem to get to work. Regards Boris -Original Message- From: Jim Kleckner [mailto:[EMAIL PROTECTED]] Sent: 15. januar 2003 19:47 To: Boris Gegenheimer Cc: [EMAIL PROTECTED] Subject: Re: Rsync over SSH v2 with strong authentication but not encrypted to get the highest speed on Rsync? Also, if any of the clients are windows clients, there is a known problem that the small packet sizes used by the common rsync versions cause some kind of massive performance problem that appears attributed to ssh or rsync. Buffering patches dramatically speed up rsync. Jim Aaron Morris wrote: I do not believe there is a way to turn encryption off, however, you might be able to use blowfish (or DES, if possible) as a cipher instead of 3DES which is much slower (and the default for SSH2 in OpenSSH). Also, make sure you are not using compression in rsync or ssh, since it will increase CPU load as well. Boris Gegenheimer wrote: Hello I am trying to set up a backup server running Solaris 8 with rsync 2.5.5 and ipfilter the latest version. The problem i have is i have about 16 different interfaces that are secured via ipfilter , and i tried running rsync via rsh but ipfilter would not set up a keepstate with rsh which meant i had to open up and that is not acceptable. So what i tried then was via ssh and that worked fine but it was very slow and cpu heavy due to the encryption . What i am asking is there some way to configure ssh and rsync to use strong authentication but drop the encryption so that it works faster? Regards Boris -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Wed Jan 15 16:14:07 2003 Author: dwd Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv25177 Modified Files: flist.c Log Message: Fix bug in --copy-unsafe that made it totally broken, and re-enable the tests that tested it. As far as I can tell, it was always broken since the day I put it in years ago. In my investigation into this I was unable to figure out what in the world I was thinking back then, to introduce a global variable with the wrong information in it rather than using a parameter that was already available in readlink_stat function. That still bothers me a bit but I decided to stop worrying about it. Revisions: flist.c 1.127 = 1.128 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c?r1=1.127r2=1.128 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
