Re: File fragmentation
On 8/14/06, Matt McCutchen wrote: Only rsync knows how much data it plans to write, so rsync would have to make some sort of call to provide the eventual file size. I don't know anything about the Windows call for preallocating space or whether it is available through Cygwin. If I get a chance, I will try to find out. I don't think this is available via cygwin: http://cygwin.com/ml/cygwin/2006-08/msg00108.html -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature request: rsync of device content
On 6/11/05, Martin Scharrer wrote: It would be nice if rsync would also be able to synchronise the content of a device, i.e. handle the device like a regular file. Of course only if this is requested with a special command line option. The idea of a block-device-rsync comes up from time to time. April was the last time I saw it pop up and I gave a summary of what others had said: http://lists.samba.org/archive/rsync/2005-April/012196.html The consensus seems to be that this would be a great idea for a librsync-based standalone tool rather than rsync itself. However, I have still seen no signs that anyone (including myself) has had the time to build such a tool. -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Stupid --delete mistake
On 6/3/05, Ed wrote: I just made a rather stupid commandline ordering error and managed to delete a whole load of data (mostly word files/some pics fwiw) with rsync delete. I know, I'm an idiot (and a big one at that). To make matters worse this was off an ext3 partition, which from what I can find out seems pretty much undeleteable at least if deleted in the normal way by rm. They key to this lies in the pretty much, more below. ;-) So the question is, does rsync delete files in a way which will allow me to recover them, or am I, as I suspect, royally screwed :( Rsync deletes files in the same way as rm, so you are pretty screwed unless you want to spend some time learning a bit about filesystem forensics. The last time I recovered a file I'd nuked off an ext3 filesystem, I ran across The Sleuth Kit: http://www.sleuthkit.org/ It made the undelete process fairly painless. The Sleuth Kit requires a separate system with lots of disk space since it needs to hold the entire contents of the filesystem off which you deleted your files plus associated analysis, logs, etc. The first thing you need to do is dd a copy of your raw filesystem onto some other linux host with lots of disk space. The sooner you do this, the less chance there is of those disk blocks getting reallocated. For example, from the system that will be used as the dumping ground: ssh hosed-host dd if=/dev/hda3 obs=1500 | dd of=/home/sleuth/hosed-host-hda3.img ibs=1500 This does a dd to stdout of your hda3 partition (substitute the real device name) and sends it over the ssh connection to your receiving host which presumably has a lot of free space in /home/sleuth. The obs and ibs parameters are there to ensure that we can fill up a 1500 byte ethernet packet on each read. It works fine without them, just a bit slower. There are other ways to do this, one method using netcat is described on The Sleuth Kit's website: http://www.sleuthkit.org/informer/sleuthkit-informer-11.html#dd Once you have the image saved off you can spend some time learning how to run The Sleuth Kit. It goes to great pains to keep your dd image unmodified, however as a fail-safe I always keep these owned by root, mode 444, and run The Sleuth Kit as a non-root user. The Autopsy Forensic Browser is a webserver-based front end to The Sleuth Kit and is a good way to get started. RPMs can be found at: http://dag.wieers.com/packages/sleuthkit/ http://dag.wieers.com/packages/autopsy/ Good luck! This will be a valuable Learning Experience. ;-) -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On 4/12/05, Jeff Schoby wrote: I did what steve suggested here, on both machines. I didn't use any arguments for configure... just did ./configure Odd, I thought we had it nailed on that one. Try the following on each system using the new rsync you just built with the mkstemp use commented out: 1) Create a large file, preferably a sparse file 2) Try to rsync it locally For example: $ echo smallfile $ dd if=smallfile of=sparsefile_01 bs=1024 count=1 seek=2097153 $ rsync -avvvS sparsefile_01 sparsefile_02 If this works, then the problem may be in your network transport (e.g. ssh). -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On 4/12/05, Jeff Schoby wrote: I did what steve suggested here, on both machines. I didn't use any arguments for configure... just did ./configure Odd, I thought we had it nailed on that one. Try the following on each system using the new rsync you just built with the mkstemp use commented out: 1) Create a large file, preferably a sparse file 2) Try to rsync it locally For example: $ echo smallfile $ dd if=smallfile of=sparsefile_01 bs=1024 count=1 seek=2097153 $ rsync -avvvS sparsefile_01 sparsefile_02 If this works, then the problem may be in your network transport (e.g. ssh). -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: An idea: rsyncfs, an rsync-based real-time replicated filesystem
On 4/12/05, Lester Hightower wrote: The actual replication happens in user-land with rsync as the transport. I think rsync will have to be tweaked a little to make this work, but given all the features already in rsync I don't think this will be a big deal. I envision an rsync running on Host A like: # rsync --constant --from0 --files-from-fifo=/vol/0/chglog.fifo ... that will be communicating with an rsync --constant ... on the other end. The --constant flag is my way of stating that both rsyncs should become daemons and plan to constantly exchange syncing information until killed -- that is, this is a constant rsync, not just one run. Lester: Something like this is very high on my list of products I wish I had. I frequently use rsync to replicate data on a near real-time basis. My biggest pain point here is replicating filesystems with many (millions) of small files. The time rsync spends traversing these directories is immense. There have been discussions in the past of making an rsync that would replicate the contents of a raw device directly, saving the time spent checking each small file: http://lists.samba.org/archive/rsync/2002-August/003545.html http://lists.samba.org/archive/rsync/2003-October/007466.html It seems that the consensus from the list at those times is that rsync is not the best utility for this since it's designed to transfer many files rather than just one really big file (the contents of the device.) Despite the fact that the above discussions are almost 18 months ago, I have seen no sign of the rsync-a-device utility. If it exists, this might be the solution to what you propose-- and it would work on more than Linux. To achieve your goal with this proposed utility you would simply do something like this: + for each device ++ make a snapshot if your LVM supports it ++ transfer the diffs to the remote device + go back and do it all again If the appropriate permissions were in place this could be done entirely in user-mode, which is a great advantage for portability. As you touched on in your original message, knowing what's changed since the last run would be very helpful in reducing the amount of data that needs to be read on the source side. In my experience, sequential reads like this, even on large devices, don't take a huge amount of time compared with accessing large numbers of files. If there were only a few files on a mostly-empty volume the performance difference would be more substantial. ;-) Another thought to eliminate the kernel dependency is to combine the inode-walk done by the dump utility with the rsync algorithm to reduce the file data transferred. The inode walk would be filesystem-specific, but could be done in user space using existing interfaces. -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On Apr 12, 2005 10:04 AM, Jeff Schoby wrote: How do I build rsync for large file support..I'm going to need 4GB files thru rsync. It basically stops with an error message of File too large...I can re-run rsync later with the -vvv flag and post the results tomorrow. It sounds like your version of AIX has similar problems to HP-UX. You will need to re-compile rsync doing something like this: 1) run ./configure with whatever options you normally use 2) edit the config.h file and comment out the line #define HAVE_SECURE_MKSTEMP 1 3) continue normally (e.g. make, make test, etc.) Here's an older list article that describes the fix: http://lists.samba.org/archive/rsync/2004-July/010199.html -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.5, for HPUX11.11
On Apr 6, 2005 8:49 AM, Jose, Tony wrote: Any body got a copy of compiled rsync for hp-ux 11.11? The HP-UX porting center has 2.6.3: http://hpux.cs.utah.edu/hppd/hpux/Networking/Admin/rsync-2.6.3/ If you choose to run 2.5.5, don't use daemon mode as there are several security problems with this older version. (See http://rsync.samba.org/ for details.) -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync
On Apr 4, 2005 3:03 PM, Steve Bonds wrote: So how's this for a revised plan? 1) On HP-UX systems, build using mktemp() 2) Put a warning in the HP-UX section of the INSTALL file about this necessity Attached are the following patches: + configure.in patch + syscall.c patch (Note to lawyers: these are released without copyright, consider these patches public domain.) The configure.in patch puts in a check for hp as the vendor in the target system. This seemed the quickest way to detect all variants of HP-UX. Alternatively, the $target_os could be checked to see if it began with hpux. I also implemented the suggestion to directly check for lseek64 instead of inferring it from the size of off64_t. This was only a problem on the 64-bit rsync build, which ended up having a number of other problems, but it seemed like a good practice to directly check where possible instead of inferring. I decided the INSTALL file warning is probably unnecessary since I included one in the configure.in patch. The issues with the daemon test code went away when I moved back to a 32-bit binary. -- Steve configure.in.patch Description: Binary data syscall.c.patch Description: Binary data -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync
On Apr 1, 2005 5:30 PM, Wayne Davison wrote: I attempted a fix for this by having the code not use mkstemp() if open64() was around but mkstemp64() was not. However, not having any HP systems around to test this on, I never heard if this was helpful or not. Apparently not. Don't you wish there were a good PA-RISC emulator out there? HP-UX is so strange, I can't imagine writing software for HP-UX without a build/test system. I'd prefer to add rules to configure that don't require trying to write a large sparse file or to use the O_LARGEFILE kluge, but I'm open to suggestions on whether creating a 64-bit binary is better than just disabling HAVE_SECURE_MKSTEMP for HP-UX. I think the latter would probably be good enough, if we just want to go that route. What's the best way to detect HP-UX, anyway? If we can avoid using mktemp(), I'd like to since relying on file locking for security on a UNIX system is potentially dangerous. I think the chances of creating an exploitable security problem by using mktemp() in the same directory as the destination file are pretty slim, but it's possible. My main concern with a 64-bit binary would be compatibility if the binary were copied onto a system running 32-bit HPUX. These are becoming quite rare, so this is less of a concern. The comp.sys.hp.hpux FAQ number 6.2.7 has this to say on detecting HPUX: - Subject: 6.2.7 How can I detect the HP-UX version at compile time? The below macro sequence allows you to figure out the HP-UX major version number: #include sys/privgrp.h #if defined(PRIV_PSET) #define _hpux_11i #elif defined(PRIV_SPUCTL) #define __hpux_11x #elif defined(PRIV_SERIALIZE) #define __hpux_10x #elif defined(PRIV_SETRUGID) #define __hpux_9x #endif (Added by Ian, 05/02/02) - From either http://web.archive.org/web/20030324214118/faqs.org/faqs/hp/hpux-faq/ or http://www.faqs.org/faqs/hp/hpux-faq/. This was also useful, since it could be used to limit building a 64 bit binary only on 64-bit systems: - Subject: 5.5.1 How can I tell if I have a 32-bit or 64-bit kernel? First off, in all versions of HP-UX prior to 11.00, the kernel is always 32-bit. That being said, on 11.x systems, there are several ways to determine whether you're running a 32 or 64 bit kernel... From the command-line = $ getconf KERNEL_BITS ... trimmed ... (Thanks to Brian Hackley [EMAIL PROTECTED] and Ian) (Added by Ian, 04/19/01) - The above can also be accomplished via C code using sysconf(). Finally, I found a problem trying to build a 64-bit HPUX binary using the current configure script. syscall.c uses the results of AC_CHECK_SIZEOF(off64_t) to infer the existence of lseek64(). This fails on HP-UX since off64_t is defined, but lseek64() doesn't exist. I'd suggest a direct check for lseek64() instead of this inference, if possible. How's this for a plan? 1) On 64-bit HP-UX systems, build a 64-bit binary (above lseek64 fix will be needed) 2) On 32-bit HP-UX systems, build using mktemp() I've also noticed that make test fails every rsync daemon mode test on HP-UX. I'm still investigating but it looks like a either typecast in socket.c getsockname or a problem with htonl(INADDR_LOOPBACK) (of all things) leads to a failure to bind the socket when run in daemon mode. -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync
On Apr 4, 2005 10:33 AM, Steve Bonds wrote: How's this for a plan? 1) On 64-bit HP-UX systems, build a 64-bit binary (above lseek64 fix will be needed) 2) On 32-bit HP-UX systems, build using mktemp() After more pondering, I think this would be a bad idea. The only time a 64-bit binary is likely to work would be if using the optionally purchased HP C/ANSI C compiler. Using gcc on HP-UX seems to have lots of 64-bitness problems. So how's this for a revised plan? 1) On HP-UX systems, build using mktemp() 2) Put a warning in the HP-UX section of the INSTALL file about this necessity -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync
On Apr 4, 2005 10:33 AM, I wrote: I've also noticed that make test fails every rsync daemon mode test on HP-UX. I'm still investigating but it looks like a either typecast in socket.c getsockname or a problem with htonl(INADDR_LOOPBACK) (of all things) leads to a failure to bind the socket when run in daemon mode. I found the problems-- they were limited to the test code only so it doesn't look like rsync itself is the problem. However, I have not yet back-ported the changes I made to my simplified test case into rsync so there may still be something there that needs fixing. The problems were: 1) In socket.c there's the line sock2.sin_family = PF_INET; but no corresponding line for the other socket sock. The connect() blows up on HP-UX with an address family error since it's still set to 0. 2) In the same file, neither socket ever gets a TCP port assigned, so connect() fails on HP-UX. The same code works great on my Linux host, since Linux helpfully populates these missing values. Would you like a suggested patch or is the above good enough? -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
HP-UX 11i and largefiles on rsync
For all you folks out there using rsync on HP-UX, be warned that HP-UX is still broken with regard to large files. I opened a bug report with HP last year, as described here: http://lists.samba.org/archive/rsync/2004-July/010226.html I've been periodically checking on the status and today I was told that it's been officially filed as a we won't fix this, citing that the mkstemp() library call wasn't mentioned in the /usr/share/doc/lg_files.txt document. (For some reason HP favors using the insecure tmpfile() call instead.) Despite HP's intransigence, I'd like to have an rsync that builds out of the box with working large file support. I figure there are several ways to accomplish this, and I'd like some input on the best way to proceed. I've come up with five possible options, though if anyone has an idea not listed here, please feel free to suggest it! 1) Change the autoconf environment to build rsync as a 64-bit binary by default on HP-UX. This results in a working mkstemp() but gives a binary that won't work on ancient HP hardware. 2) Change the autoconf environment to detect HP-UX and unset HAVE_SECURE_MKSTEMP even though the test for it succeeded. This would fall back to mktemp() very much like the manual process I use now, but would be automatic. This also means that if HP-UX were ever fixed (HA!), the new mkstemp() wouldn't get used. 3) Write a new autoconf test for a broken mkstemp() by creating a large sparse file in the current directory and automatically unsetting HAVE_SECURE_MKSTEMP. This would work on non-HP systems that were similarly broken and allow the use of mkstemp() if HP-UX were ever fixed (HA!) This may falsely fail if the current filesystem doesn't allow for large files (common on HP systems) and if the system doesn't support sparse files this test could fill up the filesystem. 4) Implement HP's suggested workaround of a fcntl() to set O_LARGEFILE on the file handle returned by mkstemp() inside syscall.c. This is essentially non-portable and indeed won't even work on HP-UX if building a 64-bit binary. 5) Document the workarounds for HP-UX in the INSTALL file (e.g. comment out HAVE_SECURE_MKSTEMP in config.h after running configure but before running make, or build a 64-bit binary.) What do you all think the best approach would be? I'd suggest #5 at a minimum. -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Spam to this list
On Fri, 25 Mar 2005 20:42:19 +0100 (CET), Dag Wieers dag-at-wieers.com |Rsync List| ... wrote: I'm not sure what the policy of this list is and I bet everyone has a spam filter, so nobody might have noticed, but we got spammed. Can anyone send mail to the list or do you have to subscribe first ? Anyone can send mail, including phishers. I think we all got this one an hour ago or so: - We regret to inform you that your eBay account could be suspended if you don't re-update your account information. To resolve this problems please click here and re-enter your account information. If your problems could not be resolved your account will be suspended for a period of 3-4 days, after this period your account will be terminated. - This nicely done phish uses http://lenix.brinkster.net/ebay/redirect.php to collect your Ebay login information. The rsync list is one of my higher sources of spam, but I also understand the reasoning behind keeping the list open. The list managers have installed filters, but they can only do so much. For a complete history of spam discussions here, try this Google search: http://www.google.com/search?as_q=spamnum=10as_sitesearch=lists.samba.org -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: atomic transaction set option to rsync
On Mon, 3 Jan 2005 17:39:19 +0100 (CET), Dag Wieers dag-at-wieers.com wrote: Apparently a change of behaviour from rsync 2.5 to rsync 2.6 affected the way I worked. I provide RPM repositories that I mirror using rsync. It is important to have the repository meta-data in sync with the data otherwise people have errors using Yum or Apt. In the old days (with older rsyncs) I was able to influence the order in which my transaction set was processed by changing the order of directories I wanted rsync to mirror, so that the metadata was uploaded after most of the data and deletes were done at the end of the transaction. Dag: I love your RPM collection, it's unfortunate that a minor change in rsync behavior has created problems for you. The design of UNIX file operations makes it hard to create an atomic transaction the size of a normal rsync transfer. There are other ways to work around it, either by uploading in different steps (which is impractical in my scenario) or by using a staging area (which is impossible and impractical for large mirror sites). Rsync does a good job of ensuring file-level coherence by using a temporary file during the transfer and a quick rename to the original at the end. Unfortunately for you, this is only good for a single file. If this were done on a larger scale, it would serve as an atomic transaction-- but then rsync is just using a staging area of its own creation. The same thing could be accomplished by manually creating the staging area and only using rsync as the data transport. (Which is really what it's designed for.) I don't see how uploading in different steps would be impractical. The most bulletproof way to do this would be to sync each rpm and header file in one rsync session. However, for your collection of thousands of file pairs, this would indeed be impractical. Breaking it up into 10-20 sessions with several dozen file pairs each would be practical and could be automated with some shell or Perl wizardry. Another option you didn't mention would be to make use of LVM snapshots to ensure that your repository is always internally consistent even while you're in the middle of an rsync. The disadvantage would be some periodic unavailability while you removed and re-created your snapshots. (i.e. the FTP server is configured to serve files from the read-only snap volumes, which need to be unmounted and re-snapped when new files are uploaded.) There may be some full site-replication applications out there making use of rsync. I suspect someone here on the list would know. I've always just created my own custom scripts for this. Lastly, I suspect one of the rsync gurus here can probably comment on the feasibility of at least providing an option to restore the version 2.5 behavior. Thanks again for the RPMs. I hope you can find a good solution to your mirroring dilemma. -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Nessus 2.1.3 released
I just tried the Nessus 2.1.3 beta on my RedHat Enterprise WS 3.0 system, which has had plugin coredump problems with 2.0.10 and 2.0.12. It works great! I'm looking forward to the release version. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync to a destination 8TB problem...
On Tue, 10 Aug 2004, Matt Miller m.miller-at-duke.edu |Rsync List| wrote: I am trying to use a large (10TB) reiserfs filesystem as an rsync target. The filesystem is on top of lvm2 (pretty sure this doesn't matter, but just in case.) I get the following error when trying to sync a modest set of files to that 10TB target (just syncing /etc for now): rsync: writefd_unbuffered failed to write 4 bytes: phase unknown: Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(836) This is unfortunately a fairly generic error message that means the remote/child process died suddenly without passing back any information about why. Here's a message from Wayne on why the error message is so generic: http://lists.samba.org/archive/rsync/2004-July/010204.html I'd suggest following the same debugging steps I used to track down a 2GB bug on HP-UX: http://lists.samba.org/archive/rsync/2004-July/010197.html The best results were from running rsync under tusc (in your Linux case it would be strace) and sifting through the voluminous output to find the write() that failed and why. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2 (fwd)
On Mon, 2 Aug 2004, Wayne Davison wayned-at-samba.org |Rsync List| wrote: Would anyone who is seeing this problem please try out the patch that is attached to this bugzilla bug: https://bugzilla.samba.org/show_bug.cgi?id=1536 I gave it a test and it seems to work fine on HPUX 11.0. It skips the problematic mkstemp() call. It also passed the largefiles test that I sent to the list recently. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: mirroring in both directions with rsync
On Fri, 30 Jul 2004, Matt Jaffe mjaffe-at-scires.com |Rsync List| wrote: When operating in daemon mode, will rsync check to see which file is newer and decide which direction to transfer based on that? Example, We have 2 terminal servers, A and B. I want users to be able to work on either server and have their home directories on both stay in sync. If a user logs into server A and changes files, the server would know to transfer the file from A to B. If they log into server B, rsync transfers from B to A. rsync is designed to be a unidirectional transfer program, like rcp/scp, only with the addition of its clever bandwidth reduction algorithm. For a two-way sync, try Unison: http://www.cis.upenn.edu/~bcpierce/unison/ -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
On Tue, 27 Jul 2004, Wayne Davison wayned-at-samba.org |Rsync List| wrote: This would seem to indicate that the mkstemp() call is not compatible with large file systems, probably because it is not calling the right open() call in the library (on systems where large- file support redefines functions like open() into open64()). This is indeed the case, and HP has verified that this library is broken. It seems they didn't build it with -D_FILE_OFFSET_BITS=64 like they should have. The HP bug tracking number is JAGaf34426. Since it's a change to their libc package I suspect it will be a while before it emerges from their patch factory. Changes to libc get pretty thorough testing. ;-) -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
On Tue, 27 Jul 2004, Wayne Davison wayned-at-samba.org |Rsync List| wrote: This sparked a memory, and back in 2002 someone reported a similar problem for AIX. They worked around it by undefining the HAVE_SECURE_MKSTEMP define in config.h and rebuilding rsync. This would seem to indicate that the mkstemp() call is not compatible with large file systems, probably because it is not calling the right open() call in the library (on systems where large- file support redefines functions like open() into open64()). Bingo! Great memory you have there, Wayne. ;-) This did the trick on the CVS version, and I suspect it will work on older versions, too. Can you check to see if there is a mkstemp64() function available on HP-UX? Alas, no. The mkstemp man page suggests using tmpfile() instead, which generally means that HP won't fix any problems. - mktemp(3C) Remarks: These functions are provided solely for backward compatibility and importability of applications, and are not recommended for new applications where portability is important. For portable applications, use tmpfile() instead (see tmpfile(3S)). - The tempnam()/tmpfile() combination seems particularly difficult to use compared with mkstemp(). I especially liked this warning: - tmpnam(3S) WARNINGS Between the time a file name is created and the file is opened, it is possible for some other process to create a file with the same name. This can never happen if that other process is using these functions or mktemp, and the file names are chosen such that duplication by other means is unlikely. - At least HP documents their race conditions, eh? :( -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
On Thu, 29 Jul 2004, Chris Shoemaker c.shoemaker-at-cox.net |Rsync List| wrote: On Wed, Jul 28, 2004 at 10:57:58AM -0700, Steve Bonds wrote: The mkstemp man page suggests using tmpfile() instead, which generally means that HP won't fix any problems. - mktemp(3C) Remarks: These functions are provided solely for backward compatibility and importability of applications, and are not recommended for new applications where portability is important. For portable applications, use tmpfile() instead (see tmpfile(3S)). - Maybe we should follow this advice. This is actually terrible advice because of the race condition mentioned below. For example, from the Linux man page for tmpnam(3): - BUGS Never use this function. Use mkstemp(3) instead. - I tend to agree with that sentiment. ;-) The tempnam()/tmpfile() combination seems particularly difficult to use compared with mkstemp(). I especially liked this warning: - tmpnam(3S) WARNINGS Between the time a file name is created and the file is opened, it is possible for some other process to create a file with the same name. This can never happen if that other process is using these functions or mktemp, and the file names are chosen such that duplication by other means is unlikely. - Can we use tmpfile without tmpname? No, since rsync should create the temp file in the same directory as the final destination file. There is no way to feed a template to tmpfile to tell it where to create the file, so they all generally end up in /tmp unless the system defines a really odd default location. If rsync didn't create the file in the same directory I can see two issues: 1) Filling up /tmp on large file transfers 2) Instead of a fast rename in the same filesystem, the whole contents of the file would need to be copied out of /tmp to the final location, slowing things down significantly -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
Attached is some public domain source code that tests mkstemp() to see if
it functions properly by creating a 2GB sparse file in /tmp, first via the
normal open()/lseek()/write() pathway, then using
mkstemp()/lseek()/write().
Perhaps this could be migrated into an rsync test case? My only concern
would be possibly filling up someone's filesystem if it didn't support
sparse files.
-- Steve/* Demonstration code that tests the mkstemp() system call
to see if it can safely create files larger than 2GB.
Steve Bonds
July 28 2004
no Copyright claimed
This code is released into the public domain
build with:
gcc -g -D_FILE_OFFSET_BITS=64 -o output file this file
or
cc -g -D_FILE_OFFSET_BITS=64 -o output file this file
*/
#include stdio.h
#include stdlib.h
#include unistd.h
#include errno.h
#include string.h
#include fcntl.h
#include sys/types.h
#include sys/stat.h
#define BUFSIZE 1024
int main(void) {
char mkstemp_template[2048] = /tmp/mkstemp_XX\0;
char static_template[2048] = /tmp/mkstemp_static\0;
off_t file_offset;
long long twogigs = 2147483648;
int fd;
int i;
/* How many of the below buffers to write out after seeking to 2GB */
int buffers_to_write = 10;
/* Buffer that will contain zeros for writing out to disk */
void *buffer;
/* Allocate some memory for our buffer and fill it with known data */
if ( (buffer = malloc(BUFSIZE)) == 0) {
printf(Problem malloc()ing %d bytes of memory for a buffer: %s\n,
(int)BUFSIZE, strerror(errno));
}
(void) memset( buffer, 1, BUFSIZE);
/* Test that a normally created file can exceed 2GB */
printf(Testing normally opened file.\n);
if ( (fd = open(static_template,O_CREAT|O_EXCL|O_WRONLY,S_IRUSR|S_IWUSR)) 0) {
printf(Problem opening normal file %s: %s\n,
static_template,strerror(errno));
exit(1);
}
printf( open(%s,O_CREAT|O_EXCL|O_WRONLY) = %d\n,
static_template,fd);
if ( (file_offset = lseek(fd,twogigs,SEEK_SET)) 0 ) {
printf(Problem seeking to %lld in %s: %s\n,
(long long)twogigs,static_template,strerror(errno));
exit(1);
}
printf( lseek(%d,%lld,SEEK_SET) = %lld\n,
fd,(long long)twogigs,(long long)file_offset);
for (i=0;ibuffers_to_write;i++) {
if (write(fd,buffer,sizeof(buffer)) 0) {
printf(Problem writing data to %s: %s\n,
static_template,strerror(errno));
exit(1);
}
}
printf(Normally opened file writes OK\n);
close(fd);
/* Create a temporary file using mkstemp() */
printf(Creating a temporary file based on %s using mkstemp()\n,
mkstemp_template);
if ( (fd = mkstemp(mkstemp_template)) 0) {
printf(Problem creating file based on template %s: %s\n,
mkstemp_template, strerror(errno));
exit(1);
}
printf( Temp file created, fd = %d, filename = '%s'\n,
fd,mkstemp_template);
/* Seek to 2GB mark creating a sparse file in the process */
if ( (file_offset = lseek(fd,twogigs,SEEK_SET)) 0 ) {
printf(Problem seeking to %lld in %s: %s\n,
(long long)twogigs,mkstemp_template,strerror(errno));
exit(1);
}
printf( lseek(%d,%lld,SEEK_SET) = %lld\n,
fd,(long long)twogigs,(long long)file_offset);
for (i=0;ibuffers_to_write;i++) {
if (write(fd,buffer,sizeof(buffer)) 0) {
printf(Problem writing data to %s: %s\n,
mkstemp_template,strerror(errno));
exit(1);
}
}
printf(All writes OK, you may wish to remove the temp file '%s'\n,
mkstemp_template);
close(fd);
printf(Remember to remove the two sparse files that were created:\n);
printf( %s\n,static_template);
printf( %s\n,mkstemp_template);
exit(0);
}
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
I've been able to duplicate this problem using the CVS version of rsync that was current as of about 2000 UTC today (July 27 2004) This was on an HPUX 11.0 system, not 11i. When running: ./rsync --archive -v --progress /home/sbonds/bigfile /home/sbonds/dest_dir/. I get: - bigfile rsync: writefd_unbuffered failed to write 32768 bytes: phase unknown: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(902) - When working with GDB to get a backtrace, the third process spawned is the one that exits. (On my HPUX system the PIDs are assigned sequentially. The main rsync is PID, the sender is PID+1, and the receiver is PID+2.) The receiver backtrace is: - (gdb) bt #0 _exit_cleanup (code=11, file=0x56cf0 receiver.c, line=253) at cleanup.c:88 #1 0x9dc0 in receive_data (f_in=0, fname_r=0x77ff1a90 bigfile, fd_r=1,size_r=2147483647, fname=0x77ff1a90 bigfile, fd=4,total_size=2532431502) at receiver.c:253 #2 0xac30 in recv_files (f_in=0, flist=0x4000cb50, local_name=0x0) at receiver.c:538 #3 0x125b4 in do_recv (f_in=0, f_out=1, flist=0x4000cb50, local_name=0x0) at main.c:488 #4 0x129ec in do_server_recv (f_in=0, f_out=1, argc=1, argv=0x77ff085c) at main.c:589 #5 0x12b4c in start_server (f_in=0, f_out=1, argc=2, argv=0x77ff0858) at main.c:620 #6 0x12a44 in child_main (argc=2, argv=0x77ff0858) at main.c:596 #7 0x2db3c in local_child (argc=2, argv=0x77ff0858, f_in=0x77ff07b8, f_out=0x77ff07bc, child_main=0x400050ea child_main) at pipe.c:152 #8 0x11e48 in do_cmd (cmd=0x0, machine=0x0, user=0x0, path=0x4000bb20 /home/sbonds/dest_dir/., f_in=0x77ff07b8, f_out=0x77ff07bc) at main.c:332 #9 0x138cc in start_client (argc=1, argv=0x4000b540) at main.c:911 #10 0x13efc in main (argc=2, argv=0x4000b540) at main.c:1112 - This was built using CVS version 1.24 of cleanup.c, 1.97 of receiver.c, and 1.212 of main.c. - $ ./rsync --version rsync version 2.6.2 protocol version 28 Copyright (C) 1996-2004 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, no IPv6, 32-bit system inums, 64-bit internal inums rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. - Running tusc (HP's system call trace utility, like truss on Solaris or strace on Linux) leads me to the following failed system call: - write(4, 3 5 1fc85 1a, b4$ ^ \ b2y - d .., 262144) = 262143 write(4, 0x400a0b37, 1) . ERR#27 EFBIG - File descriptor four is the temporary file created by rsync, and EFBIG is File too large, which the HP man page for write(2) reports as: - [EFBIG]An attempt was made to write a file that exceeds the implementation-dependent maximum file size or the process' file size limit. - This is what I would expect to see if the VXFS filesystem was not created with the largefiles option-- but it was. (And I double-checked.) Other utilities (e.g. dd) can create large files just fine. I haven't seen anything obviously wrong with write_file or flush_write_file in fileio.c (v. 1.15). Do you know what is meant by the process' file size limit? -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
On Wed, 14 Jul 2004, Don Malloy nixuser23-at-hotmail.com |Rsync List| wrote: The rsync output is as follows: server1 # /tmp/rsync --rsync-path=/tmp/rsync --stats file1.db server2:/archive1 rsync: writefd_unbuffered failed to write 32768 bytes: phase unknown: Broken pipe *** rsync error: error in rsync protocol data stream (code 12) at io.c(836) The Broken pipe message is what the OS is telling you. Normally overflowing a 32 bit offset on write() returns a different error. Broken pipe means that a connection to a child process (usually) via a pipe has been suddently severed (usually because the child died before the filehandle was closed.) I suspect what has happened is that rsync built with 64 bit file offsets, but may have used 32-bit offsets for its pipe to the child process. Or the child process is crashing for some reason giving you the Broken pipe message. Try boosting the number of v arguments you are using and see if any useful information pops up as a result. If you're in a terrible hurry, start an FTP/scp over while debugging the problem. You can always use rsync to finish it off if you get the problem solved (or just wait for FTP if you don't.) -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Enabling checksum for individual files?
On Thu, 13 May 2004, Selwyn Tang selwyn-at-hectrix.com |Rsync List| wrote: I tried to use rsync-2.6.2 to sync a set of subdirectories inside a directory. While most of the files are being sync'd correctly, rsync seems to wrongly consider one file as uptodate. Both the source and destination have this file, with the same size and same date (according ls says). But md5sum shows that they are different. By default, rsync assumes that files with the same size and date are synced and doesn't do any comparisons. If this is a one-time issue, the simplest solution is to use the touch command to change the time on the destination file to something older, then rsync will check it when run. If this is an ongoing problem, then try the --ignore-times option in rsync which turns off this optimization, but is still MUCH faster than --checksum. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to improve transfert rate with rsync
On Thu, 13 May 2004, guy cadoche guycadoche-at-mail.com |Rsync List| wrote: 1) I am using rsync with gentoo and all emerge are very fast 400 kb/s ADSL connections. When I am using rsync with two computers with the same bandwith connection (ADSL 400 kb/s) transfert is very low (40 kb/s). options are rsync -avzub. How can I improve the rate of transfert ? ADSL is an asynchronous DSL line with different upload and download speeds. I suspect that you're hitting the upload speed cap when sending files. To verify this, grab a copy of netperf: http://www.netperf.org/ This tests the raw throughput across TCP connections. Nothing will ever be able to transfer raw data faster than the rate it shows. While you may be able to download at 400K/s, uploads might be MUCH slower. I saw That it use sftp. Is there a configuration file for sftp that improve the transfert ? Often the use of ssh (not sftp) with rsync slows it down since the system CPU needs to encrypt the data and any network devices that would normally compress the data will be thwarted. On modern CPUs across slower networks (i.e. 10Mbit or below) this shouldn't be a major bottleneck. 2) How can I see the transfert rate of the file (like with scp) Check the rsync option --progress, best used with --verbose. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Where is my bottleneck?
On Thu, 11 Mar 2004, Marc Abel m-abel-at-columbus.rr.com |Rsync List| wrote: I don't expect 100 Mbit/s by any means, but 2? Is this typical? What am I missing? One good diagnostic would be to measure the network bandwidth between the two systems directly. I use netperf (http://www.netperf.org/netperf/NetperfPage.html) which often reveals problems with ethernet duplex, cabling, or switches. If netperf tests great (i.e. 90+Mbit) I would next try a bare scp and see what kind of throughput you get. It's also possible that rsync is spending too much time looking for small changes in a large file. It may be faster to use --whole-file. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: running rsync in $HOME/bin?
On Tue, 20 Jan 2004, Peter listbot-at-gmx.de |Rsync List| wrote: But it seems like rsync, started on another machine, does not find the rsync binary. What can i proceed, to run rsync out of $HOME/bin? From the rsync man page: --rsync-path=PATH Use this to specify the path to the copy of rsync on the remote machine. Useful when it's not in your path. Note that this is the full path to the binary, not just the directory that the binary is in. Despite what's implied, I don't think that the PATH is grabbed from the normal shell config, at least not when running rsync using -e ssh. Similar to scp when using ssh, the PATH is minimally defined by SSH itself. So in short, just add --rsync-path=/absolute/path/to/rsync (on the target system) to your command line and it should work. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Maximum file size with rsync?
On Tue, 12 Aug 2003, Jeff Frost J.Frost-at-fantastic.com |Rsync List| wrote: I was thinking to use the whole-file option instead of letting it attempt to chew through the files for diffs, as I had problems with rsync dealing with files in the 12-15gb range in the past, though it worked great when using --whole-file. Would there be a more efficient program to use for just moving the bits over there? I suspect that just about any program is going to be network limited on a large transfer like this. A couple things to consider: 1) If you encrypt your data by using ssh as the rsync transport, it can no longer be compressed. If you are going over any WAN links that use transparent compression, this can really reduce your throughput 2) If your CPUs are much faster than your network, using --compress. Usually it just makes things slower, but it's worth trying if you use ssh as your transport, since you'll then be encrypting less data, which can be faster. 3) If you really want speed and don't care a whit about security, look at netcat. This avoids all that slow protocol overhead but doesn't do things like preserve permissions, filenames, etc. 4) Benchmark the transfer compared with a basic FTP or rcp. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does block level replication require local disk access (Blocklev el) or can it work with a network share too
Rsync works entirely at the file level, and stores the diffs in a temporary file until the transfer is complete. No block/device level access is needed. -- Steve On Wed, 23 Jul 2003, Antony Parkin AntonyP-at-bluearc.com |Rsync List| wrote: Could i run rsync on a linux server which is connected to a network share (NAS) and replicate the diffs of the files residing on the network share? or does this functionality require block level access i.e Local disk or SAN? Many thanks, Antony. * This e-mail and any attachment is confidential. It may only be read, copied and used by the intended recipient(s). If you are not the intended recipient(s), you may not copy, use, distribute, forward, store or disclose this e-mail or any attachment. If you are not the intended recipient(s) or have otherwise received this e-mail in error, you should destroy it and any attachment and notify the sender by reply e-mail or send a message to [EMAIL PROTECTED] * -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: ssh-basic test
The broken ssh-basic test is fixed in the CVS version of rsync, but not in 2.5.6. It should work in the next release of rsync. You can get the new test from the rsync CVS server easily via the cvsweb interface: http://www.samba.org/cgi-bin/cvsweb/rsync/testsuite/ssh-basic.test?rev=1.6content-type=text/x-cvsweb-markup -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync vs. rcp
On 20 Feb 2003, Donovan Baarda abo-at-minkirri.apana.org.au |Rsync List| wrote: RSYNC DOES NOT WORK WITH 1GB+ FILES... unless you have a sufficiently large block size. See the following; http://www.mail-archive.com/rsync@lists.samba.org/msg05219.html This probably needs to be documented somewhere newbie's can find it. I'd suggest putting in a warning if rsync detects a large file (500M? 1000M?) in its file list. This should be simple to code and will serve to let people know that rsync's usually excellent transfer rates might be suffering a bit because it had to re-checksum the file. I'd also suggest making the I had to retransmit the file message visible with just -v, rather than -vv. If you all think it's a good idea, but don't want to actually code it, let me know and I'll build a patch. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Syncing large amounts of data
A good solution for the lots of little files problem is syncing below the filesystem layer (usually either at the logical volume or disk level.) This allows you to avoid the stat() system call on loads of files while looking for the ones to send. As was already suggested, if you can avoid the search by having your application keep track of what has changed and what has not, then you can save yourself a lot of system impact. -- Steve On Wed, 12 Feb 2003, Adam Herbert aherbert-at-franklinamerican.com |Rsync List| wrote: I need some suggestions. Here's my setup: 800GB of Data 14,000,000+ Files No changes just additions Files range in size from 30k - 190k -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync - what can it do?
On Wed, 5 Feb 2003, bob parker bob_parker-at-dodo.com.au |Rsync List| wrote: Maybe I'm barking completely up the wrong tree but here goes. I've downloaded a 700meg iso by ftp using a steam powered dial up connection. It took a week and naturally many resumes. Murphy's Law did not take a holiday of course so the md5sum of the downloaded iso does not match the md5sum of the one at the mirror. If you are downloading RedHat ISOs, make sure you're getting the correct version. There were two versions of 7.3 out there with different MD5SUM values. Some mirrors had one, others had the other. Really confused me for a while. ;-) Can rsync be used to selectively fetch the part or parts of the iso that are in error assuming that the mirror is also an rsync mirror? Yes. I've done this for the same thing under quite similar circumstances. Rsync does this automagically just by asking it to copy the file. How can I tell? When the rsync is finished, you can tell since the MD5SUM will be what it should be. ;-) When done the speedup factor shows you how much faster it went than if it had to send the whole file. If this factor is 1.0 you just copied the whole thing again. The --stats option also gives you some interesting info on the transfer. Using --progress will show you the effective transfer rate. If you're only validating that the file is the same so far, this transfer rate will be quite high. (Much higher than your modem bandwidth.) The hard part is finding an anonymous rsync server with the exact same ISO. Be sure to grab the (preferably GPG-signed) MD5SUM from that server BEFORE you start and verify it against the one from your original server. Otherwise you'll just be downloading the whole thing again since it's a different file. This goes without saying, but be sure to save a copy of the original downloaded file just in case something goes wrong with the rsync. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync 1tb+ each day
On Tue, 4 Feb 2003, Kenny Gorman kgorman-at-paypal.com |Rsync List| wrote: My question is this: Is it possible to tell rsync to update the blocks of the target file 'in-place' without creating the temp file (the 'dot file')? It does not look like this is possible. In receiver.c around line 452 you can see: /* recv file data */ recv_ok = receive_data(f_in,buf,fd2,fname,file-length); fd2 looks to always be a file descriptor generated via do_mkstemp. I can guarantee that no other operations are being performed on the file at the same time. The docs don't seem to indicate such an option. Rsync works best over low bandwidth links for which the disk I/O is minimal. You might try one of these ideas for your high-bandwidth environment: + hack receiver.c so that receive_data uses fd1 (the original file) - also comment out finish_transfer, which does the rename and sets the permissions. If perms are important, then set them manually. - check the dest file closely to make sure it's not mangled - This is completely untested. If you're not comfortable with hacking and the usual followup debugging, then skip it. + compile and run rsync with profiling libraries to make sure that it's slow where you think it's slow. I've been surprised before. ;-) + use a dumber file transfer method (FTP, netcat) that will be faster on beefier hardware. Netcat is especially fast if you have a private network where you don't have to worry about pesky things like authentication. For Oracle datafiles I've had excellent luck with a homebrew file transfer that compresses blocks of zeros by sending a message that means Hey! I just read XXX blocks of nothing but zeroes. The receiver then creates a sparse file on the destination for this area of zeroes. It's very handy for making copies of the database for read-only purposes and it saves lots of disk space. It doesn't help at all vs. something like netcat if your datafiles are mostly full, tho. -- Steve PS: You can get info on netcat from: http://www.sans.org/rr/audit/netcat.php -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync 1tb+ each day
On Tue, 4 Feb 2003, Steve Bonds wrote: You might try one of these ideas for your high-bandwidth environment: + hack receiver.c so that receive_data uses fd1 (the original file) - also comment out finish_transfer, which does the rename and sets the permissions. If perms are important, then set them manually. - check the dest file closely to make sure it's not mangled - This is completely untested. If you're not comfortable with hacking and the usual followup debugging, then skip it. I just thought of several ways where this could result in some nasty infinite loops within rsync. You're probably better off skipping this hack it section and focussing on something else. ;-) -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Rsync 2.5.6 ssh-basic.test patch
Rsync folks:
I couldn't get the ssh tests to work in rsync 2.5.6 and it doesn't appear
that they ever could have worked. There were two problems with the script
I noticed:
1) The from directory is never populated with files to test. It looked
like the hands_setup function from rsync.fns was designed for this, so I
added it in.
2) The F1 variable used to pick a random(?) file to rename in order to
check rsync's ability to find differences was never set. This means that
the following mv command would fail since it would try to move a
directory into a subdirectory of itself.
Attached in MIME format is the patch I used to fix the problem. (Patch is
against current CVS version)
-- Steve Bonds
PS: Just in case your lawyers care, I release this code into the public
domain and claim no copyright.
Index: testsuite/ssh-basic.test
===
RCS file: /cvsroot/rsync/testsuite/ssh-basic.test,v
retrieving revision 1.5
diff -u -r1.5 ssh-basic.test
--- testsuite/ssh-basic.test22 Mar 2002 06:07:50 - 1.5
+++ testsuite/ssh-basic.test3 Feb 2003 20:02:44 -
@@ -23,7 +23,20 @@
test_skipped Skipping SSH tests because ssh conection to localhost not
authorised
fi
+# Added by Steve Bonds Feb 2 2003
+# Without this, there are no files in the ${FROM} directory, so rsync has
+# nothing to do.
+hands_setup
+
runtest ssh: basic test 'checkit $RSYNC -avH -e ssh --rsync-path=$RSYNC ${FROM}/
localhost:${TO} ${FROM}/ ${TO}'
+
+# Added by Steve Bonds Feb 2 2003
+# I assumed that F1 was intended to hold a single file for testing if
+# rsync could detect a renamed file over ssh. Without this line below
+# it was unset so the mv tried to move a parent directory into a
+# subdirectory of itself. There is probably a better way of pulling out
+# a sample file to rename.
+F1=`ls ${TO} | head -5 | tail -1`
mv ${TO}/${F1} ${TO}/ThisShouldGo
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [patch] for rsync
Dave: I don't suppose you could upload this as a context diff? (diff -c original new file) My patch program (2.5.4) couldn't get it to apply vs. the current CVS version of rsync. -- Steve -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.5, HPUX, getting unexplained error at main.c(578)
I've seen this error on HP-UX systems as well. I spent a couple hours picking through the source trying to track it down, but so far no luck. What I have discovered: + It's still broken as of last night's CVS version + It only seems to happen when sending files from an HP-UX system (receiving files on HP-UX works OK) + The remote system can be either HP-UX or Linux (and probably others), which suggests that it's a sender-side issue + It only happens when using the --owner or --group options (both implied by -a or --archive) + It does happen when using those options as a non-root user (when their action would normally be ignored). It also happens when sending the files to root on the destination. + The return status of the wait_process() call appears to be normal (i.e. not obviously mangled), but is nonzero. The actual value seems to be random. I plan to look at this more, but hopefully this info might help someone else track down this elusive HP-UX problem. -- Steve Bonds -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
