Rsync / Cygwin : Cannot Allocate Memory
Hi, I'm running the latest cygwin, with rsync 2.6.6 protocol 26. My command is: c:\cygwin\bin\rsync -vrtz --delete /cygdrive/e [EMAIL PROTECTED] etc I get the following error when trying to rsync 500,000 files: rsync: fork failed in do_recv: Cannot allocate memory (12) rsync error: error in IPC code (code 14) at main.c(550) rsync: connection unexpectedly closed (4 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/tmp/rsync-2.6.6/io.c(434) Of course, this doesn't happen with a smaller fileset. I'd prefer to not have to break up the transfer, as the programmers tend to just create directories wherever they please and don't tell me when they do, then complain that their stuff isn't backed up (been there). But I'm kinda lost - I found a post that says each file uses 100 bytes, the reference poster was trying to archive 5,000,000 files - which is far more than I am. According to that post, my memory requirements would only be 50,000,000 bytes. 50MB. Even if that doubled since the post, it'd still only be 100MB. The odd thing is, It WAS working with rsync 2.5.1-dev (from an rsync_minimal archive I found on the web) - except that one was crashing on a 2GB file (damn outlook pst files), but I was getting past the filelist part. I tried upping the memory for cygwin, because the new version has a limit of 384MB, and I thought MAYBE the dll I grabbed had no limit - but upping to 1024MB didn't work. This is on a Win2k box. Any suggestions? Thanks Rick -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: AIX 5.1 rsync large file
Thank you for your response. I compiled rsync 2.6.7 and installed in and that did the trick. I don't know if it had the dynamic_hash patch or not. But I think that I was too impatient previously and the 2.6.4 would have worked had I not killed it. thanks! Jackie Andrason Sr. Global IT Infrastructure Analyst Office: 919-998-2933 Fax: 919-998-2339 [EMAIL PROTECTED] Shachar Shemesh [EMAIL PROTECTED] z To [EMAIL PROTECTED] 03/24/2006 12:14 cc AMrsync@lists.samba.org Subject Re: AIX 5.1 rsync large file Let's try. [EMAIL PROTECTED] wrote: There is one file that is over 45 GB, and it started having trouble with that file. I don't know what the size of the file was before it started having trouble We tried pushing and pulling both, and it gets that error both ways. Rsync has two, unrelated, definitions for connection endpoints. The side initiating the connection is called the client, while the other side is called the server. Then again, the side that has the original files to be synched is called the sender, while the other side is called the receiver. Switching from pushing to pulling switches the client and server sides, but does nothing to change the sender and receiver tags. The code that freaks out on you is in the sender, and so the switch has no effect. The error message was: ERROR: out of memory in build_hash_table ulimit -a shows: memory(kbytes) 32768 So you are telling rsync to synchronize a 45GB file using no more than 32MB of memory? Try setting it much higher, if unlimited is not an option. Under bash and Linux, this is done with ulimit -v unlimited. I'm not sure that the corresponding AIX command will be the same. Any help and suggestions would be appreciated, I'm at my wit's end with this stuff, since I've never used it previous to this. Yes. rsync 2.6.7 has a change that is meant to reduce the memory consumption of the hash table during transfer of large files. While on the subject, the patches directory has a patch that is meant to speed things up when using files larger than 2.5GB. Switching to 2.6.7 is, therefor, adviseable (though it may well be that 32MB of memory will still not be enough). While on the matter, any feedback regarding comparing 2.6.7 with and without the dynamic_hash patch would be greatly appreciated. I'm trying to push Wayne into putting it into the actual rsync code. A benchmark saying it took so and so time with vanilla 2.6.7 and so and so with 2.6.7 + dynamic_hash would be much appreciated. Shachar IMPORTANT--PLEASE READ*** This electronic message, including its attachments, is COMPANY CONFIDENTIAL and may contain PROPRIETARY or LEGALLY PRIVILEGED information. If you are not the intended recipient, you are hereby notified that any use, disclosure, copying, or distribution of this message or any of the information included in it is unauthorized and strictly prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and permanently delete this message and its attachments, along with any copies thereof. If this electronic message contains a zipped attachment and you do not have a decompression tool, you may download unZIP (free of cost) from: http://www.mk-net-work.com/us/uz/unzip.htm. Alternatively, you may request that the attachment be resent in an uncompressed format.Thank you. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync / Cygwin : Cannot Allocate Memory
On Fri, Mar 24, 2006 at 08:20:18AM -0600, Rick Romero wrote: rsync: fork failed in do_recv: Cannot allocate memory (12) This error is happening on the receiving side, and you seem to have only mentioned the OS and memory for the sending system (I assume -- you didn't explicitly say). Is the receiving system also running cygwin? What is its version rsync? ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync / Cygwin : Cannot Allocate Memory
Wayne Davison wrote: On Fri, Mar 24, 2006 at 08:20:18AM -0600, Rick Romero wrote: rsync: fork failed in do_recv: Cannot allocate memory (12) This error is happening on the receiving side, and you seem to have only mentioned the OS and memory for the sending system (I assume -- you didn't explicitly say). Is the receiving system also running cygwin? What is its version rsync? ..wayne.. Ahhh The receiving side is a fresh install of Debian, kernel 2.6.8-2 with a whopping 256MB of Ram. I have 2Mb 'free', no swap (What the heck did I do there? Watch my confidence start droping drastically :P ) and 5 rsync processes eating about 35Mb each.. The receiving system is running rsync 2.6.4 protocol v29. Ok, so I'll throw another 256Mb in there asap - I still don't know why just changing the sending side's version of rsync would cause the receiving side to run out of memory.. Thanks Rick -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multiplexing overflow
On Wed, Mar 22, 2006 at 08:08:28PM +0300, ?? ?? wrote: I'm using rsync v.2.6.6 (with protocol version 29). Is this the same on both ends of the connection? If the version of rsync on the sending side is older than 2.6.4, I know of one protocol- corrupting bug that might be causing this. If it's 2.6.4 or newer, then it's either an unknown rsync bug or a bug in the data being transported inaccurately by the remote shell or networking code. There is one way figure out which of these two areas is the actual cause: There is a program in the support dir named savetransfer.c. If you build that program by typing make inside the support dir, you can copy the resulting binary onto both ends of the connection and run it like this: /usr/local/bin/rsync -avze /tmp/savetransfer -o /tmp/from.server ssh --rsync-path=/tmp/savetransfer -o /tmp/to.client rsync [EMAIL PROTECTED]:remote_path local_path This makes a copy of the data that the server is sending back to the client (receiver), and a copy of what the client actually received. The two resulting files should be identical (or at least only differ by the to.client file being slightly more truncated than the from.server file). If there is a data difference, then you have a problem with the sending of the bytes that is outside the control of rsync. If the bytes are the same, you might want to contact me off the list if you'd be willing to let me see the to.client data, or at least some portion of it from near the end. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync / Cygwin : Cannot Allocate Memory
Rick Romero wrote: Wayne Davison wrote: On Fri, Mar 24, 2006 at 08:20:18AM -0600, Rick Romero wrote: rsync: fork failed in do_recv: Cannot allocate memory (12) This error is happening on the receiving side, and you seem to have only mentioned the OS and memory for the sending system (I assume -- you didn't explicitly say). Is the receiving system also running cygwin? What is its version rsync? ..wayne.. Ahhh The receiving side is a fresh install of Debian, kernel 2.6.8-2 with a whopping 256MB of Ram. I have 2Mb 'free', no swap (What the heck did I do there? Watch my confidence start droping drastically :P ) and 5 rsync processes eating about 35Mb each.. The receiving system is running rsync 2.6.4 protocol v29. Ok, so I'll throw another 256Mb in there asap - I still don't know why just changing the sending side's version of rsync would cause the receiving side to run out of memory.. Adding more memory to the receiving side seems to have solved that issue. Now if I could only go back a day and find my own thread in the archives.. no wait, that wouldn't work.. damnit Thanks :) Rick -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
