Re: AIX 5.1 rsync large file
Let's try. [EMAIL PROTECTED] wrote: >There is one file that is over 45 GB, and it started having >trouble with that file. > > I don't know what the size of the file was before >it started having trouble > >We tried pushing and pulling both, and it gets that error both ways. > Rsync has two, unrelated, definitions for connection endpoints. The side initiating the connection is called the "client", while the other side is called the "server". Then again, the side that has the original files to be synched is called the "sender", while the other side is called the "receiver". Switching from pushing to pulling switches the client and server sides, but does nothing to change the sender and receiver tags. The code that freaks out on you is in the sender, and so the switch has no effect. >The error message was: >ERROR: out of memory in build_hash_table > >ulimit -a shows: >memory(kbytes) 32768 > > So you are telling rsync to synchronize a 45GB file using no more than 32MB of memory? Try setting it much higher, if "unlimited" is not an option. Under "bash" and Linux, this is done with "ulimit -v unlimited". I'm not sure that the corresponding AIX command will be the same. >Any help and suggestions would be appreciated, I'm at my wit's end with >this stuff, since I've never used it previous to this. > > Yes. rsync 2.6.7 has a change that is meant to reduce the memory consumption of the hash table during transfer of large files. While on the subject, the "patches" directory has a patch that is meant to speed things up when using files larger than 2.5GB. Switching to 2.6.7 is, therefor, adviseable (though it may well be that 32MB of memory will still not be enough). While on the matter, any feedback regarding comparing 2.6.7 with and without the "dynamic_hash" patch would be greatly appreciated. I'm trying to push Wayne into putting it into the actual rsync code. A benchmark saying "it took so and so time with vanilla 2.6.7 and so and so with 2.6.7 + dynamic_hash" would be much appreciated. Shachar -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: buffer overflow in receive_file_entry
On Thu, Mar 23, 2006 at 07:22:22PM -0500, Robert Fitzpatrick wrote: > I assume this means it is having problems with the length of the file > name? Yes. The output you cited indicates that the combined filename would be 124+137 = 261 bytes, which overflows the horrendously small MAXPATHLEN size under MS Windows. The only way to work around this at present is to start the transfer deeper in the hierarchy. (The patch you cited is for something different, and is already present in 2.6.6.) ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
buffer overflow in receive_file_entry
I have rsync 2.6.6 on unix source to a cygwin setup on a Windows 2003
server. Was working well, then I moved a folder down one subdirectory
and started getting the following. I assume this means it is having
problems with the length of the file name? I have the same problem with
long file names using cwRsync. Are there any options other than
modifying the file names?
backup# ./820.remote-backup
No matching processes were found
No matching processes were found
rsync: readdir("/home/backup/Ydrive/tpgsharedfiles/ACCTS/FELCOR/Crowne
Plaza Secaucus - Meadowlands/Due Diligence Materials for Website"): Bad
file descriptor (9)
rsync: readdir("/home/backup/Ydrive/tpgsharedfiles/ACCTS/MeriStar/South
Seas Resort"): Bad file descriptor (9)
overflow: flags=0x3a l1=124 l2=137
lastname=Ydrive/tpgsharedfiles/ACCTS/Goldman Sachs/Wyndham Westshore/Due
Diligence/CD Content/20. - Property Condition Report/28.20.03 ENSR
Property Condition Assessment dated June 1997 vol.
2/28.20.03_ENSR_Property_Condition_Assessment_dated_June_1997_vol._3.pdf
ERROR: buffer overflow in receive_file_entry
rsync error: error allocating core memory buffers (code 22)
at /home/lapo/packaging/tmp/rsync-2.6.6/util.c(126)
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(443)
I googled a bit and found the following, is this patch a possibility? How would
I apply that to my cygwin package?
http://lists.samba.org/archive/rsync/2005-April/012104.html
--
Robert
P.S. - Sorry for the word wrap, I have not figured out how to avoid in
Evo 2.4
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
AIX 5.1 rsync large file
Hello, I have inherited a setup where there are 2 AIX 5.1 systems in 2 separate sites. There are large database files that are backed from each site to the other via rsync. Currently, it is using rsync version 2.5.4. It does it via ssh with the options -avz. This has all been merrily plodding along for some time. There is one file that is over 45 GB, and it started having trouble with that file. I don't know what the size of the file was before it started having trouble or if they made a bunch of changes in this .dbf fille before it started having trouble. There are other large files that are smaller than this (i.e., 30 GB, etc.). The error message was: ERROR: out of memory in build_hash_table rsync error: error allocating core memory buffers (code 22) at util.c(115) rsync: connection unexpectedly closed (585911608 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) rsync: connection unexpectedly closed (1224 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) Rsyncget completed file transfer at Sat Mar 11 02:49:52 EST 2006 We tried pushing and pulling both, and it gets that error both ways. I tried downloading an already compiled version of rsync 2.6.4 (and a version I compiled as well that includes support for AIX large files), and using this on both ends, it seems to hang, and the hidden .file.xyz that it creates stalls at size 2097152. ulimit -a shows: time(seconds)unlimited file(blocks) unlimited data(kbytes) 131072 stack(kbytes)32768 memory(kbytes) 32768 coredump(blocks) 2097151 nofiles(descriptors) 2000 Any help and suggestions would be appreciated, I'm at my wit's end with this stuff, since I've never used it previous to this. Thanks a lot! Jackie Andrason IMPORTANT--PLEASE READ*** This electronic message, including its attachments, is COMPANY CONFIDENTIAL and may contain PROPRIETARY or LEGALLY PRIVILEGED information. If you are not the intended recipient, you are hereby notified that any use, disclosure, copying, or distribution of this message or any of the information included in it is unauthorized and strictly prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and permanently delete this message and its attachments, along with any copies thereof. If this electronic message contains a zipped attachment and you do not have a decompression tool, you may download unZIP (free of cost) from: http://www.mk-net-work.com/us/uz/unzip.htm. Alternatively, you may request that the attachment be resent in an uncompressed format.Thank you. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: feature request: generate symlink shadow tree
On 2006-03-23 17:40:11 +1100, russell wrote: > I was wondering if anyone has used rsync to generate forests of > symlinks which point > to a specified hierarchy. Is lndir(1) what you want? Best Martin -- http://www.tm.oneiros.de -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: error in protocol stream
On Wed, 22 Mar 2006, David Bear <[EMAIL PROTECTED]> wrote: > I am trying to use the syntax: > > rsync -av -e "ssh -l ssh-user" [EMAIL PROTECTED]::module /dest > > found at http://rsync.samba.org/ftp/rsync/rsync.html > There is an rsync daemon running on rhost.asu.edu and that has a > module named home. That comment implies to me that you missed the fact that the syntax that you are trying to use does not access a persistent daemon, but fires up a single-use daemon. Pay especial attention to the part about rsync looking for the conf file in the user's home directory. I'm pretty sure that's your problem, here. Also make sure any log paths defined in the conf file are writable by the user. John -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
On Wed 22 Mar 2006, Linus Hicks wrote: > Paul Slootman wrote: > > > >I'd recommend doing --inplace, as chances are that data won't move > >within a file with oracle data files (so it's not useful to try to find > >moved data), and copying the 4TB to temp. files every time could become > >a big timewaster. Also the -t option could be handy, not all files > >change all the time IIRC. > > The above remark about not being "useful to try to find moved data" > provoked an idea. But my understanding of --inplace is apparently different > from yours. I thought --inplace only meant that the destination file would > be directly overwritten, not that it would turn off any of the > optimizations for finding moved data. I go on what's in the manpage: --inplace This causes rsync not to create a new copy of the file and then move it into place. Instead rsync will overwrite the existing file, meaning that the rsync algorithm can't accomplish the full amount of network reduction it might be able to otherwise (since it does not yet try to sort data matches). One exception to this is if you combine the option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync distribution for HP-UX
Hello all, I've created rsync depots for HP-UX 11.11 and 11.23. How to place its to rsync.samba.org download section? Jiri -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
