Re: opendir(somedir/somefile): Not enough space -- why?
On Fri, Jan 24, 2003 at 04:02:28PM -0800, jw schultz wrote: > On Fri, Jan 24, 2003 at 04:49:20PM -0700, Bill Geddes wrote: > > Am I running out of memory on the source server? Seems perhaps more > > plausible than on the client - it has nothing going on and 1Gb RAM. > > The server has rsync version 2.5.5. The client has 2.5.6. > > Yes, it looks like the sender is running out of memory. > Break the job up. The important factor is the number of files. Rsync uses up some memory for every file it looks at in a run. That's why JW said to break it up into smaller jobs. > > I am currently mounting the server directory via NFS, and using rsync to > > copy from the mounted directory to the target. Slow, but it is running! > > If you continue over NFS be sure you do checksums of > everything afterward. NFS has (or at least used to) a > measurable error rate. Rsync is optimized for a slow "network" and high speed "disk" access. If you're doing NFS over a slow network, that looks to rsync as a slow "disk". If you can, it's probably better to run rsync directly on the server. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Cygwin issues: modify-window and hangs
On Sat, Jan 25, 2003 at 12:10:39AM -, Max Bowsher wrote: > Dave Dykstra wrote: > > While doing the tests we too experienced hangs at the end of copies. > > We were going over openssh from a Solaris 9 box to Windows 2000 > > Cygwin. > > We tried the test from > > http://lists.samba.org/pipermail/rsync/2002-August/008130.html > > but it still experienced hangs. It wasn't clear if the patch reduced > > the frequency or not. > > Data point: I regularly rsync from sources.redhat.com (Linux, I presume. Not > Cygwin, anyway) to my local machine (Cygwin). I have never experienced a > hang. Since you posted the above message, does that mean you used to experience hangs and that the rsyncs you do now are with the patch in? > > Has *anybody* been able to figure out a fix for this that really > > works? > > It sure would be nice to get something into 2.5.6 but we're about out > > of time for that because I need to put it out this weekend before I > > start a new (temporary) job on Monday. I imagine that the issue I > > was experiencing could be a separate one and this patch really would > > help other hangs people are having; can anybody give me an argument > > for putting in the calls to "shutdown()" anyway? I would make it > > only happen on Cygwin, because it is unknown if they will cause harm > > on other platforms. > > Those platforms would have to be fairly broken. The behaviour of shutdown is > pretty clearly defined. It is a relatively rarely used function, however, so it probably doesn't get much testing. Cygwin/windows is what's really broken and I'd rather not risk all the other ports to suit it. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Cygwin issues: modify-window and hangs
Dave Dykstra wrote: > While doing the tests we too experienced hangs at the end of copies. > We were going over openssh from a Solaris 9 box to Windows 2000 > Cygwin. > We tried the test from > http://lists.samba.org/pipermail/rsync/2002-August/008130.html > but it still experienced hangs. It wasn't clear if the patch reduced > the frequency or not. Data point: I regularly rsync from sources.redhat.com (Linux, I presume. Not Cygwin, anyway) to my local machine (Cygwin). I have never experienced a hang. > Has *anybody* been able to figure out a fix for this that really > works? > It sure would be nice to get something into 2.5.6 but we're about out > of time for that because I need to put it out this weekend before I > start a new (temporary) job on Monday. I imagine that the issue I > was experiencing could be a separate one and this patch really would > help other hangs people are having; can anybody give me an argument > for putting in the calls to "shutdown()" anyway? I would make it > only happen on Cygwin, because it is unknown if they will cause harm > on other platforms. Those platforms would have to be fairly broken. The behaviour of shutdown is pretty clearly defined. Max. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
On Fri, Jan 24, 2003 at 04:49:20PM -0700, Bill Geddes wrote: > Am I running out of memory on the source server? Seems perhaps more > plausible than on the client - it has nothing going on and 1Gb RAM. > The server has rsync version 2.5.5. The client has 2.5.6. Yes, it looks like the sender is running out of memory. Break the job up. > I am currently mounting the server directory via NFS, and using rsync to > copy from the mounted directory to the target. Slow, but it is running! If you continue over NFS be sure you do checksums of everything afterward. NFS has (or at least used to) a measurable error rate. > On Fri, 24 Jan 2003 14:52:10 -0800 > jw schultz <[EMAIL PROTECTED]> wrote: > > > On Fri, Jan 24, 2003 at 02:20:33PM -0700, Bill Geddes wrote: > > > More error data: > > > > > > The usage I have described returns an error code of 12 - > > > Error in rsync protocol data stream > > > > > > Also, on the tail end of the std out: > > > > > > opendir(somedir): Not enough space > > > done > > > somedir/ > > > somedir/.somefile is uptodate > > > somedir/somefile.HTML > > > ERROR: out of memory in map_ptr > > > rsync error: error allocating core memory buffers (code 22) at > > > util.c(232) > > > somedir/doc/somefile/ > > > rsync: read error: Connection reset by peer > > > rsync error: error in rsync protocol data stream (code 12) at io.c(162) > > > > malloc has failed. You're out of memory. rsync can be a > > memory pig. Upgrading to cvs or 2.5.6pre* _may_ help. > > > > It is probably necessary to break the job up. A better > > gauge of the memory requirements of an rsync transfer would > > be df -i or find . | wc -l -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
Am I running out of memory on the source server? Seems perhaps more plausible than on the client - it has nothing going on and 1Gb RAM. The server has rsync version 2.5.5. The client has 2.5.6. I am currently mounting the server directory via NFS, and using rsync to copy from the mounted directory to the target. Slow, but it is running! On Fri, 24 Jan 2003 14:52:10 -0800 jw schultz <[EMAIL PROTECTED]> wrote: > On Fri, Jan 24, 2003 at 02:20:33PM -0700, Bill Geddes wrote: > > More error data: > > > > The usage I have described returns an error code of 12 - > > Error in rsync protocol data stream > > > > Also, on the tail end of the std out: > > > > opendir(somedir): Not enough space > > done > > somedir/ > > somedir/.somefile is uptodate > > somedir/somefile.HTML > > ERROR: out of memory in map_ptr > > rsync error: error allocating core memory buffers (code 22) at > > util.c(232) > > somedir/doc/somefile/ > > rsync: read error: Connection reset by peer > > rsync error: error in rsync protocol data stream (code 12) at io.c(162) > > malloc has failed. You're out of memory. rsync can be a > memory pig. Upgrading to cvs or 2.5.6pre* _may_ help. > > It is probably necessary to break the job up. A better > gauge of the memory requirements of an rsync transfer would > be df -i or find . | wc -l > > -- > > J.W. SchultzPegasystems Technologies > email address: [EMAIL PROTECTED] > > Remember Cernan and Schmitt > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Cygwin issues: modify-window and hangs
I had a friend run some Cygwin tests and we found that --modify-window=1 works just as well as --modify-window=2 on FAT filesystems to copy files from Unix and detect the difference in granularity. FAT filesystems always have timestamps that have an even number of seconds. On the other hand, NTFS filesystems can store the modification time down to the second, whereas previously people on this mailing list thought it was just like FAT for modification time. Nevertheless, I decided to change the default value for --modify-window to 1 on Cygwin rather than 0, under the thinking that even on NTFS filesystems it wouldn't do any harm because it's highly unlikely that any individual file will be modified again within one second after it is copied by rsync. While doing the tests we too experienced hangs at the end of copies. We were going over openssh from a Solaris 9 box to Windows 2000 Cygwin. We tried the test from http://lists.samba.org/pipermail/rsync/2002-August/008130.html but it still experienced hangs. It wasn't clear if the patch reduced the frequency or not. Has *anybody* been able to figure out a fix for this that really works? It sure would be nice to get something into 2.5.6 but we're about out of time for that because I need to put it out this weekend before I start a new (temporary) job on Monday. I imagine that the issue I was experiencing could be a separate one and this patch really would help other hangs people are having; can anybody give me an argument for putting in the calls to "shutdown()" anyway? I would make it only happen on Cygwin, because it is unknown if they will cause harm on other platforms. - Dave Dykstra -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
On Fri, Jan 24, 2003 at 08:57:02AM +0200, Ville Herva wrote: > On Thu, Jan 23, 2003 at 01:55:40PM -0600, you [Dave Dykstra] wrote: > > Why did you skip the fopen in log.c, which appends to the log file? > > I thought about that for a while. My reasoning was that the log file is > probably read with unix/cygwin tools anyway - if not, the administrator is > free to mount the filesystem binary or ascii. So I didn't want to enforce > CR/LF on anyone. > > In general, I think forcing ascii makes more sense for reading than writing, > because it in most imaginable cases will do no harm. > > > Because of the question about whether or not the "t" is ignored on all > > platforms, I'm a little nervous about putting this into 2.5.6. > > It could be ifdefed, would it be safe to assume the following to work? > > #if !defined(O_TEXT) > #define O_TEXT 0 > #define O_TEXT_STR "" > #else > #define O_TEXT_STR "t" > #endif > #if !defined(O_BINARY) > #define O_BINARY 0 > #define O_BINARY_STR "" > #else > #define O_BINARY_STR "b" > #endif > > and then instead of > > fopen(foo, "rt"); > > do this > > fopen(foo, "r" O_TEXT_STR); > > Comments? > > > I don't have any problem with it for 2.6.0. Maybe it should be just in > > the 'patches' directory this time. > > > > Did you check to see whether any of the code that's reading config files > > is ignoring carriage returns at the end of lines anyway? > > ISTR at least the password files were not recognized and I *think* > rsync.conf was not parsed correctly when I originally wrote the patch. > > Looking at the source, though > > - exclude.c uses fgets() to read the lines from exclude file. I don't think > it handles '\r' > - param.c Ignores all '\r' in values, so rsync.conf should be fine. I'm not > sure, whether reading the file as text would be more elegant (in theory > there _could_ be other issues than the \r\n thing...) > - authentivate.c seems to skip '\r' for passwords as well. > > So I guess you are right for the most crucial part of code (as of current > rsync). I think I'll go ahead and put in your patch with the modification of using O_TEXT_STR as you suggest. I think the risk is low. I had been concerned that perhaps older CPPs might not be able to handle that syntax, but I see other places in the rsync code that are depending on it so I think it's OK. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
On Fri, Jan 24, 2003 at 02:20:33PM -0700, Bill Geddes wrote: > More error data: > > The usage I have described returns an error code of 12 - > Error in rsync protocol data stream > > Also, on the tail end of the std out: > > opendir(somedir): Not enough space > done > somedir/ > somedir/.somefile is uptodate > somedir/somefile.HTML > ERROR: out of memory in map_ptr > rsync error: error allocating core memory buffers (code 22) at > util.c(232) > somedir/doc/somefile/ > rsync: read error: Connection reset by peer > rsync error: error in rsync protocol data stream (code 12) at io.c(162) malloc has failed. You're out of memory. rsync can be a memory pig. Upgrading to cvs or 2.5.6pre* _may_ help. It is probably necessary to break the job up. A better gauge of the memory requirements of an rsync transfer would be df -i or find . | wc -l -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: send_files failed to open filename ...
On Fri, Jan 24, 2003 at 12:48:20PM -0500, Green, Paul wrote:
> Do-Risika RAFIEFERANTSIARONJY wrote:
> > but I don't really understand this last part, especially the *double
> > buffering* ?
>
> I think JW means that you should first make a local copy of the directory
> hierarchy you want to back up to another system, and then run rsync on the
> copy. If you use different disk drives, then this local copy should be
> pretty fast, and will minimize the chances that your set of files is
> internally inconsistent (html links pointing to files that are gone). You
> can do this in either direction; if you are pushing files, you could push
> them to a temporary location, then use directory rename ("mv") to have the
> whole set appear at once. If you are pulling files, you can make the local
> copy, then rsync with the local copy.
I explained it to him offline, where such belonged.
For public benefit i'll correct the above with the statement
that you want to make the "local" copy only on the machine
actually using the data.
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
More error data: The usage I have described returns an error code of 12 - Error in rsync protocol data stream Also, on the tail end of the std out: opendir(somedir): Not enough space done somedir/ somedir/.somefile is uptodate somedir/somefile.HTML ERROR: out of memory in map_ptr rsync error: error allocating core memory buffers (code 22) at util.c(232) somedir/doc/somefile/ rsync: read error: Connection reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(162) I know that rsync works only as well as the NIC card drivers in use. I hope that this is not another problem like that. On Fri, 24 Jan 2003 11:54:13 -0700 Bill Geddes <[EMAIL PROTECTED]> wrote: > > I am attempting to use rsync to copy a large filesystem from an > HP-UX server to a Linux server with more than enough filespace. > This operation fails. A small directory from the same HP-UX server > can be transfered just as expected. > > The HP-UX server is the source. It has 1Gb RAM - the output of bdf for > the volume the source files is on is: > > Filesystem kbytesused avail %used Mounted on > /dev/vg02/apps 6144 52722896 8652256 86% /apps > > On the target Linux server, I have 1Gb RAM and 1TB of free space: > > Filesystem 1k-blocks Used Available Use% Mounted on > /dev/tcvg1/tcvol11007898916 33408 956667148 1% /vol1 > > I have "apps" configured properly as a service in the rsyncd.conf file. > I started rsync in daemon mode on the HP-UX server, and from the Linux > server I invoked: > > rsync --numeric-ids -vvva root@hpuxsrv::apps /vol1/asic_apps/ > > The file list get transferred, but then I just get repeated lines of errors: > recv_file_name(some_dir/fm_v200209) > opendir(some_dir/fm_v200209): Not enough space > > I have tried with --blocking-io and --no-blocking-io, with --bwlimitXXX. > Same problem each time. > > Any insight is appreciated. > > -- > Bill Geddes > [EMAIL PROTECTED] > > -- > > UNIX is user friendly, it's just picky about who its friends are. > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync 2.5.5
Patrick Amirian wrote: > Hi, > Can anyone tell me please where I can get the latest version of rsync > for AIX ? Download source and compile it? Max. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
rsync 2.5.5
Hi, Can anyone tell me please where I can get the latest version of rsync for AIX ? Thank you.
change codepage on Linux to overcome strange characters
Hi all, I've posted this question before, without luck unfortunately: I'm doing an rsync in between the Novell Netware ported version (daemon) and a Linux client (version 2.5.5). Filenames with odd characters in them (like "é" for instance, hope you can see this) are not synchronized correctly. Can I overcome this by changing the codepage on Linux? Or should I take steps to solve this? Thx! Kind regards, Bart Coninckx Network Administrator CNE, ASE * Sita ICT Services Lilsedijk 19 B-2340 Beerse Belgium e-mail: [EMAIL PROTECTED] Tel: + 32 (0) 14 62 28 22 Fax: + 32 (0) 14 62 41 47 * -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
opendir(somedir/somefile): Not enough space -- why?
I am attempting to use rsync to copy a large filesystem from an HP-UX server to a Linux server with more than enough filespace. This operation fails. A small directory from the same HP-UX server can be transfered just as expected. The HP-UX server is the source. It has 1Gb RAM - the output of bdf for the volume the source files is on is: Filesystem kbytesused avail %used Mounted on /dev/vg02/apps 6144 52722896 8652256 86% /apps On the target Linux server, I have 1Gb RAM and 1TB of free space: Filesystem 1k-blocks Used Available Use% Mounted on /dev/tcvg1/tcvol11007898916 33408 956667148 1% /vol1 I have "apps" configured properly as a service in the rsyncd.conf file. I started rsync in daemon mode on the HP-UX server, and from the Linux server I invoked: rsync --numeric-ids -vvva root@hpuxsrv::apps /vol1/asic_apps/ The file list get transferred, but then I just get repeated lines of errors: recv_file_name(some_dir/fm_v200209) opendir(some_dir/fm_v200209): Not enough space I have tried with --blocking-io and --no-blocking-io, with --bwlimitXXX. Same problem each time. Any insight is appreciated. -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: how to send only the diff ?
But this is what rsync does by default. Or am I missing something? That version is well out of date. Please upgrade to a current version. PG -Original Message-From: Patrick Amirian [mailto:[EMAIL PROTECTED]]Sent: Thursday, January 23, 2003 3:46 PMTo: [EMAIL PROTECTED]Subject: how to send only the diff ? Hi, I’m trying to do so that rsync does not send a file that exists but is different on the remote. I want rsync to send only the difference to the remote. I prefer to send the difference of a 500 MB file than sending the whole file over. I use AIX and rsync version rsync version 2.4.5 protocol version 24 Oh and my command is : rsync --stats --progress --delete --partial --timeout=60 --bwlimit=10 -aRtvz patrick@risc1::test /special thank you.
RE: send_files failed to open filename ...
Do-Risika RAFIEFERANTSIARONJY wrote:
> but I don't really understand this last part, especially the *double
> buffering* ?
I think JW means that you should first make a local copy of the directory
hierarchy you want to back up to another system, and then run rsync on the
copy. If you use different disk drives, then this local copy should be
pretty fast, and will minimize the chances that your set of files is
internally inconsistent (html links pointing to files that are gone). You
can do this in either direction; if you are pushing files, you could push
them to a temporary location, then use directory rename ("mv") to have the
whole set appear at once. If you are pulling files, you can make the local
copy, then rsync with the local copy.
HTH
PG
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
(no subject)
_ STOP MORE SPAM with the new MSN 8 and get 2 months FREE* http://join.msn.com/?page=features/junkmail -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [PATCH] open O_TEXT and O_BINARY for cygwin/windows
Ville Herva wrote: > On Thu, Jan 23, 2003 at 01:55:40PM -0600, you [Dave Dykstra] wrote: >> Why did you skip the fopen in log.c, which appends to the log file? > > I thought about that for a while. My reasoning was that the log file > is probably read with unix/cygwin tools anyway - if not, the > administrator is free to mount the filesystem binary or ascii. So I > didn't want to enforce CR/LF on anyone. Yes, please don't write in explicit text mode. Keeping the flexibility here is good. Max. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
