Rsync dies
I´m trying to rsync a 210 GB Filesystem with approx 1.500.000 Files. Rsync always dies after about 29 GB without any error messages. I´m Using rsync version 2.5.5 protocol version 26. Has anyone an idea ? Thank´s Clemens -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [patch] suggestions for -v option
On Thursday 2002-05-16 15:55, Dave Dykstra wrote:
| I'm afraid we've got too much history behind the current way to change
| that now. Undoubtedly there's lots of scripts around that expect the
| current behavior. The --log-format option is intended for situations
| like yours. Try --log-format %f. If the log-format option isn't
| flexible enough for what you want, a patch to it is much more likely
| to be accepted, especially if it is upward compatible.
I overlooked the --log-format option. I tried it, and it comes close
to what I want, but it doesn't report directory updates, file
deletions and hard links. While I don't care very much about
directory time updates, deleted files and hard links are important.
I'm not sure whether the log_send() and log_recv() interfaces are
suitable to log these actions.
To ensure backward compatibility, a new option to suppress the
unwanted messages could be added, e.g. --no-stats. The attached patch
implements this. For completeness, I've attached a separate patch
for the second proposed change.
--
Dick Streefland Altium Software BV
[EMAIL PROTECTED] (@ @) http://www.altium.com
oOO--(_)--OOo---
--- rsync-2.5.5/flist.c.origThu Mar 14 22:20:20 2002
+++ rsync-2.5.5/flist.c Fri May 17 12:01:04 2002
@@ -34,6 +34,7 @@
extern struct stats stats;
extern int verbose;
+extern int do_stats;
extern int do_progress;
extern int am_server;
extern int always_checksum;
@@ -72,7 +73,7 @@
static int show_filelist_p(void)
{
- return verbose recurse !am_server;
+ return verbose do_stats = 0 recurse !am_server;
}
static void start_filelist_progress(char *kind)
--- rsync-2.5.5/main.c.orig Wed Mar 27 06:10:44 2002
+++ rsync-2.5.5/main.c Fri May 17 11:53:18 2002
@@ -57,7 +57,7 @@
extern int remote_version;
int send_stats;
- if (do_stats) {
+ if (do_stats 0) {
/* These come out from every process */
show_malloc_stats();
show_flist_stats();
@@ -93,7 +93,7 @@
stats.total_read = r;
}
- if (do_stats) {
+ if (do_stats 0) {
if (!am_sender !send_stats) {
/* missing the bytes written by the generator */
rprintf(FINFO, \nCannot show stats as receiver because remote
protocol version is less than 20\n);
@@ -118,7 +118,7 @@
(double)stats.total_read);
}
- if (verbose || do_stats) {
+ if ((verbose do_stats == 0) || do_stats 0) {
rprintf(FINFO,wrote %.0f bytes read %.0f bytes %.2f bytes/sec\n,
(double)stats.total_written,
(double)stats.total_read,
--- rsync-2.5.5/options.c.orig Tue Mar 19 21:16:42 2002
+++ rsync-2.5.5/options.c Fri May 17 11:51:58 2002
@@ -260,6 +260,7 @@
rprintf(F, --blocking-io use blocking IO for the remote shell\n);
rprintf(F, --no-blocking-ioturn off --blocking-io\n);
rprintf(F, --stats give some file transfer stats\n);
+ rprintf(F, --no-stats suppress file transfer stats with -v\n);
rprintf(F, --progress show progress during transfer\n);
rprintf(F, --log-format=FORMAT log file transfers using specified
format\n);
rprintf(F, --password-file=FILEget password from FILE\n);
@@ -345,7 +346,8 @@
{compress,'z', POPT_ARG_NONE, do_compression , 0, 0, 0 },
{daemon, 0, POPT_ARG_NONE, am_daemon , 0, 0, 0 },
{no-detach,0, POPT_ARG_NONE, no_detach , 0, 0, 0 },
- {stats,0, POPT_ARG_NONE, do_stats , 0, 0, 0 },
+ {stats,0, POPT_ARG_VAL,do_stats , 1, 0, 0 },
+ {no-stats, 0, POPT_ARG_VAL,do_stats , -1, 0, 0 },
{progress, 0, POPT_ARG_NONE, do_progress , 0, 0, 0 },
{partial, 0, POPT_ARG_NONE, keep_partial , 0, 0, 0 },
{ignore-errors,0, POPT_ARG_NONE, ignore_errors , 0, 0, 0 },
--- rsync-2.5.5/rsync.1.origWed Feb 6 22:21:19 2002
+++ rsync-2.5.5/rsync.1 Fri May 17 11:59:36 2002
@@ -305,6 +305,7 @@
--blocking-io use blocking IO for the remote shell
--no-blocking-ioturn off --blocking-io
--stats give some file transfer stats
+ --no-stats suppress file transfer stats with -v
--progress show progress during transfer
--log-format=FORMAT log file transfers using specified format
--password-file=FILEget password from FILE
@@ -778,6 +779,10 @@
This tells rsync to print a verbose set of statistics
on the file transfer, allowing you to tell how effective the rsync
algorithm is for your data\.
+.IP
+.IP \fB--no-stats\fP
+When you specify the -v option, rsync will show a summary of file
Re: Status Query - Please respond - Re: Patch to avoid 'Connection reset by peer' error for rsync on cygwin
Combined reply: Mark - Point taken. But even if it worked correctly everywhere, to me there seems to be something aesthetically wrong about just letting sockets close themselves. Kind of like a malloc() without a free(). Wayne - Wouldn't the atexit solution require that we keep a list of fds to close? Anyway, I think it is now up to me to make a patch which fixes the problem in the most discreet and unsweeping manner possible, and then post it for Martin to decide. Max. Mark Eichin [EMAIL PROTECTED] wrote: Just because Linux lets you get away without it doesn't mean its a good idea. Except this has nothing to do with linux - this is unix behaviour that goes all the way back, it's part of the process model. It's part of what exiting *does*, so it *is* a bug in cygwin if it isn't doing the cleanup (and really, cygwin does so many other bizarre things under the covers to create Unix/POSIX semantics in the Very Different win32 environment, I'm more than a little surprised it isn't doing this already... skeptical, even, though I haven't been deep inside cygwin in years.) Wayne Davison [EMAIL PROTECTED] wrote: On Thu, 16 May 2002, Max Bowsher wrote: That just moves the shutdown call from where you finish with the fd to where you start using the fd - that's got to be less intuitive. Being more or less intuitive is not the point. The idea was to have as little cygwin kludge code as possible. Thus, we'd just have one call to atexit() during startup, with the single cleanup function being able to handle any and all opened sockets, and we're done (if this is even feasible -- I haven't looked into it). This was prompted by Martin's statement that he considers this a cygwin bug -- I was assuming that he didn't want to make sweeping changes to all the cleanup code in rsync. Whether he wants to handle this in a more invasive manner is up to him. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsynch related question
He's on two different cells so I don't think he would be able to do what you're stating. I think he would have to have a db server in his environment thats part of the source cell. But I'm not sure about that. sri On Fri, May 17, 2002 at 11:34:45AM -0600, [EMAIL PROTECTED] wrote: Doesn't AFS do inter-cell communication? Why not just have first1 - /afs/tr/software be the RW replica, and second1 - /afs/ddc/software be a RO replica? You just release them all instead of messing with rsync at all? With the source constantly changing, you'll generate a lot fewer errors that way, and it'll be a lot easier on your resources. If AFS can't do this, I apologize for the useless info. My experience is with IBM/TransARC DCE/DFS (and that's almost 2 years stale). Tim Conway [EMAIL PROTECTED] 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), .\n ' There are some who call me Tim? Umadevi C Reddy [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 05/16/2002 05:08 PM To: [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject:rsynch related question Classification: Hello, I had a question on rsynch'ing, please do help me on this, I have scenario where,I need to synch the two softwares, where both the software are across the network, on different cells( AFS cells). For ex: first1 - /afs/tr/software , second1 - /afs/ddc/software Both the softwares are same fist1 cell will be constantly updating and I need to synch this software to sencond1. In this scenario what command I should use ? I will appreciate your great help. Thanks in advance Thanks Uma -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsynch related question
Hi Thanks for info. Let me clear what I want to achieve from this. Right now first1(/afs/tr/software ) is running has a master and a period of time second1(/afs/ddc/software) will take over, after that first1 will not exists. I am in the process of transition work. Currently first1 is owned by some one else and second1 is, we are going to maintain in the future. One solution is I can cutover and transfer directly, but it is not immediate and mean time I should see how it works on second1 environment and for that purpose I am started maintaining second1( need to modify hardcode paths and things like that) and also I would like to reflect changes made in first1. So thinking of using rsync. Now you can give suggestion, which one is feasible? Thanks a lot Warm regards, Uma IBM Pittsburgh, 11 Stanwix Street, Pittsburgh, PA 15222 Tel: (412) 667-3121 e-mail: [EMAIL PROTECTED] tim.conway@phi lips.com To: Umadevi C Reddy/India/IBM@IBMIN cc: [EMAIL PROTECTED] 05/17/2002 Subject: Re: rsynch related question 01:34 PM Please respond to tim.conway Doesn't AFS do inter-cell communication? Why not just have first1 - /afs/tr/software be the RW replica, and second1 - /afs/ddc/software be a RO replica? You just release them all instead of messing with rsync at all? With the source constantly changing, you'll generate a lot fewer errors that way, and it'll be a lot easier on your resources. If AFS can't do this, I apologize for the useless info. My experience is with IBM/TransARC DCE/DFS (and that's almost 2 years stale). Tim Conway [EMAIL PROTECTED] 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), .\n ' There are some who call me Tim? Umadevi C Reddy [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 05/16/2002 05:08 PM To: [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject:rsynch related question Classification: Hello, I had a question on rsynch'ing, please do help me on this, I have scenario where,I need to synch the two softwares, where both the software are across the network, on different cells( AFS cells). For ex: first1 - /afs/tr/software , second1 - /afs/ddc/software Both the softwares are same fist1 cell will be constantly updating and I need to synch this software to sencond1. In this scenario what command I should use ? I will appreciate your great help. Thanks in advance Thanks Uma -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Rsync dies
Allen, John L. wrote: In my humble opinion, this problem with rsync growing a huge memory footprint when large numbers of files are involved should be #1 on the list of things to fix. It seems that every fifth post is a complaint about this problem! Sorry if this sounds like ungrateful whining, but I think rsync would be greatly improved if the overall way it processes multiple files were overhauled to be much more resource friendly. Thanks for that opening ;-) My guess is that such a change would be a fairly big undertaking. If someone considers going that far, I'd like to suggest at least considering dividing rsync functionality into two (or a few) programs. (This isn't completely fair, but when I hear about Linux programs that do one thing only (and do it well), I laugh to myself and think about the --brush_teeth option that must be in there somewhere. ;-) ) It seems to me it would be nice to have a simple module (i.e., standalone program) whose basic function in life is to use the rsync algorithm to transfer a single file efficiently. Then, one or more other programs (that can call the first program as needed) can be provided to do things like traverse a directory to find out what files must be rsync'd, change ownership and permission if required, consider includes and excludes, and whatever else. Seems to me the programming would be simpler, the documentation could be simpler and more understandable, and users could be less confused. Just another $.02. Randy Kramer -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Rsync dies
In my humble opinion, this problem with rsync growing a huge memory footprint when large numbers of files are involved should be #1 on the list of things to fix. I think many would agree. If it were trivial, it'd probably be done by now. Fix #1 (what most people do): Split the files/paths to limit the size of each job. What someone could/should do here is at least edit the BUGS section of the manual to talk about the memory restrictions. Fix #2 (IMHO, what should be done to rsync): File caching of results (or using a file-based database of some sorts) is the way to go. Instead of maintaining a data structure entirely in memory, open a (g)dbm file or add hooks into the db(3) libraries to store file metadata and checksums. It'll be slower than an all-memory implementation, but large jobs will at least finish predictably. Fix #3 (what I did): If you really really need to efficiently transfer large numbers of files, come up with your own custom process. I used to run a large web site with thousands of files and directories that needed to be distributed to dozens of servers atomically. Using rsync, I'd run into memory problems and worked around them with Fix #1. Another problem was running rsync in parallel. The source directory was scanned order(N) times when it needed to be scaned only once. The source content server was pummeled from the multiple simultaneous instances. I resorted to making my own single-threaded rsync-like program in Perl to behave more like Fix #2 and runs very efficiently. I've spent a some time cleaning up this program so that I can publish it, but priorities (*) are getting in the way. When I get some time, you'll see it posted here. -- Eric Ziegast (*) Looking for a full-time job is a full-time job. :^( Will consult for food. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Improving the rsync protocol (RE: Rsync dies)
On Fri, 17 May 2002, Allen, John L. wrote: In my humble opinion, this problem with rsync growing a huge memory footprint when large numbers of files are involved should be #1 on the list of things to fix. I have certainly been interested in working on this issue. I think it might be time to implement a new algorithm, one that would let us correct a number of flaws that have shown up in the current approach. Toward this end, I've been thinking about adding a 2nd process on the sending side and hooking things up in a different manner: The current protocol has one sender process on the sending side, while the receiving side has both a generator process and a receiver process. There is only one bi-directional pipe/socket that lets data flow from the generator to the sender in one direction, and from the sender to the receiver in the other direction. The receiver also has a couple pipes connecting itself to the generator in order to get data to the sender. I'd suggest changing things so that a (new) scanning process on the sending side would have a bi-directional link with the generator process on the receiving side. This would let both processes descend through the tree incrementally and simultaneously (working on a single directory at a time) and figure out what files were different. The list of files that needed to be transferred PLUS a list of what files need to be deleted (if any) would be piped from the scanner process to the sender process, who would have a bi-directional link to the receiver process (perhaps using ssh's multi-channel support?). There would be no link between the receiver and the generator. The advantage of this is that the sender and the receiver are really very simple. There is a list of file actions that is being received on stdin by the sending process, and this indicates what files to update and which files to delete. (It might even be possible to make sender be controlled by other programs.) These programs would not need to know about exclusion lists, delete options, or any of the more esoteric options, but would get told things like the timeout settings via the stdin pipe. In this scenario, all error messages would get sent to the sender process, who would output them on stdout (flushed). The scanner/generator process would be the thing that parses the commandline, communicates the exclude list to its opposite process, and figures out exactly what to do. The scanner would spawn the sender, and field all the error messages that it generates. It would then either output the errors locally or send them over to the generator for output (depending on whether we're pushing or pulling files). As for who spawns the receiver, it would be nice if this was done by the sender (so they could work alone), but an alternative would be to have the generator spawn the receiver and then then let the receiver hook up with the sender via the existing ssh connection. This idea is still in its early stages, so feel free to tell me exactly where I've missed the boat. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: I/O error when deleting files
OK, but I'm not exactly sure what I'm looking for...
I don't think the link error is caused by my data (I have no symlinks).
For whatever reason, it appears that a blank is leading the file list,
and the 'stat' on NULL is what is causing the link_stat error.
write(1, b u i l d i n g f i l.., 23) = 23
brk(0x00058668) = 0
brk(0x00060668) = 0
lstat64(, 0xFFBEF8D8) Err#2 ENOENT
write(2, l i n k _ s t a t :.., 39) = 39
The rsync child then just says an error has occured and tells the
parent to not bother deleting files because of it.
waitid(P_PID, 10233, 0xFFBEF890, WEXITED|WTRAPPED|WNOHANG) = 0
Received signal #18, SIGCLD, in poll() [caught]
siginfo: SIGCLD CLD_EXITED pid=10233 status=0x000C
poll(0xFFBEF870, 0, 20) Err#4 EINTR
waitid(P_ALL, 0, 0xFFBEF3A8, WEXITED|WTRAPPED|WNOHANG) = 0
waitid(P_ALL, 0, 0xFFBEF3A8, WEXITED|WTRAPPED|WNOHANG) Err#10 ECHILD
setcontext(0xFFBEF558)
poll(0xFFBEF870, 0, 11) = 0
waitid(P_PID, 10233, 0xFFBEF890, WEXITED|WTRAPPED|WNOHANG) Err#10 ECHILD
time() = 1021666775
write(1, w r o t e 7 9 8 5 5 .., 54) = 54
write(1, t o t a l s i z e i.., 43) = 43
sigaction(SIGUSR1, 0xFFBEF848, 0xFFBEF8C8) = 0
sigaction(SIGUSR2, 0xFFBEF848, 0xFFBEF8C8) = 0
waitid(P_PID, 10233, 0xFFBEF888, WEXITED|WTRAPPED|WNOHANG) Err#10 ECHILD
write(2, r s y n c e r r o r :.., 55) = 55
I see that I can turn on ignore-errors mode, but that just seems
dangerous if there were ever anything other than the case I am seeing
here. So I guess my question is what's with the lstat64()??
--bill
I suggest running truss -f -o trussfile before that command and look for
the error in the trussfile and see what's happening just before it. Under
some circumstances rsync can print that message when a symlink goes
nowhere, but I'm not sure what the circumstances are. Perhaps it does that
because you're using -x, I'm not sure.
- Dave
On Wed, May 15, 2002 at 02:30:00PM -0700, Bill Houle wrote:
/usr/local/bin/rsync \
-axv --delete \
${MAINDIR}/${FSERVER}/ \
${MAINDIR}/${TSERVER}/
Both locations are on the same (non-NFS) partition.
All old files are updated and new files copied, but the orphans
are never deleted...
--bill
What is the command you are running?
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]On
Behalf Of Bill Houle
Sent: Monday, 1 January 1601 11:00 AM
To: [EMAIL PROTECTED]
Subject: I/O error when deleting files
When I run rsync in no-op mode, all appears fine (it knows what to
copy, what to delete). But when I run it for real, I get:
building file list ... link_stat : No such file or
directory done
IO error encountered - skipping file deletion
followed by:
total size is 98117172 speedup is 563.78
rsync error: partial transfer (code 23) at main.c(578)
Any help/pointers -- Solaris 8 on Ultra -- is appreciated.
--bill
--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.tuxedo.org/~esr/faqs/smart-questions.html
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Improving the rsync protocol (RE: Rsync dies)
Wayne: If anybody can make that work, I'd bet you could. The basic rsync algorythm is in place, so as you say, it would mostly be a matter of list generation. You'd have to hold on to any files with 1 link, in a seperate list, to find all the linkage relationships, which could grow a bit, but it does sound more efficient. Maybe a third pipe to send the files over as you go? Mind: I'm not offering to help. It's too complicated for my tiny mind. Tim Conway [EMAIL PROTECTED] 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), .\n ' There are some who call me Tim? Wayne Davison [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 05/17/2002 02:42 PM To: rsync users [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject:Improving the rsync protocol (RE: Rsync dies) Classification: On Fri, 17 May 2002, Allen, John L. wrote: In my humble opinion, this problem with rsync growing a huge memory footprint when large numbers of files are involved should be #1 on the list of things to fix. I have certainly been interested in working on this issue. I think it might be time to implement a new algorithm, one that would let us correct a number of flaws that have shown up in the current approach. Toward this end, I've been thinking about adding a 2nd process on the sending side and hooking things up in a different manner: The current protocol has one sender process on the sending side, while the receiving side has both a generator process and a receiver process. There is only one bi-directional pipe/socket that lets data flow from the generator to the sender in one direction, and from the sender to the receiver in the other direction. The receiver also has a couple pipes connecting itself to the generator in order to get data to the sender. I'd suggest changing things so that a (new) scanning process on the sending side would have a bi-directional link with the generator process on the receiving side. This would let both processes descend through the tree incrementally and simultaneously (working on a single directory at a time) and figure out what files were different. The list of files that needed to be transferred PLUS a list of what files need to be deleted (if any) would be piped from the scanner process to the sender process, who would have a bi-directional link to the receiver process (perhaps using ssh's multi-channel support?). There would be no link between the receiver and the generator. The advantage of this is that the sender and the receiver are really very simple. There is a list of file actions that is being received on stdin by the sending process, and this indicates what files to update and which files to delete. (It might even be possible to make sender be controlled by other programs.) These programs would not need to know about exclusion lists, delete options, or any of the more esoteric options, but would get told things like the timeout settings via the stdin pipe. In this scenario, all error messages would get sent to the sender process, who would output them on stdout (flushed). The scanner/generator process would be the thing that parses the commandline, communicates the exclude list to its opposite process, and figures out exactly what to do. The scanner would spawn the sender, and field all the error messages that it generates. It would then either output the errors locally or send them over to the generator for output (depending on whether we're pushing or pulling files). As for who spawns the receiver, it would be nice if this was done by the sender (so they could work alone), but an alternative would be to have the generator spawn the receiver and then then let the receiver hook up with the sender via the existing ssh connection. This idea is still in its early stages, so feel free to tell me exactly where I've missed the boat. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
