Re: --delete-sent-files (AKA --move-files)
On Fri, Jan 16, 2004 at 10:24:02PM -0800, jw schultz wrote: Long term this would give us the ability to log at the right point, after it is actually been completed but done on the local end. Right now a push logs the completion when the sender has finished with the file even if the last bit fails. The code logs whatever the local side did. If the local side sent data, it logs that the send happened and what the stats were. If the local side received the file, it logs that instead. It doesn't currently log whether the file was successfully updated. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete-sent-files (AKA --move-files)
On Sat, Jan 17, 2004 at 09:02:27AM -0800, Wayne Davison wrote: On Fri, Jan 16, 2004 at 10:24:02PM -0800, jw schultz wrote: Long term this would give us the ability to log at the right point, after it is actually been completed but done on the local end. Right now a push logs the completion when the sender has finished with the file even if the last bit fails. The code logs whatever the local side did. If the local side sent data, it logs that the send happened and what the stats were. If the local side received the file, it logs that instead. It doesn't currently log whether the file was successfully updated. Almost but not quite. If the generator creates a non-file node it logs it regardless of local or remote. What i'm suggesting is that the logging (stdout) can be improved to actaully indicate completion. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--delete-sent-files (AKA --move-files)
Yes, it's time once again to return to the subject of moving files.
With the recent changes to the communications code between the receiver
and the generator, there is now a non-clogging channel that we can use
to signal the sender when a file has been successfully transferred,
which allows us delete the original for all transferred files. I have
in the past waffled on whether this feature needs to be in rsync. I'm
currently back on the side that it's a nice thing to support. YMMV.
Here's a new implementation of the feature that adds a generic message
(MSG_SUCCESS) for the receiver to send back to the sender (through the
generator). I made the generator simply forward this message on to the
sender, but to do that it means that the generator must be in multi-
plexed IO mode, which it used to be only when it was on the server side.
My patch adds a new internal flag that lets the code request that the
generator have messaging capability even when it is on the client side
(the non-delete-specific variable makes the code more generic). The one
not-so-nice thing about this setup is that the sender process gets the
MSG_SUCCESS for a particular index when it is down in the I/O code, and
that code doesn't know about the file list. I decided to make this code
call a new function, successful_send(), which is back in sender.c. This
function is the one that handles translating the index into a file name
and deleting the source file (assuming that the delete_sent_files flag
is on, which is currently the only time that MSG_SUCCESS gets sent). I
also added a run-time flag to mark the items we sent off to the
receiver, just to make sure that nothing funny is going on in the
sequence of events (aside: the sender side has no copy-on-write issues
to make us avoid tweaking the flags).
So, feel free to take a look and see if you like what I've done.
..wayne..
Index: flist.c
--- flist.c 17 Jan 2004 01:16:49 - 1.165
+++ flist.c 17 Jan 2004 05:04:54 -
@@ -602,7 +602,7 @@ void receive_file_entry(struct file_stru
if (!file-basename)
out_of_memory(receive_file_entry 1);
- file-flags = flags;
+ file-flags = flags LIVE_FLAGS;
file-length = read_longint(f);
if (!(flags SAME_TIME))
modtime = (time_t)read_int(f);
Index: io.c
--- io.c16 Jan 2004 16:31:47 - 1.119
+++ io.c17 Jan 2004 05:04:54 -
@@ -222,6 +222,14 @@ static void read_msg_fd(void)
read_loop(fd, buf, 4);
redo_list_add(IVAL(buf,0));
break;
+ case MSG_SUCCESS:
+ if (len != 4) {
+ rprintf(FERROR, invalid message %d:%d\n, tag, len);
+ exit_cleanup(RERR_STREAMIO);
+ }
+ read_loop(fd, buf, 4);
+ io_multiplex_write(MSG_SUCCESS, buf, 4);
+ break;
case MSG_INFO:
case MSG_ERROR:
case MSG_LOG:
@@ -637,6 +645,16 @@ static int read_unbuffered(int fd, char
}
read_loop(fd, buffer, remaining);
bufferIdx = 0;
+ break;
+ case MSG_SUCCESS:
+ if (remaining != 4) {
+ rprintf(FERROR, invalid multi-message %d:%ld\n,
+ tag, (long)remaining);
+ exit_cleanup(RERR_STREAMIO);
+ }
+ read_loop(fd, line, 4);
+ successful_send(IVAL(line, 0));
+ remaining = 0;
break;
case MSG_INFO:
case MSG_ERROR:
Index: main.c
--- main.c 17 Jan 2004 05:04:04 - 1.181
+++ main.c 17 Jan 2004 05:04:54 -
@@ -41,6 +41,7 @@ extern int list_only;
extern int local_server;
extern int log_got_error;
extern int module_id;
+extern int need_messages_from_generator;
extern int orig_umask;
extern int preserve_hard_links;
extern int protocol_version;
@@ -558,6 +559,8 @@ void start_server(int f_in, int f_out, i
io_start_multiplex_out(f_out);
if (am_sender) {
+ if (need_messages_from_generator)
+ io_start_multiplex_in(f_in);
if (!read_batch) {
recv_exclude_list(f_in);
if (cvs_exclude)
@@ -623,6 +626,9 @@ int client_run(int f_in, int f_out, pid_
io_flush(FULL_FLUSH);
exit_cleanup(status);
}
+
+ if (need_messages_from_generator)
+ io_start_multiplex_out(f_out);
if (argc == 0) {
list_only = 1;
Index: options.c
--- options.c 15 Jan 2004 17:43:34 - 1.124
+++ options.c 17 Jan 2004 05:04:55 -
@@ -81,12 +81,14 @@ int copy_unsafe_links=0;
int size_only=0;
int bwlimit=0;
int delete_after=0;
+int delete_sent_files
Re: --delete-sent-files (AKA --move-files)
On Fri, Jan 16, 2004 at 09:30:57PM -0800, Wayne Davison wrote: Yes, it's time once again to return to the subject of moving files. With the recent changes to the communications code between the receiver and the generator, there is now a non-clogging channel that we can use to signal the sender when a file has been successfully transferred, which allows us delete the original for all transferred files. I have in the past waffled on whether this feature needs to be in rsync. I'm currently back on the side that it's a nice thing to support. YMMV. Here's a new implementation of the feature that adds a generic message (MSG_SUCCESS) for the receiver to send back to the sender (through the generator). I made the generator simply forward this message on to the sender, but to do that it means that the generator must be in multi- plexed IO mode, which it used to be only when it was on the server side. My patch adds a new internal flag that lets the code request that the generator have messaging capability even when it is on the client side (the non-delete-specific variable makes the code more generic). The one not-so-nice thing about this setup is that the sender process gets the MSG_SUCCESS for a particular index when it is down in the I/O code, and that code doesn't know about the file list. I decided to make this code call a new function, successful_send(), which is back in sender.c. This function is the one that handles translating the index into a file name and deleting the source file (assuming that the delete_sent_files flag is on, which is currently the only time that MSG_SUCCESS gets sent). I also added a run-time flag to mark the items we sent off to the receiver, just to make sure that nothing funny is going on in the sequence of events (aside: the sender side has no copy-on-write issues to make us avoid tweaking the flags). So, feel free to take a look and see if you like what I've done. I'll leave aside the issue of whether we want a delete-after. What you have to underpin the delete-after is almost a per-file status being communicated. Long term this would give us the ability to log at the right point, after it is actually been completed but done on the local end. Right now a push logs the completion when the sender has finished with the file even if the last bit fails. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
