Not removing source files when using --remove-source-files
I have rsync -rvz --prune-empty-dirs --remove-source-files --log-file="rsync.log" --backup -e 'ssh' -f'- Saved/*' -f'- Screenshots/*' -f'- Boondocks/*' -f'- Dilbert/*' -f'+ */' -f'+ /*' -f'+ *.jp*g' -f'+ *.png' -f'+ *.nef' -f'+ *.gif' -f'- *' linuxuser@10.0.0.11:/home/linuxuser/Pictures/ /mnt/c/users/windowsuser/ The files are transferred but they are not removed from the source. I've deleted all files and directories from the target and tried again but still, it didn't remove the source. ~ In all things, Be Intentional. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
> Upgrade to 3.2.6 (or 3.2.7pre1): https://download.samba.org/pub/rsync/NEWS#3.2.6 Thank you very much Wayne. And thank you personally to whoever made the code modifications (I wish I could find the pull request!). I can't explain how important this is to me. The new binary file is worth its weight in gold. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
On Sun, Oct 16, 2022 at 11:01 PM Sridhar Sarnobat wrote: > 90% of my data losses are caused by rsync'ing from dir A to dir A > Upgrade to 3.2.6 (or 3.2.7pre1): https://download.samba.org/pub/rsync/NEWS#3.2.6 ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
Rsync does not have a --no-clobber. -n is --dry-run. It shows what rsync would do without actually doing it. But I didn't know you were using --backup. Anyway, I tried it again with --backup and --remove-source-files and it did in fact delete all the files (but not the dirs). It didn't have any output indicating such so it seems like a problem to me. BTW, it isn't the --backup either. Just the --remove-source-files is enough I just didn't know it because it isn't in the output so --dry-run gives no clue. On 10/17/22 23:51, Sridhar Sarnobat wrote: I think I must be missing something. If source and dest are the same place rsync shouldn't do anything unless it it responding to changes happening at the same time. For example, when I do 'rsync -vain --remove-source-files /tmp/ /tmp/' rsync does nothing. Thanks for the response. You are using "-n" / "--no-clobber". In my case I'm using "-b" / "--backup" (or overwrite). My use case is to get everything out of the source location, and "--no-clobber" will leave things behind. So I guess I should refine my issue to "when you're not using --no-clobber." Do I have a case here? Or is this erasing both the source and destination a valid behaviour in my case? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
> > I think I must be missing something. If source and dest are the same > place rsync shouldn't do anything unless it it responding to changes > happening at the same time. For example, when I do 'rsync -vain > --remove-source-files /tmp/ /tmp/' rsync does nothing. > Thanks for the response. You are using "-n" / "--no-clobber". In my case I'm using "-b" / "--backup" (or overwrite). My use case is to get everything out of the source location, and "--no-clobber" will leave things behind. So I guess I should refine my issue to "when you're not using --no-clobber." Do I have a case here? Or is this erasing both the source and destination a valid behaviour in my case? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
I think I must be missing something. If source and dest are the same place rsync shouldn't do anything unless it it responding to changes happening at the same time. For example, when I do 'rsync -vain --remove-source-files /tmp/ /tmp/' rsync does nothing. On 10/17/22 23:12, Sridhar Sarnobat via rsync wrote: >> why not avoid using "--remove-source-files" and delete files manually/via extra step afterwards Multiple reasons. Among them: 1) my jobs run a long time and there are a lot of them. Trying to remember what I did over ssh (where history doesn't get saved) is an extra cognitive load and the system is in a transient state (I've ended up with duplicate files unless I do the extra work later - potentially much later depending on the size of the job). I may also have multiple rsync commands running in separate terminals. 2) I am managing a lot of large files, and my collection of disks doesn't have 2x free space. 3) 2 operations instead of 1 gives more room to make a mistake. This is a workaround that I can occasionally use, and you could even say "why not just use ext4magic?" to recover accidentally deleted files etc. Regardless, the semantics of deleting source files without leaving one copy of your files intact should never happen right? -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
>> why not avoid using "--remove-source-files" and delete files manually/via extra step afterwards Multiple reasons. Among them: 1) my jobs run a long time and there are a lot of them. Trying to remember what I did over ssh (where history doesn't get saved) is an extra cognitive load and the system is in a transient state (I've ended up with duplicate files unless I do the extra work later - potentially much later depending on the size of the job). I may also have multiple rsync commands running in separate terminals. 2) I am managing a lot of large files, and my collection of disks doesn't have 2x free space. 3) 2 operations instead of 1 gives more room to make a mistake. This is a workaround that I can occasionally use, and you could even say "why not just use ext4magic?" to recover accidentally deleted files etc. Regardless, the semantics of deleting source files without leaving one copy of your files intact should never happen right? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
why not avoid using "--remove-source-files" and delete files manually/via extra step afterwards ? Am 17.10.22 um 08:59 schrieb Sridhar Sarnobat via rsync: 90% of my data losses are caused by rsync'ing from dir A to dir A (accidental incorrect copy and paste, or where dir B is a symlink to dir A). The source gets erased unintentionally. It may seem dumb but when you get really long commands and really long file paths it will happen a 1-2 times a yearwhen I'm doing it dozens/hundreds of times every week. *Is there an existing mechanism to protect against this?* rsync should logically never erase all copies of data. I know I can backup files (which I do for my smaller drives), or create wrapper scripts but I'm not asking about those.I don't think adopting --dry-run is practical being such a heavy user (and moreover, it's unlikely to warn what will happen). I'm not a C developer so I probably can't do this myself but if anyone's encouraging me to then I'd give it a shot. Sridhar Sarnobat San Jose, CA 95128 | USA Phone: +1 (650) 260-3851 | ss...@cornell.edu SMS me via email: 6073395...@txt.att.net Google Chat: sarnobat.hotm...@gmail.com http://www.facebook.com/sridhar.sarnobat http://www.linkedin.com/in/sarnobat <http://www.linkedin.com/in/sarnobat> http://github.com/sarnobat -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files
90% of my data losses are caused by rsync'ing from dir A to dir A (accidental incorrect copy and paste, or where dir B is a symlink to dir A). The source gets erased unintentionally. It may seem dumb but when you get really long commands and really long file paths it will happen a 1-2 times a year when I'm doing it dozens/hundreds of times every week. *Is there an existing mechanism to protect against this?* rsync should logically never erase all copies of data. I know I can backup files (which I do for my smaller drives), or create wrapper scripts but I'm not asking about those. I don't think adopting --dry-run is practical being such a heavy user (and moreover, it's unlikely to warn what will happen). I'm not a C developer so I probably can't do this myself but if anyone's encouraging me to then I'd give it a shot. Sridhar Sarnobat San Jose, CA 95128 | USA Phone: +1 (650) 260-3851 | ss...@cornell.edu SMS me via email: 6073395...@txt.att.net Google Chat: sarnobat.hotm...@gmail.com http://www.facebook.com/sridhar.sarnobat http://www.linkedin.com/in/sarnobat http://github.com/sarnobat -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 14394] Add an option for --remove-source-files to remove files right away
https://bugzilla.samba.org/show_bug.cgi?id=14394 Wayne Davison changed: What|Removed |Added Resolution|--- |WONTFIX Status|NEW |RESOLVED --- Comment #1 from Wayne Davison --- Rsync needs to ensure that the receiver created the file before the sender removes it or you could have the file vanish when an error occurs (and exist on neither side). The deletion delay is somewhat large due to how much data is pushed down the socket from the generator. You could try specifying the --whole-file (-W) option to speed it up, but this would mean that any updated files that exist on both sides would be resent in total with no matching-data computations. You could also try breaking up the copy operations into single directories (use -d instead of -r) so that rsync finishes up each (smaller) operation more rapidly. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 14394] New: Add an option for --remove-source-files to remove files right away
https://bugzilla.samba.org/show_bug.cgi?id=14394 Bug ID: 14394 Summary: Add an option for --remove-source-files to remove files right away Product: rsync Version: 3.2.0 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 Component: core Assignee: wa...@opencoder.net Reporter: g...@zumbi.com.ar QA Contact: rsync...@samba.org Target Milestone: --- Hello I want to have an option to change the behavior of --remove-source-files to remove the file right after it was transfer instead of after the whole operation finished. My use case is to move files between systems with bad connectivity and low disk space, on the recv side the files are deleted soon after receiving them making rsync to download them again if the last sync operation didn't finish correctly. I use rsync like a queue in this case, sort of thank you so much for the wonderful rsync! -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
copy-links with remove-source-files doesn't remove the source links
I noticed this issue and found a Debian bug report about it, but didn't find any discussion from the actual rsync maintainers or in bugzilla, so I thought I would post here and let you know it doesn't do what I expect. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732228 Here are the steps for a small proof-of-concept, although I get the same results over ssh. $ rsync 2>&1 | head -1 rsync version 3.1.3 protocol version 31 $ mkdir src dst $ echo content > file $ ln -s ../file src/link $ rsync -a --copy-links --remove-source-files src/ dst/ ERROR: Skipping sender remove for changed file: link rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1189) [sender=3.1.3] $ ls -l src/link dst/link lrwxr-xr-x 1 afresh1 afresh1 7 Jan 6 16:23 src/link -> ../file -rw-r--r-- 1 afresh1 afresh1 8 Jan 6 16:23 dst/link $ cat dst/link content $ I was expecting, and hoping, that it would remove the src/link symlink, but as described in the above bug report, that doesn't happen. Thanks for your time. l8rZ, -- andrew - http://afresh1.com A printer consists of three main parts: the case, the jammed paper tray and the blinking red light. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10494] remove-source-files fails with symlinks
https://bugzilla.samba.org/show_bug.cgi?id=10494 --- Comment #5 from Adrian Torregrosa --- I embedded the modified sender.c into 3.1.3's original source code, compiled and tested, and I found it to have solved the problem: first I tried uploading a softlink and deleting, then I tried uploading a normal file and deleting. It worked the two times. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10494] remove-source-files fails with symlinks
https://bugzilla.samba.org/show_bug.cgi?id=10494 --- Comment #4 from Wayne Davison --- In my prior testing I apparently missed that the symlink verification issue was caused by -L (--copy-links). I have amended the change to use do_stat() when --copy-links is enabled. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10494] remove-source-files fails with symlinks
https://bugzilla.samba.org/show_bug.cgi?id=10494
--- Comment #3 from Adrian Torregrosa ---
Another option that I tried and verified, and that could make more sense, would
be to replace
141 if (do_lstat(fname, ) < 0) {
with
141 if (do_stat(fname, ) < 0) {
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10494] remove-source-files fails with symlinks
https://bugzilla.samba.org/show_bug.cgi?id=10494
--- Comment #2 from Adrian Torregrosa ---
We have found this same problem using rsync 3.1.2.
I have downloaded the source code for version 3.1.3, and found the following
lines in sender.c:
146 if (S_ISREG(file->mode) /* Symlinks & devices don't need this
check: */
147 && (st.st_size != F_LENGTH(file) || st.st_mtime != file->modtime
148 #ifdef ST_MTIME_NSEC
149 || (NSEC_BUMP(file) && (uint32)st.ST_MTIME_NSEC !=
F_MOD_NSEC(file))
150 #endif
151 )) {
152 rprintf(FERROR_XFER, "ERROR: Skipping sender remove for
changed file: %s\n", fname);
153 return;
154 }
I added some debugging info, compiled and tested, and found out that when
trying to retrieve symlinks the "file" pointer refers to the hard file, whereas
the "st" register refers to the symlink. Therefore "S_ISREG(file->mode)" will
be true, so will be "st.st_size != F_LENGTH(file)" and most likely "st.st_mtime
!= file->modtime" too. As a result the error gets printed and the symlink is
not deleted, as intended.
I have tried modifying the first condition and it worked as expected:
146 if (S_ISREG(st.st_mode) /* Symlinks & devices don't need this
check: */
Best regards.
--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10494] remove-source-files fails with symlinks
https://bugzilla.samba.org/show_bug.cgi?id=10494 Wayne Davison way...@samba.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #1 from Wayne Davison way...@samba.org --- That's weird, as it means that the source symlink must differ in size or modify time from what rsync put into the source file list. However, since that safety check is only really needed for regular files, I've added some code that should avoid this failure for symlinks and devices. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 6307] rsync --remove-source-files foo/ foo/ should not delete files
https://bugzilla.samba.org/show_bug.cgi?id=6307 --- Comment #2 from tombosho...@gmail.com tombosho...@gmail.com --- Same for me. No data loss here, but a bunch of work undeleting the files and moving them back to the right directories. Files should only be removed if they are actually copied to another location. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Flush delete command on rsync exit with --remove-source-files
When the option --remove-source-files is specified, files are not immediately deleted on the remote machine after being transfered. Instead, it is (as far as I understand) queued up in the list of actions. If rsync is ended with SIGTERM or SIGINT, then a transfered file may not be deleted on the remote machine. If the user moves the transfered files (for example sorting them into different folders) and then starts the transfer again, some files may be transfered more than once. Would it make sense / be possible to flush the delete commands during the graceful exit in order to insure that all transfered files have been deleted? Thanks, Christopher -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10494] New: remove-source-files fails with symlinks
https://bugzilla.samba.org/show_bug.cgi?id=10494 Summary: remove-source-files fails with symlinks Product: rsync Version: 3.1.0 Platform: x64 OS/Version: Linux Status: NEW Severity: major Priority: P5 Component: core AssignedTo: way...@samba.org ReportedBy: afr...@deteque.com QAContact: rsync...@samba.org Up thru 3.0.9 transferring symlinks using -L --remove-source-files properly transferred the files and removed the symlinks. Beginning with 3.1.0 this no longer works; the files transfer properly but the symlinks on the source server fail to get deleted. This error message is displayed: ERROR: Skipping sender remove for changed file: [transferred file name] I have not been able to find a workaround to this problem, other than downgrading from 3.1.0 back to 3.0.9. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--remove-source-files problem when used with --write-batch
I used this rsync command:
rsync -az --only-write-batch=tmp.rsync --inplace --remove-source-files
--include='*' --include='*/' tmp/ nonexistent_dir
which created file tmp.rsync.sh containing this rsync command:
rsync --filter=._- -az --read-batch=tmp.rsync --inplace
--remove-source-files ${1:-nonexistent_dir} '#E#
which when executed gives the error:
--read-batch cannot be used with --remove-source-files
and furthermore, the source directory, tmp/, still contains its files.
rsync seems to be incorrectly including --remove-source-files into the rsync.sh
file and ignoring it at the sending side.
My rsync is: rsync version 3.0.6 protocol version 30
My system is: CentOS release 5.7 (Final)
What I wanted to do was try using rsync as an archive tool, like tar/gzip.
Tar/gzip compresses the archive after it is created. I wanted each file in the
archive to be compressed before adding it to the archive, and further I wanted
the files to be deleted as soon as they were added to the archive, because I'm
trying to compress a very large directory in-place on an almost-full device.
It seemed like the above command would do the job, and in fact it does work,
except for (lack of) deletion of the sending-side file immediately after it is
added to the batch (archive) file.
It would be nice if there were a --remove-source-dirs argument also, to remove
directories on the source as well as files.
I don't know how possible or impossible this might be with rsync as it is
currently structured, but it would be nice if the batch file created by
--write-batch could be used as the destination device also. I.e., if I do the
command shown above and partway through it fails for some reason (say the
system is shut down), it would be great if the batch file were intact and
usable with its contents as they are up to that point, and if the rsync command
could be restarted and it would use that batch file as the destination, and
continue adding to it. In short, what I want is an rsync where the
destination, instead of being a directory tree, is a single file that retains
the compressed form of data used by rsync to send it.
ted
~
Old saw: 'If it ain't broke, don't fix it'
Toal's corollary: 'If it's broke, fix it'
~
Ted Toal
t...@tedtoal.net
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 3845] Make --remove-source-files remove directories too
https://bugzilla.samba.org/show_bug.cgi?id=3845 --- Comment #7 from Tammer Ibrahim tamme...@gmail.com 2012-05-07 12:56:59 UTC --- Hello, I'm also quite interested in this feature. Rsync is already the perfect augmentation of the cp command, and adding this feature would similarly allow an easy drop-in augmentation of the mv command. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 7691] rsync remove-source-files and opened file
https://bugzilla.samba.org/show_bug.cgi?id=7691 Wayne Davison way...@samba.org changed: What|Removed |Added Status|ASSIGNED|RESOLVED Version|3.0.6 |3.1.0 Resolution||FIXED --- Comment #1 from Wayne Davison way...@samba.org 2011-06-18 19:32:24 UTC --- While the user should not be getting themselves into this situation (if they're smart about it), it is good for rsync to help the user not shoot themselves in the foot. I have made rsync re-stat the file and skip the remove if the size or mtime is different. This will be fixed in 3.1.0. I've also added some helpful hints in the manpage on how best to use --remove-source-files. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 8076] New: Rsync --remove-source-files Rule Filters Enhancement
https://bugzilla.samba.org/show_bug.cgi?id=8076 Summary: Rsync --remove-source-files Rule Filters Enhancement Product: rsync Version: 3.1.0 Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P5 Component: core AssignedTo: way...@samba.org ReportedBy: the...@hotmail.com QAContact: rsync...@samba.org A suggested enhancement is to allow sender-side protection rule filters to prevent/allow deletion of files. Currently there are receiver side protection rules but none for senders. Suppose a user wants transfer file A and file B using rsync. After the transfer file B on the sender side is desired to be deleted by using the --remove-source-files option. However currently there doesn't seem to be a way to protect file A from being deleted if both file A and B are included in the same rsync call. The user could separate the single rsync call into two rsync calls, one with and one without the --remove-source-files option. If this sender-side functionality already exists then please disregard. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 7691] New: rsync remove-source-files and opened file
https://bugzilla.samba.org/show_bug.cgi?id=7691 Summary: rsync remove-source-files and opened file Product: rsync Version: 3.0.6 Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: way...@samba.org ReportedBy: theb...@zoznam.sk QAContact: rsync...@samba.org Hi, when I use option remove-source-files and source files is opened and growing rsync copy partial source file and after copying remove it even though in time after copying the files are different I suggest to check difference after file is copying and if source and destination are different don`t remove the source file next improve is to setup options like remove-source-file-delay to setup some time after copying to check difference between source and destination and remove source only if the files are the same next improve is to setup options like remove-source-file-loop to setup numbers of loop to try if source file was already closed if remove-source-file-loop=0 it will be forever I hope this scenario could be implemented independently on OS and help many people to use rsync for opened files. peter -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
on remote machine: --remove-source-files: unknown option
I'm trying to sync from 3.0.5 to 2.5.5 (SCO, *sigh*) The source machine errors out b/c the --remove-source-files (or --remove-sent-files) options don't exist on 2.5.5. But, since these option are only acted on by the sending machine (which is where I'm typing the command line), it would be so nice if the source machine would NOT pass the option on to the receiver (or at least, not barf when it doesn't work). Any way to do this? Thanks! -- _ Nachman Yaakov Ziskind, FSPA, LLM aw...@ziskind.us Attorney and Counselor-at-Law http://ziskind.us Economic Group Pension Services http://egps.com Actuaries and Employee Benefit Consultants -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: on remote machine: --remove-source-files: unknown option
On Thu, 2010-04-22 at 12:53 -0400, N. Yaakov Ziskind wrote: I'm trying to sync from 3.0.5 to 2.5.5 (SCO, *sigh*) The source machine errors out b/c the --remove-source-files (or --remove-sent-files) options don't exist on 2.5.5. But, since these option are only acted on by the sending machine (which is where I'm typing the command line), it would be so nice if the source machine would NOT pass the option on to the receiver (or at least, not barf when it doesn't work). Any way to do this? No, actually --remove-source-files is passed on to the receiver in order to get a confirmation when the receiver finishes with a file, so that the sender can delete the source file. Just hacking the code not to pass the option to the receiver would not work. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6307] rsync --remove-source-files foo/ foo/ should not delete files
https://bugzilla.samba.org/show_bug.cgi?id=6307 cool...@gmail.com changed: What|Removed |Added CC||cool...@gmail.com --- Comment #1 from cool...@gmail.com 2010-02-20 20:34 CST --- A big _me too_ on this one. Gee, my main storage space is nearly full, I need to move it off to another disk immediately... It's kinda important so I don't want to use mv and just hope everything goes okay... You know, if I use rsync, I'll get file checksuming, and some stats. Perfectly. Let me just wrap-off a quick cmd-line before we run out of space. Perfect! Hmm, where'd they go? Dammit! 200 GBytes of 'didn't have a chance to back it up yet' data GONE. Thanks rsync! While I recognize my mistake in all of this, and while important and irreplaceable, the data wasn't critical, but I remain in a mild furor over pretty much the one and only possible way this could go wrong, not being checked for, despite how trivial doing so would be... This is just a complete lack of design fore-thought on a critical-data tool. Combine this with the terrible performance I'm seeing on filesystems with large numbers of files/folders, and my days of promoting rsync are over. I guess it's time I really check out Unison... -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 3845] Make --remove-source-files remove directories too
https://bugzilla.samba.org/show_bug.cgi?id=3845 l...@kraav.com changed: What|Removed |Added CC||l...@kraav.com --- Comment #6 from l...@kraav.com 2009-10-18 16:05 CST --- registered a new account here for this one. really nobody has cared enough to implement it yet? # rsync --version rsync version 3.0.5 protocol version 30 -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Delay for --remove-source-files
On Tue, 2009-10-13 at 03:51 -0700, Martin Scharrer wrote: I'm using rsync with -aP --remove-source-files to move files from one machine to another while watching the progress. I'm under the impression that rsync is deleting the transmitted source files on-the-fly, not at the very end, but with a delay of 2-3 files, i.e. if 10 files are moved the first source file is deleted after the third of fourth file got transmitted. However, if rsync is aborted (CTRL+C) all fully source transmitted files are deleted. Can anyone tell me if this delay is intended behavior and if it can be manipulated/configured differently? My source machine has only limited space (10GB), my files are rather big (.4 - 1.2GB each) and my bandwidth around 300-500KB, so sometimes I want to get rid of the sent files as fast as possible without waiting for the next files being transmitted. IIUC, the delay occurs due to the pipelining in rsync; since it is not a problem for most users, no special effort was made to avoid it. If it is a problem for you, you might consider running rsync once per file. It sounds like the files are big enough that doing so wouldn't be unreasonably wasteful. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Delay for --remove-source-files
Matt McCutchen-7 wrote: On Tue, 2009-10-13 at 03:51 -0700, Martin Scharrer wrote: I'm using rsync with -aP --remove-source-files to move files from one machine to another while watching the progress. I'm under the impression that rsync is deleting the transmitted source files on-the-fly, not at the very end. [...] IIUC, the delay occurs due to the pipelining in rsync; since it is not a problem for most users, no special effort was made to avoid it. If it is a problem for you, you might consider running rsync once per file. It sounds like the files are big enough that doing so wouldn't be unreasonably wasteful. Thanks Matt for pointing this out. I will consider to change my rsync script. Martin -- View this message in context: http://www.nabble.com/Delay-for---remove-source-files-tp25870695p25908907.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Delay for --remove-source-files
Hi, I'm using rsync with -aP --remove-source-files to move files from one machine to another while watching the progress. I'm under the impression that rsync is deleting the transmitted source files on-the-fly, not at the very end, but with a delay of 2-3 files, i.e. if 10 files are moved the first source file is deleted after the third of fourth file got transmitted. However, if rsync is aborted (CTRL+C) all fully source transmitted files are deleted. Can anyone tell me if this delay is intended behavior and if it can be manipulated/configured differently? My source machine has only limited space (10GB), my files are rather big (.4 - 1.2GB each) and my bandwidth around 300-500KB, so sometimes I want to get rid of the sent files as fast as possible without waiting for the next files being transmitted. I couldn't find more information about this. The --delete* options are all for receiving side deletions, not for the source files. Thanks, Martin -- View this message in context: http://www.nabble.com/Delay-for---remove-source-files-tp25870695p25870695.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --remove-source-files
On Tue, Jun 02, 2009 at 04:08:26PM +, C?sar Tejeda Hern?ndez wrote:
rsync -tai --remove-source-files --ignore-existing --no-g --no-o --no-p
u...@source:operativo/almacen/{ESP05/recorte/*.tar,EU/recorte/*.tar}
/media/BACKUP3_PROMES10/PROMES10km/
[...]
¿Is there any way to specify that rsync should not report an error if it
finds no files to backup?
Your shell is probably including the raw wildcards when they don't match
anything, so you should be using include/exclude options instead of the
wildcards. For instance:
rsync -rti --remove-source-files --ignore-existing --include=*.tar --exclude=* \
u...@source:operativo/almacen/{ESP05,EU}/recorte/
/media/BACKUP3_PROMES10/PROMES10km/
..wayne..
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
remove-source-files checking for open files?
hi We have a box here connected to an antenna receiving rather large amounts of metheorological data from a satellite. the data is received and ransferred to another box and removed from the receiving server. I first thought of using rsync for this, but it seems --remove-source- files has no way of checking if the file is open or not, so if the receiving process is still writing to a file, rsync will transfer the file and unlink() it without bothering if the file has been modified any further, effectively resulting in a corrupt file. I solved this with a rather clumsy script doing md5sums for fils at both ends and checking if I have the exact same file at both sides some time after transfer. this works, but it's not really optimal. Does anyone know if rsync could be better on this somehow? roy -- Roy Sigurd Karlsbakk (+47) 97542685 / 98013356 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: remove-source-files checking for open files?
On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote: We have a box here connected to an antenna receiving rather large amounts of metheorological data from a satellite. the data is received and ransferred to another box and removed from the receiving server. I first thought of using rsync for this, but it seems --remove-source- files has no way of checking if the file is open or not, so if the receiving process is still writing to a file, rsync will transfer the file and unlink() it without bothering if the file has been modified any further, effectively resulting in a corrupt file. I solved this with a I would solve this not using rsync, but by writing the active file in a separate directory, or by first writing to a special filename and renaming when done (like firefox adds .part to the filename while it is still downloading it). Paul Slootman -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: remove-source-files checking for open files?
On 3. juni. 2009, at 12.34, Paul Slootman wrote: On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote: We have a box here connected to an antenna receiving rather large amounts of metheorological data from a satellite. the data is received and ransferred to another box and removed from the receiving server. I first thought of using rsync for this, but it seems --remove-source- files has no way of checking if the file is open or not, so if the receiving process is still writing to a file, rsync will transfer the file and unlink() it without bothering if the file has been modified any further, effectively resulting in a corrupt file. I solved this with a I would solve this not using rsync, but by writing the active file in a separate directory, or by first writing to a special filename and renaming when done (like firefox adds .part to the filename while it is still downloading it). Why not just something like --ignore-open-files? roy -- Roy Sigurd Karlsbakk (+47) 97542685 / 98013356 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: remove-source-files checking for open files?
On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote: I would solve this not using rsync, but by writing the active file in a separate directory, or by first writing to a special filename and renaming when done (like firefox adds .part to the filename while it is still downloading it). Why not just something like --ignore-open-files? Because checking whether a process has a file open can be very costly in system resources. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: remove-source-files checking for open files?
On 3. juni. 2009, at 18.16, Paul Slootman wrote: On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote: I would solve this not using rsync, but by writing the active file in a separate directory, or by first writing to a special filename and renaming when done (like firefox adds .part to the filename while it is still downloading it). Why not just something like --ignore-open-files? Because checking whether a process has a file open can be very costly in system resources. Not really. I'm writing some code doing this now. It will be Linux specific, but it will be fairly quick. Traversing /proc/pid/fd in search for symlinks to your file isn't really that bad. I'll post the code when I'm done with it. roy -- Roy Sigurd Karlsbakk (+47) 97542685 / 98013356 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: remove-source-files checking for open files?
On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote: On 3. juni. 2009, at 18.16, Paul Slootman wrote: Because checking whether a process has a file open can be very costly in system resources. Not really. I'm writing some code doing this now. It will be Linux Well, it's not something to do lightly... specific, but it will be fairly quick. Traversing /proc/pid/fd in search for symlinks to your file isn't really that bad. I'll post the code when This also only works if you own the process keeping the file open, or if you're the superuser. I'm done with it. The linux-specific thing is also a reason why I suspect it won't become part of rsync. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: remove-source-files checking for open files?
On 3. juni. 2009, at 19.31, Paul Slootman wrote: On Wed 03 Jun 2009, Roy Sigurd Karlsbakk wrote: On 3. juni. 2009, at 18.16, Paul Slootman wrote: Because checking whether a process has a file open can be very costly in system resources. Not really. I'm writing some code doing this now. It will be Linux Well, it's not something to do lightly... specific, but it will be fairly quick. Traversing /proc/pid/fd in search for symlinks to your file isn't really that bad. I'll post the code when This also only works if you own the process keeping the file open, or if you're the superuser. The code I'm writing only checks for its own UID unless UID == 0 I'm done with it. The linux-specific thing is also a reason why I suspect it won't become part of rsync. I guess others will be able to port the code to other platforms if it seems to work. Having more features for one OS and pending for others won't really hurt the project roy -- Roy Sigurd Karlsbakk (+47) 97542685 / 98013356 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--remove-source-files
Hello all,
I am using rsync version 3.0.3 protocol version 30, and I was
wondering if I could use --remove-source-files option in this fashion:
I use a crontab to dayly backup data,
something like:
rsync -tai --remove-source-files --ignore-existing --no-g --no-o --no-p
u...@source:operativo/almacen/{ESP05/recorte/*.tar,EU/recorte/*.tar}
/media/BACKUP3_PROMES10/PROMES10km/
and I check the output status to send me an e-mail if there is an error,
but I do not consider error the fact that maybe there are no new files to
backup.
¿Is there any way to specify that rsync should not report an error if it finds
no files to backup?
Thank you very much in advance,
César.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6307] New: --remove-source-files removes files if source and destination are the same
https://bugzilla.samba.org/show_bug.cgi?id=6307 Summary: --remove-source-files removes files if source and destination are the same Product: rsync Version: 3.0.5 Platform: x86 OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: way...@samba.org ReportedBy: cdauth+samba@cdauth.de QAContact: rsync...@samba.org I wanted to move the files of a directory (whose path was only slightly different from the current directory) to the current directory. Unfortunately, I errorneously passed the current directory as first parameter instead of the other one. The rsync call was rsync --append -avAHSP --remove-source-files /path/to/current/directory/ . The files in the current directory were deleted. I think rsync should check whether the source and the destination files are the same (like mv does, for instance) and in that case prevent the files from being deleted. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Exclude some source files from being deleted when --remove-source-files is used
Hi, I'm using rsync to transfer some files from host A to B and using --remove-source-files to remove them from A after the transfer. Now however I have one file 'md5sum' which should be transfered but not removed. I tried the protect filter rule but this seems not to work in this case. All other ways I tried don't work either. Does anyone know how to do this using a single rsync call? I know I could transfer the md5sum file first in a separate rsync operation and then exclude it from the second one, but I'm looking for a single-rsync-operation solution. Thanks, Martin -- View this message in context: http://www.nabble.com/Exclude-some-source-files-from-being-deleted-when---remove-source-files-is-used-tp21702836p21702836.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Exclude some source files from being deleted when --remove-source-files is used
On Wed, 2009-01-28 at 01:58 -0800, Martin Scharrer wrote: I'm using rsync to transfer some files from host A to B and using --remove-source-files to remove them from A after the transfer. Now however I have one file 'md5sum' which should be transfered but not removed. I tried the protect filter rule but this seems not to work in this case. All other ways I tried don't work either. Does anyone know how to do this using a single rsync call? I know I could transfer the md5sum file first in a separate rsync operation and then exclude it from the second one, but I'm looking for a single-rsync-operation solution. I don't believe there's a way to do it in a single rsync run. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hangs during --remove-source-files
Wayne Davison wrote: On Fri, Oct 17, 2008 at 03:34:03PM +0800, Thomas Gutzler wrote: I had another rsync hang. I've attached a text file with the stack traces you asked for and some other info. [...] So, the strange thing is that the receiver is waiting around for more files to process, which raises the question of why the generator doesn't know that the files it initiated are all done (or make me wonder if some pipe data didn't get delivered, but that is less likely). Did your transfer report any errors? No, there are no errors in the log file expect the one when I killed the process. I've just switched back to rsync version 2.6.9 - see if that works. Happy to test/debug stuff, you just have to tell me what to do. Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hangs during --remove-source-files
On Fri, Oct 17, 2008 at 03:34:03PM +0800, Thomas Gutzler wrote: I had another rsync hang. I've attached a text file with the stack traces you asked for and some other info. The generator backtrace shows that it is waiting for the receiver to finish some files before going on. This attempts to sprinkle the source removes more evenly during the transfer instead of having them all get bunched up at the end. So, the strange thing is that the receiver is waiting around for more files to process, which raises the question of why the generator doesn't know that the files it initiated are all done (or make me wonder if some pipe data didn't get delivered, but that is less likely). Did your transfer report any errors? ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hangs during --remove-source-files
Hi, I had another rsync hang. I've attached a text file with the stack traces you asked for and some other info. There were three rsync processes running at the time. I haven't heard anything from Wayne yet - maybe this generates more interest. Cheers, Tom Matt McCutchen wrote: On Mon, 2008-10-13 at 21:19 +0800, Thomas Gutzler wrote: I'm using the following command to move files from a to b: rsync -a --remove-source-files --ignore-existing /path/to/dir1/ /path/to/dir2 logfile dir2 isn't empty so I only move files from dir1 to dir2 that don't exist already. Every so often rsync stops in the middle and doesn't continue with the result that I have to kill it. Any suggestions how to debug this so that I can find out what's going on? Unfortunately I haven't found a way to reproduce it. Every time I kill and execute the same command again, it works. The next time rsync hangs, before you kill it, attach gdb to each of the three rsync processes (gdb /PATH/TO/rsync PID) and get a stack trace (bt). Wayne should have more suggestions for what information would be helpful for debugging. Matt [EMAIL PROTECTED]:~# ps ax | grep rsync 5249 ?S 0:00 sh -c /usr/local/rsync3/bin/rsync -a --remove-source-files --ignore-existing /home/backup/snapshots/frequent_2008-10-09_18-00-01/ /home/backup/snapshots/daily_2008-10-17_05-00-01 /home/backup/logs/frequent_2008-10-17_12-00-01.log 21 5250 ?S 0:02 /usr/local/rsync3/bin/rsync -a --remove-source-files --ignore-existing /home/backup/snapshots/frequent_2008-10-09_18-00-01/ /home/backup/snapshots/daily_2008-10-17_05-00-01 5251 ?S 0:00 /usr/local/rsync3/bin/rsync -a --remove-source-files --ignore-existing /home/backup/snapshots/frequent_2008-10-09_18-00-01/ /home/backup/snapshots/daily_2008-10-17_05-00-01 5252 ?S 0:02 /usr/local/rsync3/bin/rsync -a --remove-source-files --ignore-existing /home/backup/snapshots/frequent_2008-10-09_18-00-01/ /home/backup/snapshots/daily_2008-10-17_05-00-01 [EMAIL PROTECTED]:~# lsof | grep rsync | grep backup COMMAND PIDUSER FD TYPE DEVICESIZE NODE NAME rsync 5250root cwd DIR 254,04096 15290437 /home/backup/snapshots/frequent_2008-10-09_18-00-01 rsync 5250root1w REG 254,0 1258828 76689435 /home/backup/logs/frequent_2008-10-17_12-00-01.log rsync 5250root2w REG 254,0 1258828 76689435 /home/backup/logs/frequent_2008-10-17_12-00-01.log rsync 5251root cwd DIR 254,04096 15294467 /home/backup/snapshots/daily_2008-10-17_05-00-01 rsync 5251root2w REG 254,0 1258828 76689435 /home/backup/logs/frequent_2008-10-17_12-00-01.log rsync 5252root cwd DIR 254,04096 15294467 /home/backup/snapshots/daily_2008-10-17_05-00-01 rsync 5252root2w REG 254,0 1258828 76689435 /home/backup/logs/frequent_2008-10-17_12-00-01.log [EMAIL PROTECTED]:/home/backup/logs# tail -2 frequent_2008-10-17_12-00-01.log touch_up_dirs: home/io/obel/acuratolo/.winnt/profile/Cookies (20) set modtime of home/io/obel/acuratolo/.winnt/profile/Cookies to (1223546878) Thu Oct 9 18:07:58 2008 [EMAIL PROTECTED]:/home/backup/snapshots# ll -d daily_2008-10-17_05-00-01/home/io/obel/acuratolo/.winnt/profile/Cookies frequent_2008-10-09_18-00-01/home/io/obel/acuratolo/.winnt/profile/Cookies drwx-- 2 1161 1097 4096 2008-10-09 18:07 daily_2008-10-17_05-00-01/home/io/obel/acuratolo/.winnt/profile/Cookies/ drwx-- 2 1161 1097 4096 2008-10-17 12:15 frequent_2008-10-09_18-00-01/home/io/obel/acuratolo/.winnt/profile/Cookies/ [EMAIL PROTECTED]:~# gdb /usr/local/rsync3/bin/rsync 5250 GNU gdb 6.4.90-debian Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i486-linux-gnu...Using host libthread_db library /lib/tls/i686/cmov/libthread_db.so.1. Attaching to program: /usr/local/rsync3/bin/rsync, process 5250 Reading symbols from /lib/tls/i686/cmov/libc.so.6...done. Loaded symbols for /lib/tls/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/tls/i686/cmov/libnss_compat.so.2...done. Loaded symbols for /lib/tls/i686/cmov/libnss_compat.so.2 Reading symbols from /lib/tls/i686/cmov/libnsl.so.1...done. Loaded symbols for /lib/tls/i686/cmov/libnsl.so.1 Reading symbols from /lib/tls/i686/cmov/libnss_nis.so.2...done. Loaded symbols for /lib/tls/i686/cmov/libnss_nis.so.2 Reading symbols from /lib/tls/i686/cmov/libnss_files.so.2...done. Loaded symbols for /lib/tls/i686/cmov
rsync hangs during --remove-source-files
Hi, I'm using the following command to move files from a to b: rsync -a --remove-source-files --ignore-existing /path/to/dir1/ /path/to/dir2 logfile dir2 isn't empty so I only move files from dir1 to dir2 that don't exist already. Every so often rsync stops in the middle and doesn't continue with the result that I have to kill it. The last time it happened, rsync had successfully moved 1320 files, there were 2943 files in dir2 and 1101 files in dir1. lsof showed no open files inside dir1 or dir2, the log file was opened 4 times, dir1 itself showed up and dir2 twice. This is on a dual core CPU with rsync version 3.0.4 protocol version 30. The log file shows no errors. Last 4 lines say: renaming dir/res/.svn/text-base/.somefile.svn-base.mDBdUb to dir/res/.svn/text-base/somefile.svn-base sender removed dir/.svn/text-base/somefile.svn-base touch_up_dirs: dir/.svn/prop-base (84) set modtime of dir/res/.svn/prop-base to (1223212308) Sun Oct 5 21:11:48 2008 Any suggestions how to debug this so that I can find out what's going on? Unfortunately I haven't found a way to reproduce it. Every time I kill and execute the same command again, it works. Cheers, Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hangs during --remove-source-files
On Mon, 2008-10-13 at 21:19 +0800, Thomas Gutzler wrote: I'm using the following command to move files from a to b: rsync -a --remove-source-files --ignore-existing /path/to/dir1/ /path/to/dir2 logfile dir2 isn't empty so I only move files from dir1 to dir2 that don't exist already. Every so often rsync stops in the middle and doesn't continue with the result that I have to kill it. Any suggestions how to debug this so that I can find out what's going on? Unfortunately I haven't found a way to reproduce it. Every time I kill and execute the same command again, it works. The next time rsync hangs, before you kill it, attach gdb to each of the three rsync processes (gdb /PATH/TO/rsync PID) and get a stack trace (bt). Wayne should have more suggestions for what information would be helpful for debugging. Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5327] New: rsync hangs when using --remove-source-files with a bunch of files
https://bugzilla.samba.org/show_bug.cgi?id=5327 Summary: rsync hangs when using --remove-source-files with a bunch of files Product: rsync Version: 3.0.0 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] as reported by David Leverton in Gentoo bug 193243: mkdir remove-src-files-tst cd remove-src-files-tst mkdir src dst for i in `seq -w 0 99`; do touch src/$i; done rsync --recursive --remove-source-files --dry-run --verbose src/ dst/ it gets up to 49 and then hangs tested with rsync-2.6.9 and rsync-3.0.0 with same behavior -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5327] rsync hangs when using --remove-source-files with a bunch of files
https://bugzilla.samba.org/show_bug.cgi?id=5327 --- Comment #1 from [EMAIL PROTECTED] 2008-03-16 09:00 CST --- Created an attachment (id=3175) -- (https://bugzilla.samba.org/attachment.cgi?id=3175action=view) Fix a hang when using --remove-source-files and --dry-run This patch is for the development version, but is also relevant for 2.6.9. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5327] rsync hangs when using --remove-source-files with a bunch of files
https://bugzilla.samba.org/show_bug.cgi?id=5327 [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED --- Comment #2 from [EMAIL PROTECTED] 2008-03-16 09:01 CST --- This is now fixed in the git repository. Thanks for the report! -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 3845] Make --remove-source-files remove directories too
https://bugzilla.samba.org/show_bug.cgi?id=3845 [EMAIL PROTECTED] changed: What|Removed |Added CC||[EMAIL PROTECTED] --- Comment #5 from [EMAIL PROTECTED] 2008-03-10 04:45 CST --- What's the status of this feature? The directory deletion feature is still missing. I'm on 2.6.9 but I couldn't find changes in the Changelog of 3.0. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using --remove-source-files with backup?
On 5/7/07, ScottZ [EMAIL PROTECTED] wrote: With --source-backup the tree structure of the original source file is also being included in the --backup-dir directory. That is an idiosyncracy of the quick-and-dirty way I implemented the source backup, and I documented it in the patched man page. To avoid it, cd into the source directory first so you can give the source argument as . or just a filename. When Wayne made a modified version of the source-backup patch to include with HEAD of rsync, he found it more convenient to follow the backup-dir path starting from the source dir rather than the source argument path starting from the backup dir. It's not clear to me whether either of these behaviors is the best possible. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using --remove-source-files with backup?
On 5/6/07, Matt McCutchen [EMAIL PROTECTED] wrote:
The attached patch to rsync 2.6.9 adds a --source-backup
option that backs up removed source files.
Oops, I made a typo in the addition to the man page. The fixed patch
is attached.
Matt
This patch to rsync 2.6.9 adds a --source-backup option that backs up source
files removed due to --remove-source-files. In my limited testing, it seems
to work.
-- Matt McCutchen [EMAIL PROTECTED]
--- old/options.c
+++ new/options.c
@@ -30,6 +30,7 @@ extern struct filter_list_struct filter_
extern struct filter_list_struct server_filter_list;
int make_backups = 0;
+int make_source_backups = 0;
/**
* If 1, send the whole file as literal data rather than trying to
@@ -324,6 +325,7 @@ void usage(enum logcode F)
rprintf(F, --existing skip creating new files on receiver\n);
rprintf(F, --ignore-existing skip updating files that already exist on receiver\n);
rprintf(F, --remove-source-files sender removes synchronized files (non-dirs)\n);
+ rprintf(F, --source-backup ... and backs up those files\n);
rprintf(F, --del an alias for --delete-during\n);
rprintf(F, --deletedelete extraneous files from destination dirs\n);
rprintf(F, --delete-before receiver deletes before transfer (default)\n);
@@ -517,6 +519,7 @@ static struct poptOption long_options[]
{itemize-changes, 'i', POPT_ARG_NONE, 0, 'i', 0, 0 },
{bwlimit, 0, POPT_ARG_INT,bwlimit, 0, 0, 0 },
{backup, 'b', POPT_ARG_NONE, make_backups, 0, 0, 0 },
+ {source-backup,0, POPT_ARG_NONE, make_source_backups, 0, 0, 0},
{backup-dir, 0, POPT_ARG_STRING, backup_dir, 0, 0, 0 },
{suffix, 0, POPT_ARG_STRING, backup_suffix, 0, 0, 0 },
{list-only,0, POPT_ARG_VAL,list_only, 2, 0, 0 },
@@ -1655,6 +1658,9 @@ void server_options(char **args,int *arg
args[ac++] = arg;
}
+ if (make_source_backups !am_sender)
+ args[ac++] = --source-backup;
+
if (am_sender) {
if (delete_excluded)
args[ac++] = --delete-excluded;
--- old/rsync.yo
+++ new/rsync.yo
@@ -341,6 +341,7 @@ to the detailed description below for a
--existing skip creating new files on receiver
--ignore-existing skip updating files that exist on receiver
--remove-source-files sender removes synchronized files (non-dir)
+ --source-backup ... and backs up those files
--del an alias for --delete-during
--deletedelete extraneous files from dest dirs
--delete-before receiver deletes before transfer (default)
@@ -929,6 +930,15 @@ dit(bf(--remove-source-files)) This tell
side the files (meaning non-directories) that are a part of the transfer
and have been successfully duplicated on the receiving side.
+dit(bf(--source-backup)) Makes the sender back up the source files it removes
+due to bf(--remove-source-files). This option is independent of
+bf(--backup) but uses the same bf(--backup-dir) and bf(--suffix) settings,
+if any. With bf(--backup-dir), each backup file is placed inside the backup
+dir according to the source file's full path from the working directory
+(source argument path + file-list path); if you want files placed according
+to the file-list path, you could either make appropriate symlinks or have the
+sender cd into the source directory so that the source argument is just ..
+
dit(bf(--delete)) This tells rsync to delete extraneous files from the
receiving side (ones that aren't on the sending side), but only for the
directories that are being synchronized. You must have asked rsync to
--- old/sender.c
+++ new/sender.c
@@ -37,6 +37,7 @@ extern int protocol_version;
extern int remove_source_files;
extern int updating_basis_file;
extern int make_backups;
+extern int make_source_backups;
extern int do_progress;
extern int inplace;
extern int batch_fd;
@@ -123,6 +124,7 @@ void successful_send(int ndx)
char fname[MAXPATHLEN];
struct file_struct *file;
unsigned int offset;
+ int result;
if (ndx 0 || ndx = the_file_list-count)
return;
@@ -135,7 +137,11 @@ void successful_send(int ndx)
offset = 0;
f_name(file, fname + offset);
if (remove_source_files) {
- if (do_unlink(fname) == 0) {
+ if (make_source_backups)
+ result = !make_backup(fname);
+ else
+ result = do_unlink(fname);
+ if (result == 0) {
if (verbose 1)
rprintf(FINFO, sender removed %s\n, fname + offset);
} else
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Using --remove-source-files with backup?
Many, Many Thanks for this Matt! Seems to be working great so far. Just one question though. My initial tests with --backup-dir and -b had just the file being backed up to the specified dir. With --source-backup the tree structure of the original source file is also being included in the --backup-dir directory. example: rsync --remove-source-files --source-backup --backup-dir=/arc -av /home/dir/file remotehost:/dir/file results in the backup file being put in /arc/home/dir/file instead of /arc/file. It isn't a problem for me as my tree structure is limited to just a couple of directories, but could be a problem for those with a deeper one. Or am I not specifying something correctly? Again, many thanks for this. -Scott Original Message Subject: Re: Using --remove-source-files with backup? From: Matt McCutchen [EMAIL PROTECTED] Date: Sun, May 06, 2007 6:38 am To: ScottZ [EMAIL PROTECTED] Cc: rsync@lists.samba.org On 5/6/07, Matt McCutchen [EMAIL PROTECTED] wrote: The attached patch to rsync 2.6.9 adds a --source-backup option that backs up removed source files. Oops, I made a typo in the addition to the man page. The fixed patch is attached. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Using --remove-source-files with backup?
Essentially you want to move each source file to another directory after it is successfully transferred. Exactly. I'm sure there's a way to accomplish this, but it might not be simple. I'll tell you if I find one. Thanks for looking at this Matt. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Using --remove-source-files with backup?
I have a scenario where I am copying from a local system to remote target and need to remove the local source files once they have been successfully transfered. --remove-source-files works fine for this. An issue has come up in that I need to be able to backup the removed local source files as well. With rsync 2.6.9 I tried: rsync --remove-source-files --backup-dir=/archive abv (source) (remotehost:target) The result was that the backup files were made in /archive on the remotehost system and not the local system (where the original files were removed). Is this a bug or is the backup option only able to backup deleted target files? Is there another way around this? Note: As a WAG, I tried using a myhost:/archive (where myhost is my source host) type of entry for --backup-dir, but that just created a myhost: directory on the remotehost:target directory. Like I said, it was a WAG. Thanks, -Scott -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using --remove-source-files with backup?
On 5/4/07, ScottZ [EMAIL PROTECTED] wrote: Is this a bug or is the backup option only able to backup deleted target files? The latter: --backup only backs up files deleted from the target. Is there another way around this? Essentially you want to move each source file to another directory after it is successfully transferred. I'm sure there's a way to accomplish this, but it might not be simple. I'll tell you if I find one. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 3845] Make --remove-source-files remove directories too
https://bugzilla.samba.org/show_bug.cgi?id=3845 [EMAIL PROTECTED] changed: What|Removed |Added Summary|Add --remove-source-files |Make --remove-source-files ||remove directories too --- Comment #4 from [EMAIL PROTECTED] 2007-02-21 19:13 MST --- A --remove-source-files option was added in 2.6.9, but it does not remove source directories, and I would like it to remove them so that rsync --remove-source-files can be used as a mv replacement. Specifically, I would like the sender to do the following: (1) Remove each empty source directory as soon as the generator processes it. (For this to work, the generator would need to send MSG_SUCCESS after processing a directory with no contents in the file list; maybe the --prune-empty-dirs code could be reused to flag such directories.) (2) After removing each source file, check whether the parent directory is empty and in the file list. If so, remove it and recursively consider /its/ parent for removal. I would be satisfied if a separate option or repetition of --remove-source-files activated this behavior. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 3845] Add --remove-source-files
https://bugzilla.samba.org/show_bug.cgi?id=3845 --- Comment #3 from [EMAIL PROTECTED] 2006-06-30 11:03 MST --- Created an attachment (id=2010) -- (https://bugzilla.samba.org/attachment.cgi?id=2010action=view) Add --remove-source-files option Here is a patch that adds the --remove-source-files option and deprecates the --remove-sent-files option. I've also checked this into CVS, so it is also available from there or the latest nightly tar file. The code is young, but should be safe as most of the functionality was taken from remove-sent-files. I've run basic tests on it, and it works fine. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 3845] Add --remove-source-files
https://bugzilla.samba.org/show_bug.cgi?id=3845 --- Comment #2 from [EMAIL PROTECTED] 2006-06-27 22:59 MST --- Is there a possible ETA for this feature? It would nicely solve the problem I have with files getting left over after a GPRS link goes down in mid-transfer. My current, and rather gross, solution is to summarily delete everything in the transfer directory after rsync completes successfully. --jc (In reply to comment #0) Let's have a --remove-source-files option that works like --remove-sent-files but removes from the source everything in the file list, not just transferred files. Then I could use rsync for my primary move command (via a wrapper called mv2) as well as my primary copy command (cp2). -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 3845] New: Add --remove-source-files
https://bugzilla.samba.org/show_bug.cgi?id=3845 Summary: Add --remove-source-files Product: rsync Version: 2.6.9 Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] Let's have a --remove-source-files option that works like --remove-sent-files but removes from the source everything in the file list, not just transferred files. Then I could use rsync for my primary move command (via a wrapper called mv2) as well as my primary copy command (cp2). -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
