Not removing source files when using --remove-source-files
I have rsync -rvz --prune-empty-dirs --remove-source-files --log-file="rsync.log" --backup -e 'ssh' -f'- Saved/*' -f'- Screenshots/*' -f'- Boondocks/*' -f'- Dilbert/*' -f'+ */' -f'+ /*' -f'+ *.jp*g' -f'+ *.png' -f'+ *.nef' -f'+ *.gif' -f'- *' linuxuser@10.0.0.11:/home/linuxuser/Pictures/ /mnt/c/users/windowsuser/ The files are transferred but they are not removed from the source. I've deleted all files and directories from the target and tried again but still, it didn't remove the source. ~ In all things, Be Intentional. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is there a better way to transfer data that doesn't use so much cache?
I've decided to rewrite the script and use cp and mv rather than rsync. In the past, I've had some lost data using just cp and mv which is why I moved to rsync to put the data into a staging directory. Now that I've been creating more data (newer cameras with higher megapixel files and more files), rsync doesn't work as well as it used to. Trying to get nocache or something similar to work seemed like it would take more time than to rewrite the script. Thanks all for your assistance and suggestions. ~ In all things, Be Intentional. On Fri, Aug 5, 2022 at 1:22 AM Wayne Davison via rsync < rsync@lists.samba.org> wrote: > On Wed, Aug 3, 2022 at 7:10 PM Dan Stromberg wrote: > >> However, if you transfer a large amount of data and do not intend to >> retransmit that data any time soon, then the memory isn't really put to >> good use, and can actually cause your system to slow down significantly - >> particularly if there's a lot of such data transferred. >> > > I have always rejected overcomplicating rsync with cache control code (the > complexity of a --drop-cache patch I saw was quite horrifying). In the > past I pointed people towards https://github.com/Feh/nocache as one way > to get posix_fadvise used by an rsync copy. That project now apparently > suggests creating a memory-bounded cgroup, which sounds interesting. > > ..wayne.. > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Is there a better way to transfer data that doesn't use so much cache?
I've created a script that syncs (and removes) data from as many as 4 places and puts them all in one of 2 directories. The commands are: rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef' -f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *' "$D850/DCIM/100ND850/" $STAGINGP/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef' -f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *' "$Z9/DCIM/100NCZ_9/" $STAGINGP/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'+ /*' -f'- *' "$DASHCAM/CARDV/VIDEO/" $STAGINGV/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'- Screenshots/' -f'+ *.nef' -f'+ *.jpg' -f'+ *.jp*g' -f'+ *.png' -f'+ *.dng' -f'+ *.gif' -f'- *.thumbnails' -f'- *.android' -f'+ */' -f'+ DCIM/*' -f'+ Snapbridge/*' -f'+ Pictures/*' -f'+ Download/*' -f'+ Textgram/*' -f'- *' $PHONE/ $STAGINGP/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'- *' --files-from=<(find $PHONE -type f ! -path "*Download*" ! -path "*.trashed*" ! -iname .mp4 ! -iname '*.mp4\.*')/ $STAGINGV/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'+ Movies/*' -f'+ *Recordings/*' -f'+ DCIM/*' -f'+ Snapbridge/*' -f'- */' -f'- *' $PHONE/ $STAGINGV/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'- *' --files-from=<(find $PHONE -iname .mp4) / $STAGINGV/TIKTOK/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *' $PHONE/Downloads/ $COMPUTER/Downloads/ rsync -avt --progress --remove-source-files --info=progress2 -f'- screenshot*' -'f- Screenshot*' -f'- Boondocks/' -f'- Dilbert/' -f'+ *.png' -f'+ *.jp*g' -f'+ *.dng' -f'+ *.gif' -f'- *20*/' -f'- *' -f'+ */' -f'- $STAGINGP/' $MYPICS/ $STAGINGP/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ Screenshot*.png' -f'- Staging/' -f'- *' $MYPICS/ $STAGINGP/Screenshots/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.3gpp' -f'+ *.mp4' -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'- *' $HOME/Downloads $STAGINGV/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'+ *.3gpp' -f'- *' $MYVIDEOS/ $STAGINGV/ | tee -a $LOG The problem isn't that there are many syncs because the problem happens on the first one that runs. Before any of them run I run: sudo free -w -h;sync && echo 1 > /proc/sys/vm/drop_caches;free -w -h I do not run this before each one because it sometimes takes a while to /proc/sys/vm/drop_caches Is there something in the logic that can be done to make this perform better or should I use something other than rsync or is what I am getting as good as it will get regardless of what I use? Some of these directories can be over a gig. Most of these are media files and should have exif data that has the timestamp so maybe I can get rid of -t but it is easier to keep the timestamp of the file rather than running exiftool to also use the create date to "touch" the file but maybe using exiftool is a faster way? ~ In all things, Be Intentional. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Syncing multiple sub-directories to one directory
After more testing, I found that that command only syncs files that are in
that top dir. It doesn't recurse through all the dirs.
~
When you are principled, set standards and stick to them some people will
fall out of your life; let them!
On Sun, Oct 4, 2020 at 6:20 PM Rob Campbell
wrote:
> My deepest apologies. I had an inotify running that anytime something was
> added to that directory it would run a script that was doing my original
> sync of pulling in all the directories and images. After turning that off,
> I was able to see your line work as you said it should. Thank you very
> much and again, I apologize for mistakenly stating the code you provided
> didn't work.
>
> ~
> When you are principled, set standards and stick to them some people will
> fall out of your life; let them!
>
>
> On Wed, Sep 30, 2020 at 10:46 PM Rob Campbell
> wrote:
>
>> Except some of those dir have subdir such as WhatsApp and DCIM has
>> multiple subdirs too. I would rather do it all with rsync though.
>>
>> On Wed, Sep 30, 2020, 12:54 PM Wayne Davison wrote:
>>
>>> On Tue, Sep 29, 2020 at 4:46 PM Rob Campbell wrote:
>>>
Thanks for your help. What you provided didn't work for me because
that still placed things in subdirectories.
>>>
>>> Ah, that's what you were trying to do. Your original email sounded like
>>> you just didn't want it to recurse into subdirectories.
>>>
>>> An easier solution than what you ended up with is to specify a trailing
>>> slash after the dir names since that tells rsync you want to copy the
>>> content of a directory rather than the directory by name:
>>>
>>> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
>>> '*.dng' --include '*.raw' --include '*.nef' --exclude '*'
>>> /my/phone/root/dir/{Duo,DCIM,WhatsApp}/ /my/backup/directory/for/images/
>>>
>>> ..wayne..
>>>
>>
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Syncing multiple sub-directories to one directory
My deepest apologies. I had an inotify running that anytime something was
added to that directory it would run a script that was doing my original
sync of pulling in all the directories and images. After turning that off,
I was able to see your line work as you said it should. Thank you very
much and again, I apologize for mistakenly stating the code you provided
didn't work.
~
When you are principled, set standards and stick to them some people will
fall out of your life; let them!
On Wed, Sep 30, 2020 at 10:46 PM Rob Campbell
wrote:
> Except some of those dir have subdir such as WhatsApp and DCIM has
> multiple subdirs too. I would rather do it all with rsync though.
>
> On Wed, Sep 30, 2020, 12:54 PM Wayne Davison wrote:
>
>> On Tue, Sep 29, 2020 at 4:46 PM Rob Campbell wrote:
>>
>>> Thanks for your help. What you provided didn't work for me because that
>>> still placed things in subdirectories.
>>>
>>
>> Ah, that's what you were trying to do. Your original email sounded like
>> you just didn't want it to recurse into subdirectories.
>>
>> An easier solution than what you ended up with is to specify a trailing
>> slash after the dir names since that tells rsync you want to copy the
>> content of a directory rather than the directory by name:
>>
>> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
>> '*.dng' --include '*.raw' --include '*.nef' --exclude '*'
>> /my/phone/root/dir/{Duo,DCIM,WhatsApp}/ /my/backup/directory/for/images/
>>
>> ..wayne..
>>
>
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Syncing multiple sub-directories to one directory
Except some of those dir have subdir such as WhatsApp and DCIM has multiple
subdirs too. I would rather do it all with rsync though.
On Wed, Sep 30, 2020, 12:54 PM Wayne Davison wrote:
> On Tue, Sep 29, 2020 at 4:46 PM Rob Campbell wrote:
>
>> Thanks for your help. What you provided didn't work for me because that
>> still placed things in subdirectories.
>>
>
> Ah, that's what you were trying to do. Your original email sounded like
> you just didn't want it to recurse into subdirectories.
>
> An easier solution than what you ended up with is to specify a trailing
> slash after the dir names since that tells rsync you want to copy the
> content of a directory rather than the directory by name:
>
> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
> '*.dng' --include '*.raw' --include '*.nef' --exclude '*'
> /my/phone/root/dir/{Duo,DCIM,WhatsApp}/ /my/backup/directory/for/images/
>
> ..wayne..
>
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Syncing multiple sub-directories to one directory
Thanks for your help. What you provided didn't work for me because that
still placed things in subdirectories.
I figured it out. This puts it all in the images directory.
find /my/phone/root/dir/ \( -path '*Duo*' -o -path '*DCIM*' -o -path
'*Pictures*' -o -path '*Camera*' -o -path "*Download*" -o -path
"*textgram*" -o -path "*WhatsApp*" ! -path '*.textgram*' ! -path
'*.thumbnails*' \) -type f \( -iname '*.jp*g' -o -iname '*.png' -o -iname
'*.dng' -o -iname '*.raw' -o -iname '*.raw' \) -exec rsync -uz --progress
"{}" /my/backup/directory/for/images/ \;
~
When you are principled, set standards and stick to them some people will
fall out of your life; let them!
On Tue, Sep 29, 2020 at 1:33 PM Wayne Davison wrote:
> On Tue, Sep 29, 2020 at 7:38 AM Rob Campbell wrote:
>
>> I would like to sync many subdirectories into one directory with no
>> subdirectories. I've tried
>>
>> rsync -rv --progress --include '*.jp*g' --include '*.png' --include
>> '*.dng' --include '*.raw' --include '*.nef' --include 'Duo' --include
>> 'DCIM' --include 'WhatsApp' --exclude '*' /my/phone/root/dir/
>> /my/backup/directory/for/images/
>>
>
> You didn't specify what isn't working as you expect. That command works
> fine for the 3 listed dir names as long as you either don't have matching
> files inside /my/phone/root/dir (or as long as it's ok to also copy those
> files) and as long as there aren't subdirs or files named the same as the 3
> top dirs you included. The latter can be fixed by anchoring your dir names
> and making them only match a dir (e.g. --include '/Duo/'). If you want to
> have the copy avoid files in the top dir you can either change all the
> file-based includes to have a "*/" prefix (such as "--include '*/*.png'
> ...") or you can change the dir includes into args and exclude '*/*'
> instead of "*". The last option would look like this (I also tossed in -i):
>
> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
> '*.dng' --include '*.raw' --include '*.nef' --exclude '*/*'
> /my/phone/root/dir/{Duo,DCIM,WhatsApp} /my/backup/directory/for/images/
>
> That assumes you've got bash to do the brace expansion, but you could
> change that into 3 arg paths if you need to. You may also want to add --del
> if you want rsync to delete inside the 3 listed dirs.
>
> I'm not sure why you listed an extra command with "NewDir" when it's not
> mentioned in the first command. If that is an indication that you really
> want to copy all dirs under root/dir (not just the 3 named dirs in the
> first command) then you could use an include of "/*/" to match any dir in
> the root of the transfer, like this:
>
> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
> '*.dng' --include '*.raw' --include '*.nef' --include '/*/' --exclude '*'
> /my/phone/root/dir/ /my/backup/directory/for/images/
>
> One last suggestion, I like to make the args shorter by using -f (filter)
> commands, so an include example is -f '+ *.png' and an exclude example is
> -f '- /*/' (those are identical to the equivalent include/exclude args, so
> that's just personal preference).
>
> ..wayne..
>
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Syncing multiple sub-directories to one directory
I would like to sync many subdirectories into one directory with no subdirectories. I've tried rsync -rv --progress --include '*.jp*g' --include '*.png' --include '*.dng' --include '*.raw' --include '*.nef' --include 'Duo' --include 'DCIM' --include 'WhatsApp' --exclude '*' /my/phone/root/dir/ /my/backup/directory/for/images/ I'd like to do this w/o separate rsync commands such as rsync -rv --progress --include '*.jp*g' --include '*.png' --include '*.dng' --include '*.raw' --include '*.nef' --exclude '*' /my/phone/root/dir/DCIM /my/backup/directory/for/images/ rsync -rv --progress --include '*.jp*g' --include '*.png' --include '*.dng' --include '*.raw' --include '*.nef' --exclude '*' /my/phone/root/dir/Duo /my/backup/directory/for/images/ rsync -rv --progress --include '*.jp*g' --include '*.png' --include '*.dng' --include '*.raw' --include '*.nef' --exclude '*' /my/phone/root/dir/WhatsApp /my/backup/directory/for/images/ -- This has subdirectories rsync -rv --progress --include '*.jp*g' --include '*.png' --include '*.dng' --include '*.raw' --include '*.nef' --exclude '*' /my/phone/root/dir/NewDir /my/backup/directory/for/images/ ~ When you are principled, set standards and stick to them some people will fall out of your life; let them! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
