I've created a script that syncs (and removes) data from as many as 4 places and puts them all in one of 2 directories. The commands are:
rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef' -f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *' "$D850/DCIM/100ND850/" $STAGINGP/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef' -f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *' "$Z9/DCIM/100NCZ_9/" $STAGINGP/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'+ /*' -f'- *' "$DASHCAM/CARDV/VIDEO/" $STAGINGV/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'- Screenshots/' -f'+ *.nef' -f'+ *.jpg' -f'+ *.jp*g' -f'+ *.png' -f'+ *.dng' -f'+ *.gif' -f'- *.thumbnails' -f'- *.android' -f'+ */' -f'+ DCIM/*' -f'+ Snapbridge/*' -f'+ Pictures/*' -f'+ Download/*' -f'+ Textgram/*' -f'- *' $PHONE/ $STAGINGP/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'- *' --files-from=<(find $PHONE -type f ! -path "*Download*" ! -path "*.trashed*" ! -iname ????????????????????????????????.mp4 ! -iname '*.mp4\.*')/ $STAGINGV/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'+ Movies/*' -f'+ *Recordings/*' -f'+ DCIM/*' -f'+ Snapbridge/*' -f'- */' -f'- *' $PHONE/ $STAGINGV/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'- *' --files-from=<(find $PHONE -iname ????????????????????????????????.mp4) / $STAGINGV/TIKTOK/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *' $PHONE/Downloads/ $COMPUTER/Downloads/ rsync -avt --progress --remove-source-files --info=progress2 -f'- screenshot*' -'f- Screenshot*' -f'- Boondocks/' -f'- Dilbert/' -f'+ *.png' -f'+ *.jp*g' -f'+ *.dng' -f'+ *.gif' -f'- *20*/' -f'- *' -f'+ */' -f'- $STAGINGP/' $MYPICS/ $STAGINGP/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ Screenshot*.png' -f'- Staging/' -f'- *' $MYPICS/ $STAGINGP/Screenshots/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.3gpp' -f'+ *.mp4' -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'- *' $HOME/Downloads $STAGINGV/ | tee -a $LOG rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4' -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'+ *.3gpp' -f'- *' $MYVIDEOS/ $STAGINGV/ | tee -a $LOG The problem isn't that there are many syncs because the problem happens on the first one that runs. Before any of them run I run: sudo free -w -h;sync && echo 1 > /proc/sys/vm/drop_caches;free -w -h I do not run this before each one because it sometimes takes a while to /proc/sys/vm/drop_caches Is there something in the logic that can be done to make this perform better or should I use something other than rsync or is what I am getting as good as it will get regardless of what I use? Some of these directories can be over a gig. Most of these are media files and should have exif data that has the timestamp so maybe I can get rid of -t but it is easier to keep the timestamp of the file rather than running exiftool to also use the create date to "touch" the file but maybe using exiftool is a faster way? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In all things, Be Intentional.
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html