Re: Bug when using rsync -r --delete -b --backup-dir
You may wish to take a look at LBackup : http://www.lbackup.orgPlease note as one of the developers I am biased.On 24/11/2020, at 10:33 AM, CRTS via rsync wrote:Consider following directory structure:mkdir -p src/empty tgt bkpwhen making an initial backup withrsync -r src/ tgtthen the empty directory in src will be copied to tgt. However, if thedirectory is removed from srcrmdir src/emptyand rsync is run likersync -r --delete -b --backup-dir=../bkp src/ tgt/then the empty directory does end up in bkp. I am using the --backup-dirand the --delete option to move an outdated snapshot to another directoryby backing up an empty directory. I was expecting that the entiredirectory structure to be preserved. Apparently this is not the casefor empty directories.Can this be fixed? Is this even the right place to file a bug report? Ido not have a github account to do this on github.-- Please use reply-all for most replies to avoid omitting the mailing list.To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsyncBefore posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: RAM speedup
You may need a lot of ram. Take a look at how much RAM ZFS needs to operate well. Henri > On 27/06/2020, at 7:58 AM, Rupert Gallagher via rsync > wrote: > > Hello, > > As disks are slow and rsync reads and writes so much that for the bus this is > the equivalent of context switching galore, would it be possible to use RAM > as a buffer? Say, you have 10GB of spare RAM, rsync uses the bus to its peak > for reading 10GB, then again for writing it down. This would be more > efficient than lot of small read/write operations. > > Thank you > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: RAM speedup
Not sure if it is useful. Will put a plug in for a rsync hard based backup system primarily focused on macOS. If it is speed you are after, then this is probably not the right / helpful tool. But feel free to pick out what ever is useful for you : http://www.lbackup.org Henri > On 29/06/2020, at 7:29 AM, Matthias Schniedermeyer via rsync > wrote: > > On 28.06.2020 16:46, Rupert Gallagher wrote: >> ? Original Message ? >> On Sunday 28 June 2020 13:58, Matthias Schniedermeyer wrote: >> >> destination: >> ST5000LM000-2AN1 sata hdd >> Writing speed : 74 MB/s >> Reading speed : 89 MB/s > > And this HDD is a SMR model(*) on top of beeing a 4k sector model emulating > 512 byte sectors. > So alignment needs to be correct and the filesystems must use 4k sectors. > > This HDD is NOT suitable for beeing used for small files and a hardlink-farm. > > SMR HDDs only reach best performance if used "like a tape drive" with large > and linear writing. > > I use SMR HDDs myself and only use them for "really large" files (>500MB per > file), otherwise they perform very poorly. > I also align my partitions correctly and set XFS to "4k sector"-size. > Which nowadaya means: I use 4k sectors for anything. As i only have been > using 4k sector HDDs for nearly as long as they are on the market (IIRC >1 > decade). > Also SSDs are usually optimized for 4k sectors too. > > > *: > https://en.wikipedia.org/wiki/Shingled_magnetic_recording > > > -- > > Matthias > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx permission issue
If you are talking about SIP, then at present SSH will have full disk access (as much as it can) by default. Take a look at this LBackup page which discusses this in more detail : http://www.lbackup.org/developer/dealing_with_sip Hope that helps. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Advanced rsync includes and excludes
Hello, http://www.lbackup.org/developer/advanced_excludes_file_examples The link above may outline a slightly different approach which may work on your system. However, you would need that approach assumes the backup path is set to root which may not be what you want to do. For quick inline reference the excludes file example is included below. You should be able to modify this to make it work for you. However, as mentioned you will need to test it out on your system and see if it is doing what you expect. > /private/etc/ssh_config/ > + private > + private/etc/*** > + private/var/*** > + home/*** > - * Henri > I'm a happy camper @ rsync (and rsnapshot) since years. Thanks for this major > piece of software. > > In an attempt to reorganize my rsnapshot backups, I stumbled across an issue, > that I'm trying to seeking a more sophisticated solution here. > > Given, I have a deeply branched tree, where I would like to include a > specific > directory deep under, while excluding anything else on that path, I find > myself doing: > > + /some/ > + /some/very/ > + /some/very/deep/ > + /some/very/deep/path/ > + /some/very/deep/path/to/ > + /some/very/deep/path/to/save/ > - /some/very/deep/path/to/* > - /some/very/deep/path/* > - /some/very/deep/* > - /some/very/* > - /some/* > > While it works, it feels rather awkward and gets very complicated, if you > have > more of such items to deal with. Sure, I could run this separately, but this > isn't the real McCoy either with a complicated tree structure. > > What are your favorite ways of doing such tasks with rsync? > > Thanks in advance, > Pete > > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
I think others have answered your questions. You may want to implement some sort of locking system in the script so that you can only run a single instance of your script at. If you are looking for a backup tool which leverages rsync you may want to take a look at LBackup : http://www.lbackup.org There is even a pre-script you can enable (in the example resources) so that the script can run a maximum of once a day / once a month etc (even if you start it many times from cron. Hope you have it all working now :) Henri >> Hi all! :-) >> >> I have a small rsync script to sync my data to a usb-disk. It works >> fine, when I start it in console. I get 3 rsync processes (look in >> top) and the backup takes ~25 min. But, when I add the script in >> cron to start it at 1am at night it takes 7 - 9 hours and I see up to >> 180 processes. When I look in top I see a hight load of 60 - 80 and 40 >> - 60 waits. Why? Can someone explain why it takes so long when it >> starts with cron? >> >> This is my rsync command: rsync -azc --delete "$QUELLORDNER" >> "$ZIELORDNER" >> This is the entry in cron (crontab -e): * 2 * * * >> /root/backupscript/backup.sh >> Data to sync: 18 Gb, 185.000 files. >> >> When I look in the log files I see errors like this: >> >> rsync: >> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" >> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) >> >> rsync error: some files/attrs were not transferred (see previousrsync >> error: >> >> some files/attrs were not transferred (see previous errors) (code 23) at >> >> main.c(1196) [sender=3.1.2] >> >> directory (2) >> >> >> >> Can you help me to solve this problem? >> >> >> >> regards, >> >> Hannes Hutmacher >> >> > A couple of questions: (I am not an expert user.) > > Does your manual job run as the same user (presumably root) as does your > cron job? > > Are you backing up any temporary files that you might be better off > excluding? > > Is anything else running at the same time as your cron job which may be > creating and destroying files in the backup source or target? (E.g. Your > firewall?) > > I'm not very familiar with fuzzy searching, but using that in places > like /var seems odd. > > One of the constant refrains on this list is that using checksums is > almost never a good idea. In combination with fuzzy, it seems even more > tenuous. > > Joe > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
