Re: Rsync 3.2.5pre1 released

[Charles via rsync]

On 02/08/2022 08:11, Wayne Davison wrote:
I have released rsync version 3.2.5pre1 for release testing. This 
includes a checksum fix that affects some architectures (such as ARM), 
improves the security of the received file list, enhances the manpage, etc.


I'd really appreciate it if people would give this release some 
extensive testing and see if the improved security results in any false 
alerts. I did a bunch of testing and found a few options that caused 
a problem (and were then fixed), but there may be more.  You can use a 
side-effect of the --old-args option 
 to work 
around an invalid complaint, but please also report it.  Thanks!


To see a summary of all the recent changes, visit this link:

https://rsync.samba.org/ftp/rsync/NEWS#3.2.5 



You can download the source tar file and its signature from here:

https://download.samba.org/pub/rsync/src-previews/rsync-3.2.5pre1.tar.gz 

https://download.samba.org/pub/rsync/src-previews/rsync-3.2.5pre1.tar.gz.asc 



See the website for other downloads, including diffs, patches, etc.:

https://rsync.samba.org/ 

See also rsync on github:

https://github.com/WayneD/rsync 

..wayne..


Thanks Wayne

For those considering testing 3.2.5pre1, the build procedure is at 
https://github.com/WayneD/rsync/blob/master/INSTALL.md


Charles


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

https://rsync.samba.org/resources.html

[Charles via rsync]

Regards "Please let us know if you have any rsync-related documents to 
add to this list"


From the bung introduction (https://github.com/CharlesMAtkinson/bung): 
"The rsync-based files backup creates a "rolling full" backup which is 
easy to browse and restore from using everyday tools. Changed and 
deleted files are kept for a configurable number of days"


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync dest high disk write request avg waiting time (w_await) and probable low speed

[Charles via rsync]

Is it likely that rsync is the cause of the symptoms?

If so, how and can the rsync command be changed to prevent the problem?

While running this command ...

rsync --filter=protect _Changed and deleted files/ --exclude=/_Changed 
and deleted files/ --backup --backup-dir=_Changed and deleted 
files/2021/Sep/13@01:20 --compress --delete-during --delete-excluded 
--devices --group --links --owner --numeric-ids --perms --timeout=600 
--protect-args --recursive --specials --times --verbose 
/home/vbox/VirtualBox VMs/aems3.iciti.av/ 
vbox2_to_backup.sar:/srv/backup/vms.vbox2.iciti.av


... Zabbix reported reported high disk write request avg waiting time 
(w_await) values of ~40 ms


The VirtualBox VM was shut down during the sync so the source files were 
static.  There were seven files to be synced, six trivially small and 
one 38 GB.  The sync took ~0.5 hours and summarised

sent 1,009,799,482 bytes  received 2,428,622 bytes  636,021.43 bytes/sec
total size is 39,736,701,202  speedup is 39.26

Investigation showed the high w_await values were associated with rsync 
rather than the HDD


1. Moved the destination to a different HDD.  Same

2. Re-scheduled to a different time.  Same

3. Wrote to the HDD locally
dd if=/dev/zero of=/srv/backup/vms.vbox2.iciti.av/trash
That wrote at ~130 MB/s and gave w_await 56 ms

4. Wrote to the HDD from the source computer
scp -p '/home/vbox/VirtualBox VMs/aems3.iciti.av/aems3.iciti.av.vdi' \
storage4.iciti.av:/srv/backup/vms.vbox2.iciti.av/trash
... 
   100% 
  37GB  76.5MB/s   08:15

...
That wrote at ~76 MB/s and gave w_await 62 ms

5. Experimented with --bwlimit
Progressively reducing from 50M to 5M made no significant difference to 
the time taken to sync (stayed around 20 to 30 m) or w_await (stayed 
around 40 ms).
Only reducing from 5M to 3M made a significant difference  to the time 
taken to sync (60 m) while w_await stayed around 40 ms.



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Utility of --backup

[Charles via rsync]

> And let me do what you suggest.  What is the difference, and is --backup
> better than the other ?

Only in conjunction with -backup-dir, for example

--backup --backup-dir=_Changed and deleted files/2021/Jul/19@21:21

The man page's entry for --backup suggests using it with either 
--backup-dir or --suffix


On 19/07/2021 19:22, lisa-as...@perso.be wrote:

From: Charles via rsync 
To: rsync@lists.samba.org
Subject: Re: Utility of --backup
Date: 19/07/2021 14:26:59 Europe/Paris

 >IThe --backup option is great for creating "rolling full" backups which
 >look exactly like the backed up tree except for the existence of the
 >backup directory

I am not really understanding the "Rolling Full Backup".  Suppose I have 
a directory

and use `rsync -av --progress --log-file="$logfl" "$source" "$destin"`

And let me do what you suggest.  What is the difference, and is --backup 
better than the other ?


Would the command be

rsync --backup -v --progress --log-file="$logfl" "$source" "$destin"

 >Here's how a Linux backup directory tree looks as created by backup
 >utility bung's bu_rsync script

+-- bin -> usr/bin
+-- boot
|   +-- grub
+-- _Changed and deleted files
|   +-- 2021
|   +-- Jul
|   |   +-- 01@17:45
|   |   |   +-- opt
|   |   |   |   +-- tomcat
|   |   |   +-- root
|   |   |   +-- var
|   |   |   +-- backups
|   |   |   +-- cache
|   |   |   +-- lib
|   |   |   +-- local
|   |   |   +-- log
|   |   |   +-- mail
|   |   |   +-- spool
...
|   |   +-- 17@17:46
...
|   |   +-- 18@17:45
...
|   +-- Jun
...
|   +-- 29@17:45
...
|   +-- 30@17:45
...
+-- dev
+-- etc
...

 >A "rolling full" backup is great to restore from for small organisations
 >which do not do enough restores to be well practised because the backup
 >looks exactly like the source except for the additional "_Changed and
 >deleted files" directory.

 >Perfect point in time restores are not possible but adequate
 >approximations (point in time but with the possibility of some extra
 >files) can be done by restoring the last backup and then each of the
 >changed and deleted files sets until the latest set after the desired
 >point in time



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Utility of --backup

[Charles via rsync]

IThe --backup option is great for creating "rolling full" backups which 
look exactly like the backed up tree except for the existence of the 
backup directory


Here's how a Linux backup directory tree looks as created by backup 
utility bung's bu_rsync script


+-- bin -> usr/bin
+-- boot
|   +-- grub
+-- _Changed and deleted files
|   +-- 2021
|   +-- Jul
|   |   +-- 01@17:45
|   |   |   +-- opt
|   |   |   |   +-- tomcat
|   |   |   +-- root
|   |   |   +-- var
|   |   |   +-- backups
|   |   |   +-- cache
|   |   |   +-- lib
|   |   |   +-- local
|   |   |   +-- log
|   |   |   +-- mail
|   |   |   +-- spool
...
|   |   +-- 17@17:46
...
|   |   +-- 18@17:45
...
|   +-- Jun
...
|   +-- 29@17:45
...
|   +-- 30@17:45
...
+-- dev
+-- etc
...

A "rolling full" backup is great to restore from for small organisations 
which do not do enough restores to be well practised because the backup 
looks exactly like the source except for the additional "_Changed and 
deleted files" directory.


Perfect point in time restores are not possible but adequate 
approximations (point in time but with the possibility of some extra 
files) can be done by restoring the last backup and then each of the 
changed and deleted files sets until the latest set after the desired 
point in time


On 19/07/2021 17:30, rsync-requ...@lists.samba.org wrote:

I would like some feedback about the --backup option in rsync. Is
it worth using it for backups, or should I just use rsync
commands that just transfer files without the use of --backup
option?

-b, --backup  make backups (see --suffix & --backup-dir)
--backup-dir=DIR  make backups into hierarchy based in DIR
--suffix=SUFFIX   backup suffix (default ~ w/o --backup-dir)

I am somewhat hesitant to use it because with the backup option,
preexisting destination files are renamed as each file is
transferred or deleted. It also says that previously backed-up
files could get deleted.  Thusly I need some assistance
understanding all the pros and cons.


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Submission for https://rsync.samba.org/resources.html: bung

[Charles via rsync]

Backup scripts next generation (bung) is a suite of bash scripts that 
runs standard backup utilities such as mysqldump, pg_dump, rsync, 
slapcat and tar


bung adds configurable support for hotplug storage devices (such as USB 
disks), LVM snapshots, file system mounting, logging, on-screen 
notifications and report emails.
bung backup jobs are normally started by scheduler (anacron or cron) or 
by udev (by plugging in backup storage devices such as USB HDDs).
bung is arbitrarily extensible via templates with integrated support for 
git.  Templated examples are provided for backing up some Cisco and 
MikroTek devices.


Documentation: https://redmine.auroville.org.in/projects/bung/documents
Files: https://redmine.auroville.org.in/projects/bung/files

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html