rsync error messages always in English (= never translated)?
I want to recognize and handle some rsync error messages in my log files (containing also the --itemize-changes output) on different computers with different language/locale settings. Can I rely on rsync to create only English error messages to have a stable pattern to recognize? PS: In the source code at https://github.com/WayneD/rsync I could not see any indication of message translations... -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?
On Thu, 2023-12-14 at 14:09 -0500, Kevin Korb wrote:
> Unfortunately, exit 23 litterally just means something else went wrong
> and might have scrolled off of the screen if you have rsync listing
> files (--verbose or --itemize_changes). Essentially, it is anything
> that doesn't have its own exit code. I just ignore it. The most common
> reaosn is file vanished.
THX for sharing your experiences and knowledge :-)
I have just tried to "reverse engineer" the possible reasons from the source
code
and have found 21 reasons that I hope will never happen ;-)
So ignoring (or treating as a warning only) sound as best option so far.
-
Looking into the rsync source code I can see only one location where exit code
23 is set:
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/errcode.h#L42
#define RERR_PARTIAL23 /* partial transfer */
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/log.c#L97
{ RERR_PARTIAL, "some files/attrs were not transferred (see previous
errors)" },
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/cleanup.c#L217-L218
if (io_error & IOERR_GENERAL || got_xfer_error)
exit_code = RERR_PARTIAL;
So the question is which reasons cause
- IOERR_GENERAL
- got_xfer_error
to be true?
IOERR_GENERAL is set for different reasons (first line is the log output format
string):
1. receive_sums failed [what is that at all?]:
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L345-L348
2. send_files failed to open %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L358-L362
3. fstat failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L373C33-L373C45
4. read errors mapping %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L433-L436
5. change_dir %s failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L366-L369
6. skipping overly long name: %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1245-L1247
7. symlink has no referent: %s
See the source code comments there when symlinks are checked:
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1271C1-L1282C28
8. readlink_stat(%s) failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1293-L1295
9. skipping file with bogus (zero) st_mode: %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1298-L1302
10. skipping symlink with 0-length value: %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1565-L1566
11. [%s] cannot convert filename: %s (%s)
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1595-L1601
12. [%s] cannot convert symlink data for: %s (%s)
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1609-L1614
13. get_acl(fname, ) < 0 // with no explicit error message!
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1628-L1632
14. get_xattr(fname, ) < 0 // with no explicit error message!
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1637-L1642
15. link_stat %s failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1809-L1811
16. opendir %s failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1841-L1843
17. filename overflows max-path len by %u: %s/%s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1863-L1871
18. cannot send file with empty name in %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1873-L1877
19. readdir(%s)
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1886-L1889
20. link_stat %s failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L2396-L2399
21. cannot add local filter rules in long-named directory: %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/exclude.c#L815-L818
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?
I am trying to find a solution for the open source Linux software "Back In Time" (https://github.com/bit-team/backintime) where we evaluate the rsync exit code when taking a backup via rsync and inform the user that an error has occured. Questions: 1. Is there full list of possible reasons available that make rsync exit with the return value 23? 'rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]' I want to decide for each reason if we treat it as an error or warning. 2. I want to ignore some reasons for exit code 23 by only logging it, mainly: symlink has no referent: "/home/user/Documents/dead-link" Is there a way to - either prevent that this error leads to exit code 23 (if no other reason occurs) - or to prevent this specific check at all (but eg. copy the symlink "as is")? THX a lot! PS 1: The typcially used rsync command line looks like this: rsync --recursive --times --devices --specials --hard-links --human-readable -s --copy-links --acls --perms --executability --group --owner -- info=progress2 --no-inc-recursive -l --delete --delete-excluded -v -i --out-format=BACKINTIME: %i %n%L --link-dest=../../20231214-143359-584/backup -- chmod=Du+wx --exclude=/home/username/temp/testBAK_profil1 --exclude=/home/username/.local/share/backintime --exclude=.local/share/backintime/mnt -- include=/home/username/Documents/ --include=/home/username/ --include=/home/ --include=/home/username/temp/deleted_folder/ -- include=/home/username/temp/ --include=/home/username/.mozilla/ --exclude=.gvfs --exclude=.cache/* --exclude=.thumbnails* -- exclude=.local/share/[Tt]rash* --exclude=*.backup* --exclude=*~ --exclude=.dropbox* --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* -- exclude=/run/* --exclude=/etc/mtab --exclude=/var/cache/apt/archives/*.deb --exclude=lost+found/* --exclude=/tmp/* --exclude=/var/tmp/* -- exclude=/var/backups/* --exclude=.Private --exclude=lock --exclude=root_only_file.txt --include=/home/username/Documents/** -- include=/home/username/temp/deleted_folder/** --include=/home/username/.mozilla/** --exclude=* / /home/username/temp/testBAK_profil1/backintime/computer1/username/1/new_snapshot/backup PS 2: You can find more details or even answer in our Github issue too: https://github.com/bit-team/backintime/issues/1587 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
Here is the missing attachment ;-)
On Fri, 2023-09-22 at 21:01 +0200, rsync--- via rsync wrote:
> On Fri, 2023-09-22 at 07:37 -0400, Kevin Korb wrote:
> > So I decided to do a quick test using the Linux kernel source tree since
> > it has lots of files.
>
> Excellent idea using kernel sources! A lot of different files...
> I will use this to create indicative benchmarks for different scenarios...
>
> > I duplicated a tree, used 'find . -type f -exec
> > chmod 444 {} +' to make read only files for rsync to want to chmod, then
> > used cp -al to make several duplicate trees using hard linked files.
> > [...]
> > But also, I did not experience the problem you are describing. My
> > surviving
> > hard links in the duplicate trees were still 444.
>
> If attached a script that does the same (with one file instead of the kernel
> source tree)
> but in my case 444 became 644 again.
>
> - Q: Which rsync version, distro and file system do you use?
>
> - Q: Does my attached script reproduce the permission change?
>
> - Q: Did my script attached to the initial post here reproduce the permission
> change?
>
>
>
> PS: In my case the attached script shows (excerpt):
>
> ./setup_cp_al.sh
>
> Tested with
> - rsync version 3.2.7 protocol version 31
> - ext4 file system
> - Ubuntu 22.04
>
> File stats BEFORE rsync --delete:
>
> File: snapshot2/read_only.txt
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 10305h/66309d Inode: 17571021 Links: 3
> Access: (0444/-r--r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
> Access: 2023-09-22 20:51:16.690961150 +0200
> Modify: 2023-09-22 20:51:16.690961150 +0200
> Change: 2023-09-22 20:51:16.694961109 +0200
> Birth: 2023-09-22 20:51:16.690961150 +0200
>
> File stats AFTER rsync --delete
>
> File: snapshot2/read_only.txt
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 10305h/66309d Inode: 17571021 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
> Access: 2023-09-22 20:51:16.690961150 +0200
> Modify: 2023-09-22 20:51:16.690961150 +0200
> Change: 2023-09-22 20:51:16.694961109 +0200
> Birth: 2023-09-22 20:51:16.690961150 +0200
>
> Results (after deleting snapshot1 via rsync --delete):
> 1. The 'Change' time of the hardlinked file was updated
> 2. The hardlinked file has now different access rights (644 instead of 444 so
> it is writable again!)
> This does NOT happen if 'rm -f snapshot1/' is used!
>
>
setup_cp_al.sh
Description: application/shellscript
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
On Fri, 2023-09-22 at 07:37 -0400, Kevin Korb wrote:
> So I decided to do a quick test using the Linux kernel source tree since
> it has lots of files.
Excellent idea using kernel sources! A lot of different files...
I will use this to create indicative benchmarks for different scenarios...
> I duplicated a tree, used 'find . -type f -exec
> chmod 444 {} +' to make read only files for rsync to want to chmod, then
> used cp -al to make several duplicate trees using hard linked files.
> [...]
> But also, I did not experience the problem you are describing. My surviving
> hard links in the duplicate trees were still 444.
If attached a script that does the same (with one file instead of the kernel
source tree)
but in my case 444 became 644 again.
- Q: Which rsync version, distro and file system do you use?
- Q: Does my attached script reproduce the permission change?
- Q: Did my script attached to the initial post here reproduce the permission
change?
PS: In my case the attached script shows (excerpt):
./setup_cp_al.sh
Tested with
- rsync version 3.2.7 protocol version 31
- ext4 file system
- Ubuntu 22.04
File stats BEFORE rsync --delete:
File: snapshot2/read_only.txt
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 10305h/66309d Inode: 17571021Links: 3
Access: (0444/-r--r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
Access: 2023-09-22 20:51:16.690961150 +0200
Modify: 2023-09-22 20:51:16.690961150 +0200
Change: 2023-09-22 20:51:16.694961109 +0200
Birth: 2023-09-22 20:51:16.690961150 +0200
File stats AFTER rsync --delete
File: snapshot2/read_only.txt
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 10305h/66309d Inode: 17571021Links: 2
Access: (0644/-rw-r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
Access: 2023-09-22 20:51:16.690961150 +0200
Modify: 2023-09-22 20:51:16.690961150 +0200
Change: 2023-09-22 20:51:16.694961109 +0200
Birth: 2023-09-22 20:51:16.690961150 +0200
Results (after deleting snapshot1 via rsync --delete):
1. The 'Change' time of the hardlinked file was updated
2. The hardlinked file has now different access rights (644 instead of 444 so
it is writable again!)
This does NOT happen if 'rm -f snapshot1/' is used!
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote: > I have heard in the past that rsyncing an empty dir over a tree to > delete the tree is faster than an rm -rf but I can't say I have ever > benchmarked it to get any actual numbers. This **may** indeed be a myth (for a long time now) re-cited again and again and - could no longer be valid today - could apply only when deleting explicitly named files but not deleting the complete folder (as we need to do in "Back in Time") At least I could not find a holistic benchmark with many files and different scenarios (file systems, rsync'ing locally vs. over network, snapshot sizes, number of files, file sizes, rsync and rm versions...) Q: Does `rsync` provide a test case that I could use as basis to prepare such a holistic benchmark? > But now that I am hearing > that rsync actually adds a bunch of pointless chmods to the process. Is > it still faster given this problem? If so maybe we should be trying to > investigate why rm is so slow. Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
Context
---
I am one of the active developers of the open source application "Back in Time"
which uses "rsync" as backend and I want to fix an open issue:
"Back in Time"-Bug:
https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507
"Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking
unchanged files in the backups/snapshots and tries to keep the permissions
by using "--perms --group --owner" by default.
Older snapshots are then deleted according to a schedule.
Issue
-
When deleting a complete snapshot folder with rsync using an empty source folder
(which is a best practice for faster deletion than "rm -f") the permissions of
hardlinked
files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted
files
with existing hardlinks:
mkdir empty
rsync -a --delete -s empty/ snapshot1/
This distorts the backup history.
There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the
issue is still unfixed:
https://bugzilla.samba.org/show_bug.cgi?id=12806
Questions
-
1. Is there a reliable workaround ("--super" is proposed in the issue but may
probably not always work)?
2. If a rsync developer is reading this: Is there any chance to fix this?
Steps to reproduce
--
I have attached a bash script "setup.sh" as txt file which makes the issue
fully reproducible (on Ubuntu 22.04).
#!/bin/bash
clear
echo
echo "Minimal reproducible example (MRE)"
echo "--"
echo
echo "Tested with"
echo "- rsync version 3.2.7 protocol version 31"
echo "- ext4 file system"
echo
rm -rf MRE
mkdir MRE
cd MRE
mkdir source
touch source/read_only.txt
chmod 444 source/read_only.txt
mkdir snapshot1
# rsync --recursive --times --devices --specials --hard-links --human-readable
-s --links --perms --executability --group --owner source/* snapshot1/
rsync --perms --executability --group --owner source/* snapshot1/
mkdir snapshot2
# rsync --recursive --times --devices --specials --hard-links --human-readable
-s --links --perms --executability --group --owner --link-dest=../snapshot1
source/* snapshot2/
rsync --perms --executability --group --owner --link-dest=../snapshot1 source/*
snapshot2/
# stat snapshot1/read_only.txt
echo "File stats BEFORE rsync --delete":
echo
stat snapshot2/read_only.txt
mkdir empty
# delete snapshot1 using an empty source directory (shall be the fastest way to
delete a subdir)
# --debug=ALL
rsync -a --delete -s empty/ snapshot1/
# rm -f snapshot1/ # works as expected (unlink does neither change 'Changed'
time nor file permissions)
# strace -c rm -rf snapshot1
echo
stat snapshot2/read_only.txt
echo
echo "Results (after deleting snapshot1 via rsync --delete):"
echo "1. The 'Change' time of the hardlinked file was updated"
echo "2. The hardlinked file has now different access rights (644 instead of
444 so it is writable again!)"
echo "This does NOT happen if 'rm -f snapshot1/' is used!"
echo
echo "Files in snapshot2:"
ls -l snapshot2
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Encrypt destination file
Rainy Days via rsync wrote: > Hi, > > My company have been using rsync to sync our file to a backup server (running > rsync daemon), and now we would like to store them encrypted. > I found some thread that discussed this issue but they are pretty old (about > 10 years ago). So I send this question again and looking for any new approach. > Is there any option or module for rsync to encrypt file on the destination? > > Thanks a lot, > Dat Le Hi, If the destination is Linux, you could maybe use ecryptfs to encrypt a section of the file system. Everything would still appear decrypted as long as the ecryptfs file system is mounted. But if you wanted it to look encrypted, then maybe you could mount it, rsync, then unmount it. Then it would at least appear encrypted most of the time (when rsync is not actively in use). But this sounds dumb. For rsync to do this automatically, it would need to be able to automatically decrypt files in memory at the destination in order to compare them to the source version which means having access to the decryption key at the destination. An option to run a process at the other end to filter each file for comparison would be needed. That would slow things down. Another option is to have full disk encryption at the destination. I know that this (and ecryptfs) still make unencrypted files accessible while the host is running, and so may not satisfy the needs of your threat model, but it does seem to satisfy the encryption-at-rest requirements of some governments for some purposes. Another approach (assuming Linux and ecryptfs), is to use ecryptfs at the source to store everything encrypted, use a mountpoint for the ecryptfs file system that isn't the same as the underlying directory so that the encrypted versions of the files are accessible (which they wouldn't be if the ecryptfs file system were mounted over the top of the underlying directory), and just rsync the underlying encrypted directory rather than the overlaid ecryptfs file system (where everything looks decrypted). This approach doesn't require any changes to rsync and it doesn't require the ability to decrypt the files on the destination host. But it does require Linux and ecryptfs at both the source and the destination. cheers, raf -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
Hallo Hannes "man 5 crontab" is your friend ;-) Am 12.07.2019 um 10:56 schrieb Hannes Hutmacher via rsync: But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron? This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh Data to sync: 18 Gb, 185.000 files. wrong crontab entry: every minute you start a new rsync process ... crontab syntax:|*1 2 3 4 5 command* field allowed values - -- 1 minute 0-59 2 hour 0-23 3 day of month 1-31 4 month 1-12 (or names, see below) 5 day of week 0-7 (0 or 7 is Sun, or use names) command = command to be run Good luck -- Beat | -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
