Re: Everything working as expected, so shouldn't ERROR be WARNING

[Paul Slootman via rsync]

On Thu 18 Jan 2024, Roger Price via rsync wrote:

> I am backing up a user's directories from local machine titan to remote
> machine maria. On the remote machine maria file /etc/rsyncd.conf contains
> 
>  [rprice-home]
>  ...
>  exclude = *.dvi
> 
> I start the backup by using this command on the local machine titan:
> 
>  rprice@titan ~ rsync -av --dry-run /mnt/home/rprice 
> rsync://rprice@maria/rprice-home
> 
> I get the messages
> 
>  sending incremental file list
>  ERROR: daemon refused to receive file "rprice/demo.dvi"
>  ...
> 
> I understand that the remote daemon has refused file demo.dvi because I
> specifically requested that dvi files not be transferred.  I choose that
> myself in a regular configuration file.  So shouldn't it be a WARNING rather
> than an ERROR?  I would expect to see

In this case you're in control of both ends of the transfer, so you know
that *.dvi files won't be transferred.

However, it could be that this rsync command is being run by someone who
expects rsync to do what they asked it to do, i.e. transfer the entire
contents of .../rprice to the remote server, and the client rsync can't
fulfil that request; hence the error.

If you don't want *.dvi files to be transferred, then you should add
--exclude '*.dvi'
to the invocation.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[Paul Slootman via rsync]

On Fri 22 Sep 2023, Kevin Korb via rsync wrote:

> 444 {} +' to make read only files for rsync to want to chmod, then used cp
> -al to make several duplicate trees using hard linked files.  An rm -rf on
> one such tree took .97 seconds while an rsync deletion took 1.25 seconds.

Be sure to drop the caches before such tests every time:
echo 3 > /proc/sys/vm/drop_caches


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Why try to update (some) permissions which are the same?

[Paul Slootman via rsync]

On Sun 03 Sep 2023, Perry Hutchison via rsync wrote:

> On the source system:
> 
> $ rsync --version
> rsync  version 2.6.8  protocol version 29

> On the destination system:

> $ rsync --version
> rsync  version 3.0.7  protocol version 30

The current version is 3.2.7, especially 2.6.8 is quite ancient.
You may want to upgrade before going bug hunting, chances are your
problem has already been fixed.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: What could cause rsync to kill ssh?

[Paul Slootman via rsync]

On Sat 03 Jun 2023, Maurice R Volaski via rsync wrote:

> I have an rsync script that it is copying one computer (over ssh) to a shared 
> CIFS mount on Gentoo Linux, kernel 6.3.4. The script runs for a while and 
> then at some point quits knocking my ssh session offline on all terminals and 
> it blocks ssh from being able to connect again. Even restarting sshd doesn’t 
> help. Rsync has apparently killed it. I have to reboot.

Note there's no such thing as an rsync script. You probably mean you
have a shell script that runs rsync at some point.

Is the script copying from the system it's running on, to the Gentoo
Linux system? Is the CIFS mount actually mounted on the Gentoo Linux, or
is the Gentoo Linux system serving the CIFS mount which actually is
mounted on the "one computer"? In that case it would be much better to
directly rsync to the filesystem on the Gentoo system.

Re: the ssh stopping working:
To me this would suggest that there's an out-of-memory situation going
on, and sshd is being killed because of this. However that would not
explain why restarting it doesn't work.
What exactly do you mean when you say restarting sshd doesn't help?
Does it not stay running, or is the daemon in fact running but not
accepting connections?
It's the age-old question: "it doesn't work" -- "_how_ is it not working?"

Does dmesg give any useful information? Or perhaps journalctl?
Usually the clues are in plain signt if you check logs.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Permission denied errors

[Paul Slootman via rsync]

On Mon 14 Nov 2022, Wes Render via rsync wrote:

> Hello, I'm running an rsync like this:
> 
> rsync -avSHP --delete-after 
> --log-file=/opt/mirrorsync/centos_mirror/rsync-1.log 
> --exclude-from=/opt/mirrorsync/centos_mirror/excludelist.txt 
> rsync://centos.mirror.rafal.ca/CentOS/ /data/centos
> 
> I'm running as a user that has full permissions on /data/ and /data/centos, 
> but then I get the following errors, when the rsync runs the second time:
> 
> 2022/11/14 10:13:54 [68738] receiving file list
> 2022/11/14 10:13:56 [68738] rsync: opendir 
> "/7.9.2009/cloud/x86_64/openstack-train/repodata/.~tmp~" (in CentOS) failed: 
> Permission denied (13)
> 2022/11/14 10:13:56 [68738] rsync: opendir 
> "/7.9.2009/updates/x86_64/repodata/.~tmp~" (in CentOS) failed: Permission 
> denied (13)
> 2022/11/14 10:13:56 [68738] 198081 files to consider
> 2022/11/14 10:13:57 [68751] sent 62 bytes  received 15,071,081 bytes  
> 6,028,457.20 bytes/sec
> 2022/11/14 10:13:57 [68751] total size is 149,535,538,669  speedup is 9,921.98
> 2022/11/14 10:13:57 [68738] rsync error: some files/attrs were not 
> transferred (see previous errors) (code 23) at main.c(1819) [generator=3.2.3]
> 
> The strange thing is, if I login as this user, and do an: ls -la 
> /data/centos//7.9.2009/cloud/x86_64/openstack-train/repodata/.~tmp~

The error message is saying it can't open that file on the source;
testing access on your local (destination) system is irrelevant.

$ rsync 
'rsync://centos.mirror.rafal.ca/CentOS/7.9.2009/updates/x86_64/repodata/.~tmp~'
drwx--  4,096 2022/11/10 22:39:21 .~tmp~

$ rsync 
'rsync://centos.mirror.rafal.ca/CentOS/7.9.2009/updates/x86_64/repodata/.~tmp~/'
rsync: change_dir "/7.9.2009/updates/x86_64/repodata/.~tmp~" (in CentOS) 
failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 
23) at main.c(1819) [Receiver=3.2.3]


You might want to --exclude '.~tmp~'


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --daemon complains parsing nonexistant /etc/rsyncd.conf

[Paul Slootman via rsync]

On Sat 17 Sep 2022, Colton Lewis via rsync wrote:

> This is on a system where /etc/rsyncd.conf does not exist and goes away if 
> /etc/rsyncd.conf is an empty file.
> 
> Version: rsync  version 3.2.5  protocol version 31
> Command: rsync --daemon
> What happens: The program outputs "Failed to parse config file: 
> /etc/rsyncd.conf"
> What I expect: The program should run with the same default values it does 
> when /etc/rsyncd.conf is an empty file.

So what functionality does rsync as a daemon have if there is an empty
rsyncd.conf file?

IMHO rsync is correct in refusing to run with a missing rsyncd.conf.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Does rsync verify its writes?

[Paul Slootman via rsync]

On Tue 12 Jul 2022, Kevin Korb via rsync wrote:

> Rsync does not verify writes.  --checksum doesn't verify anything. Sounds
> like you want a file verification tool.  The simplest would be md5sum.

Running rsync --checksum directly after transferring your files will
verify that the files are written correctly (if the source hasn't
changed in the meantime). That might help to give some peace of mind.


Paul

> On 7/12/22 02:31, Mark Filipak via rsync wrote:
> > Hello. Does rsync verify its writes?
> > 
> > Re, 'info rsync'.
> > 
> > Maybe I just being stupid, but there's no mention of verification in the
> > 'DESCRIPTION' section, so despite the words in the 'OPTIONS' section,
> > '-c, --checksum' topic (which I may be misinterpreting), I assume rsync
> > does not verify except for the checksum directive.
> > 
> > I admit that I'm paranoid. ;-) ...Please clarify.
> > 
> > Regards, and Thanks,
> > Mark Filipak.
> > 
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   https://sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync Users and Groups

[Paul Slootman via rsync]

On Fri 24 Jun 2022, Kevin Korb via rsync wrote:

> Nope.  Rsync groups are not groups of users they are just @users with their
> own password.  I believe the @ just designates that you intend multiple
> people to have that password and use that username.

I think I have to disagree here.

The manpage for rsyncd.conf states:

   In addition to username matching, you can specify groupname matching
   via a '@' prefix. When using groupname matching, the authenticating
   username must be a real user on the system, or it will be assumed to
   be a member of no groups. For example, specifying "@rsync" will match
   the authenticating user if the named user is a member of the rsync
   group.

So the user used by the client rsync should exist on the system and
belong to the specified system group for any permissions that are given
to that group in rsyncd.conf to be applicable. The password will still
need to be set in the secrets file, no system passwords are used.

Check the rsyncd.conf manpage, heading "auth users".


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: exclude include path problems

[Paul Slootman via rsync]

On Mon 07 Feb 2022, Edwardo Garcia via rsync wrote:
> 
> Lets call module mirror
> this results in about 200 or more directories (projects), but I only want
> one of them, lets call it foo, the problem is foo/  has about 50
> directories, but the one and only one we want is bar, but bar also has
> dozens of directories - but we do want these ones.
> 
> so  we want server:mirrors/foo/bar/*
> 
> How do we place our rsync command?
> 
> we have tried all kinds of   --include=foo/bar/  --exclude=*
> servername:mirrors/
> we tried an include exclude include exclude  too, infact every conceivable

Why not just do:

rsync -a server:mirrors/foo/bar/ /local/path/to/foo/bar/


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Confused as to why rsync thinks time, owner and group of many files differ

[Paul Slootman via rsync]

On Thu 03 Feb 2022, Andy Smith via rsync wrote:

> sudo rsync -iPva \
> --inplace \
> --numeric-ids \
> --delete \
> /data/backup/rsnapshot/daily.0/cacti/ \
> root@koff:/data/backup/rsnapshot/daily.0/cacti/
> 
> ...
>5,258 100%5.78kB/s0:00:00 (xfr#1276, to-chk=1/43437)

Could you try the transfer like this?:

sudo rsync -ia \
--debug=OWN,TIME \
--inplace \
--numeric-ids \
--delete \
/data/backup/rsnapshot/daily.0/cacti/var/www/index.html \
root@koff:/data/backup/rsnapshot/daily.0/cacti/var/www/

That should give detailed information about ownership and modification
times, limiting the transfer to just that index.html file to limit the
amount of output.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: trailing spaces in exclude-from file

[Paul Slootman via rsync]

On Mon 24 Jan 2022, Jürgen Bausa via rsync wrote:
> 
> However, that's just a proposal. But the behavior of trailing spaces is 
> something I guess should be corrected.

Is it?
How would you otherwise specify a space that you *do* intend to be
relevant?


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to manage root<-->root rsync keeping permissions?

[Paul Slootman via rsync]

On Tue 03 Aug 2021, Chris Green via rsync wrote:

> Is there a way to copy (for example) the /etc hierarchy from one
> system to another preserving root ownership of files and without
> revealing root passwords all over the place?

Best way is to run an rsync daemon on the source system, and be sure to
use "uid = 0" so that the daemon reads the source as root.

> So, it's easy for the sending end to be run as root as it's going to be
> run by a script in /etc/cron.daily, so it can access all the files in
> /etc even if only readable by root.

Hmm I prefer to use "pull" mechanisms as that's more secure (harder to
screw up the destination).

So create a /etc/rsyncd.conf file with the appropriate config, something
like:

[etc]
  path = /etc
  read only = yes
  hosts allow = another-system
  uid = 0

If using systemd then enable and start the daemon:

systemctl enable rsync.service
systemctl start rsync.service

Then on another-system as root run rsync:

rsync -a one-system::etc/ /backups/etc/

I usually also use -H for hard links, but /etc usually won't have those.

You can also use an rsync password to make this a bit more secure so
that not everyone on another-system can read all of /etc from
one-system. Details in the manpage.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: '--address' option on client side.

[Paul Slootman via rsync]

On Fri 26 Mar 2021, Harry Mangalam via rsync wrote:
> 
> I'm trying to improve a parallel rsync wrapper called parsyncfp (pfp)  in
> response to a user request.  He wants rsync to emit data on multiple
> interfaces (one interface per rsync instance). From the man page it seems
> like the '--address' option would do that and in fact using it as such does
> not result in an error, but it also does not result in both interfaces
> being used, either from pfp or when launched directly from different shells.
> 
> My route (working from home) shows the 2 wlan interfaces up with
> different IP #s:
> wlp3s0: flags=4163  mtu 1500
>inet 192.168.1.223  netmask 255.255.255.0  broadcast 192.168.1.255
> ...
> 
> wlx9cefd5fb0bb5: flags=4163  mtu 1500
>inet 192.168.1.186  netmask 255.255.255.0  broadcast 192.168.1.255

As both interfaces are on the same network, the kernel will use one
interface to transmit data to that network.

> and route shows:
> $ route
> 
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric RefUse
> Iface
> default router.asus.com 0.0.0.0 UG60100
> wlx9cefd5fb0bb5
> default router.asus.com 0.0.0.0 UG60200

Here you see wlx9cefd5fb0bb5 has a lower metric, hence it will
preferentially be used.

You need to dive deeper into linux policy based routing, to force
traffic e.g. from a particular IP address out over a certain interface.

This is totally outside the scope of rsync; there's nothing rsync can do
to influence this. You need the 'ip rule' and 'ip route' commands.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync support in authprogs - feedback requested

[Paul Slootman via rsync]

On Thu 18 Feb 2021, Bri Hatch via rsync wrote:
> 
> We use authprogs for more than just rsync though, and want more granularity
> than rrsync can support. If you force rrsync for the ssh key via
> command="rrsync" then that key may only be used to run rsync, you can't
> also allow additional commands. From a CI/CD perspective it may be useful
> to have the client side rsync some files, restart some services, and not
> need to use separate keys for each.

I use post-xfer scripts defined in rsyncd.conf to do useful things after
transferring files. That works well.

But I do see that there could be a use for rsync support in authprogs.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is there any way to restore/create hardlinks lost in incremental backups?

[Paul Slootman via rsync]

On Thu 10 Dec 2020, Chris Green via rsync wrote:
> 
> Occasionally, because I've moved things around or because I've done
> something else that breaks things, the hard links aren't created as
> they should be and I get a very space consuming backup increment.
> 
> Is there any easy way that one can restore hard links in the *middle*
> of a series?  For example say I have:-
> 
> day1/pictures
> day2/pictures
> day3/pictures
> day4/pictures
> day5/pictures
> 
> and I notice that day4/pictures is using as much space as
> day1/pictures but all the others are relatively small, i.e.
> day2 day3 and day5 have correctly hard linked to the previous day but
> day4 hasn't.
> 
> It needs a tool that can scan day4, check a file is identical with the
> one in day3 then hardlink it without losing the link from day5.

If you have these files that are hardlinked:

day1/pictures/1.jpg
day2/pictures/1.jpg
day3/pictures/1.jpg

And these are hardlinked, but to a different inode:

day4/pictures/1.jpg
day5/pictures/1.jpg

then there is no way of linking the second group to the first in one
step; you will have to individually link day3/pictures/1.jpg to
day4/pictures/1.jpg and then day3/pictures/1.jpg (or
day4/pictures/1.jpg) to day5/pictures/1.jpg.

It's not like a group of directory entries that are hardlinked to one
inode are some sort of actual group; they just happen to be directory
entries that point to the same inode number. There is no other relation
between those directory entries.

So you will have to incrementally process each next day against the
previous day.


If I make a significant change in such a directory structure (e.g.
renaming a directory) I try to remember to do the same thing on the
backup which some say is wrong, but it saves a lot of space, like you
discovered :)


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: feature request: exclude from path

[Paul Slootman via rsync]

On Mon 03 Aug 2020, Matt Stevens via rsync wrote:

> So I've gotten excluding paths to work as a standalone command. When I paste
> this into a script however, it ignores the exclusions. Any advice?
> 
> rsync -aXvr --times --links
> --exclude={'*.vdi','*.vmdk','*.ova','*.qcow2','.config/discord/'}
> /home/path/ user@nas:/NAS/HOME/destination/
> 
> Are there supposed to be some kind of brackets around this?

Using these brackets with bash causes the --exclude to be repeated for
each value:

$ echo TEST: --exclude={'*.vdi','*.vmdk','*.ova','*.qcow2','.config/discord/'}
TEST: --exclude=*.vdi --exclude=*.vmdk --exclude=*.ova --exclude=*.qcow2 
--exclude=.config/discord/

I suspect you've pasted this into a shell script which does not start
with #!/bin/bash but perhaps #!/bin/sh so that the script is not run by
bash but e.g. ash or dash that don't do the braces expansion.

It's always best (especially in a script) to write it out, don't rely on
the shell for such things.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Removing folder at destination

[Paul Slootman via rsync]

On Thu 14 Nov 2019, Freddie Valdez via rsync wrote:
> 
> Rsync 2.6.9

Wow, that's ancient. Released 06 Nov 2006

> Mac OS interacting with Windows servers.
> 
> My Rsync command: rsync -rvz --exclude-from=exclude.txt source destination.
> 
> I copy large files from server A to server B excluding multiple directories
> which rsync quickly and beautifully executes.
> What I end up with at destination is these folders.
>   01_us_eng...
>  2_ASSETS
>   3_web
>4_print
> I manually then move the web/print folders into the 01_us_eng... folder and
> then I manually delete the assets folder.

Why not exclude 2_ASSETS from the first run, and then do a second run to
rsync the _contents_ of 2_ASSETS to the target 01_us_eng directory?

Add /01_us_eng/2_ASSETS/ to the exclude.txt file.

rsync -rvz --exclude-from=exclude.txt source destination
rsync -rvz source/01_us_eng/2_ASSETS/ destination/01_us_eng/

(perhaps add some variation of the exclude.txt file if you're excluding
stuff under the 2_ASSETS directory)

> My humble question to samba.org is this, can I add an rsync command to move
> folders 3 and 4 into 01... and delete the 2_ASSETS folder so I dont have to
> manually do this 200 times each day?

You can't use rsync to perform remote rename operations, which is what
you're essentially asking for.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: hardlinking missing files from src to a dest: didn't work way I thought it would.

[Paul Slootman via rsync]

On Thu 14 Nov 2019, Pierre Bernhardt via rsync wrote:
> Am 14.11.19 um 10:54 schrieb Paul Slootman via rsync:
> > You need to specify the source directory as the link-dest directory.
> 
> Hi, I tried it also because it's an old question which has never worked
> for me. Instead it creates copies and not hard links:
> 
> 
> pierre@in94:~/tmp$ ls -li a b
> a:
> insgesamt 8
> 257315 -rw-r--r-- 1 pierre pierre 4 Nov 14 10:53 1
> 257316 -rw-r--r-- 1 pierre pierre 6 Nov 14 10:53 2
> 
> b:
> insgesamt 0
> pierre@in94:~/tmp$ rsync -av --link-dest=a a/ b/
> sending incremental file list
> --link-dest arg does not exist: a

There's your clue.
>From the manpage:

If DIR is a relative path, it is relative to the destination
directory.

So it's looking for b/a as the link-dest directory.

Use a full pathname for --link-dest to remove all uncertainty.
E.g.:

rsync -av --link-dest=$(pwd)/a a/ b/

In this case, as the destination is also in same current directory, you
could use:

rsync -av --link-dest=../a a/ b/


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: hardlinking missing files from src to a dest: didn't work way I thought it would.

[Paul Slootman via rsync]

On Thu 14 Nov 2019, L A Walsh via rsync wrote:

> Have a directory with a bunch rpms in it, mostly x86_64.
> 
> Have another directory with a bunch, mostly 'noarch'.
> 
> Some of the noarch files are already in the x86_64 dir
> and don't want to overwrite them.  They are on the same
> physical disk, so really, just want the new 'noarch' files
> hardlinked into the destination.
> 
> sitting in the noarch dir, I tried:
> rsync -auv --ignore-existing  \
>   --link-dest=/tumbleweed/. . /tumbleweed/.

This is not going to do anything useful, as you're telling it to look in
/tumbleweed/ for files that are to be placed in /tumbleweed/ i.e. the
exact same location. It's not going to do

ln /tumbleweed/foo/bar /tumbleweed/foo/bar

which is effectively what you're telling it to do.

You need to specify the source directory as the link-dest directory.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Seemingly impossible bug: -v not always listing every copied file

[Paul Slootman via rsync]

On Wed 30 Oct 2019, raf via rsync wrote:
> 
> I have a task that rsyncs files from a list of
> candidate files (--files-from=). It's verbose (-v) and

It would be helpful to show the complete rsync command line.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: checksum feature request

[Paul Slootman via rsync]

On Tue 01 Oct 2019, Bill Wichser via rsync wrote:
> 
> Attached is the patch we applied.  Since xxhash is in the distro, a
> dependency would be required for this RPM.  If nothing else, perhaps the
> developers should just take a look as this could benefit many.

"The distro" is a bit vague for a tool like rsync that runs on many
versions of Unix and linux, and even windows.

The problem is (AFAIK) that this would need a protocol version bump so
that the checksum algorithm to be used can be decided upon by both ends
of the transfer, it's not as simple as simply replacing the current
algorithm: that would make it impossible to rsync to / from an older
version of rsync.

It's an interesting idea, although I wonder how many users would
actually profit from this. CPU is generally fast enough to handle what
the IO subsystem can read for most people, I imagine.

Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync rewrites all blocks of large files although it uses delta transfer

[Paul Slootman via rsync]

On Thu 14 Feb 2019, Delian Krustev via rsync wrote:
> On Wednesday, February 13, 2019 6:25:59 PM EET Remi Gauvin 
>  wrote:
> > If the --inplace delta is as large as the filesize, then the
> > structure/location of the data has changed enough that the whole file
> > would have to be written out in any case.
> 
> This is not the case.
> If you see my original post you would have noticed that the delta transfer 
> finds only about 20 MB of differences within the almost 2G datafile.

I think you're missing the point of Remi's message.

Say the original file is:

ABCDEFGHIJ

The new file is:

XABCDEFGHI

Then the delta is just 10%, but the entire file needs to be rewritten as
the structure is changed.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: linux rsync <-> SSHDroid has started becoming unreliable after an upgrade of Fedora 28 to 29

[Paul Slootman via rsync]

On Sun 03 Feb 2019, Philip Rhoades via rsync wrote:
> 
> For some years I have been using rsync quite happily to send / retrieve
> files to / from SSHDroid Pro but recently I have started having a problem
> when transferring large numbers of file - I am pretty sure it started after
> upgrading from Fedora x86_64 28 to 29 - but I am not 100% sure.  Below is

[...]

> Corrupted MAC on input.
> ssh_dispatch_run_fatal: Connection to 192.168.1.100 port 22: message
> authentication code incorrect

ssh's communication gets disrupted somehow, and stops the connection,
thus causing rsync to fail.

This is a problem with ssh, not with rsync. Try enabling ssh debug
options, and try using different ssh ciphers.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: bug: xattr filter rule treated as file filter rule on the remote side

[Paul Slootman via rsync]

On Sat 28 Apr 2018, Andras Nagy via rsync wrote:

> Summary: an xattr filter rule (e.g. --filter='-x! user.*’, which is suggested 
> by the documentation) is treated as a file filter rule on the remote side.

I think that you're missing the point that filter rules affect the list
of files to be transferred. This implies that it always applies to the
*sending* side, as that is the side that builds the list to be sent.

It does not matter if the sending side is local or remote, so using that
in the description of the problem is wrong.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: What does this mean? select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)

[Paul Slootman via rsync]

On Mon 22 Jan 2018, Kevin Korb via rsync wrote:

> From man 2 select:
> int select(int nfds, fd_set *readfds, fd_set *writefds,
> fd_set *exceptfds, struct timeval *timeout);
> 
> So, it is waiting for file descriptor #1 to become available with a 60
> second timeout which it is hitting.

Actually:

> > select(1, [0], [], NULL, {60, 0})

... it's waiting for file descriptor 0 to become readable.
The 1st argument 1 is "the highest-numbered file descriptor in any of
the three sets, plus 1".


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 12568] Integer overflow still exists in xattrs.c, leading to buffer overflow

[Paul Slootman via rsync]

On Sun 08 Oct 2017, just subscribed for rsync-qa from bugzilla via rsync wrote:
> 
> --- Comment #1 from Wayne Davison  ---
> I've committed a fix for this into git. Many thanks for pointing this out.
> Sorry for how slow I've been lately.

Hey, I'm just happy you're still around :)


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync got stuck

[Paul Slootman via rsync]

On Tue 05 Sep 2017, Vangelis Katsikaros via rsync wrote:
> On 08/30/2017 05:39 PM, Paul Slootman wrote:
> > On Wed 23 Aug 2017, Vangelis Katsikaros via rsync wrote:
> > 
> >> abc   3797  3796  0 01:12 ?00:03:14 /usr/bin/rsync --compress 
> >> --compress-level=9 --bwlimit=512k --recursive --delay-updates --quiet 
> >> --update --exclude=/.* /SRC_PATH/ DEST_IP:/DEST_PATH/
> > 
> > Try running rsync without the --compress option, that has been the
> > source of problems in the past.
> > If you do need compression you could add that at the ssh level.
> 
> Thanks Paul! I've given it a go, and I'll see how it goes.
> 
> For reference, regarding the --compress options do you have any specific 
> bug(s) in mind or is this something you have noticed in practice?

I've never encountered it but there have been reports of --compress
causing such hangs which go away when --compress is removed.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to set absolute path for rsync?

[Paul Slootman via rsync]

On Tue 15 Aug 2017, Joe Qiao via rsync wrote:

> Thanks so much for the quick reply, Kevin!
> 
> I tried with ssh and --partial-dir, it looks the partial file still will be
> stored in local dir, but not in /tmp.
> 
> 
> Every 1.0s: ls -al /home/joe/rsync/ /tmp/
> Tue Aug 15 17:29:30 2017
> 
> /home/joe/rsync/:
> total 408840
> drwxr-xr-x  2 root root  4096 Aug 15 17:29 .
> drwxr-xr-x 32 joe  joe   4096 Aug 15 15:01 ..
> *-rw---  1 root root 418643968 Aug 15 17:29 .flash_image.14WoMV

No, you misunderstand the meaning of "partial" here.

--partial-dir is where the partially transferred file is stored if rsync
is interrupted with --partial (I'm not sure if --partial-dir implies
--partial, it probably does).

It does NOT mean where the temporary files are stored during transfer.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Clarifications on getting debug information when rsync freezes

[Paul Slootman via rsync]

On Wed 16 Aug 2017, Vangelis Katsikaros via rsync wrote:
> 
> I am having a problem with rsync freezing and I would like to collect the 
> proper information while the problem happens. However, I would like to ask 
> some clarifications.
> 
> rsync-debug
> ===
> 
> I see references of using rsync-debug but I cannot figure out how to use it 
> *while* the rsync is stuck. If I understand from various replies in this 
> email list it must be given as an argument to rsync beforehand?
> 
> If I do not want to change something in my current rsync setup, would it be 
> ok to do sth like this on the destination machine:
> 
> # 1. find the ssh PID from the source IP
> destination_machine $ sudo netstat -atlp | grep "192.168.23"
> tcp0 36 192.168.40.23:ssh   192.168.23.40:49187 
> ESTABLISHED 915/sshd: abc [priv
> 
> # 2. find all related processes
> destination_machine $ sudo pstree --show-pids 915
> sshd(915)───sshd(1079)───rsync(1082)───rsync(1085)
> 
> # 3. then strace these all these

Well, you'll miss anything leading up to the hang, but doing it this may
show something significant, but perhaps not.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync got stuck

[Paul Slootman via rsync]

On Wed 23 Aug 2017, Vangelis Katsikaros via rsync wrote:

> abc   3797  3796  0 01:12 ?00:03:14 /usr/bin/rsync --compress 
> --compress-level=9 --bwlimit=512k --recursive --delay-updates --quiet 
> --update --exclude=/.* /SRC_PATH/ DEST_IP:/DEST_PATH/

Try running rsync without the --compress option, that has been the
source of problems in the past.
If you do need compression you could add that at the ssh level.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Bug: rsync erroneously changes modification time

[Paul Slootman via rsync]

On Mon 12 Jun 2017, max.power--- via rsync wrote:

> How exactly does rsync determine that the copy has the incorrect timestamp
> and not the source file?

The source by definition is correct.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How do you exclude a directory that is a symlink?

[Paul Slootman]

On Fri 03 Mar 2017, Steve Dondley wrote:

> The directory I'm trying to copy from is: /home/blah/dir
> 
> The symlink is /home/blah/dir/unwanted_symlinked_dir
> 
> On Fri, Mar 3, 2017 at 8:10 AM, Paul Slootman <paul+rs...@wurtel.net> wrote:
> 
> > You don't say explicitly, is the target of the symlink inside or
> > outside the source directory?

No, I was asking about the *target* of the symlink. Not the name of the
symlink itself.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How do you exclude a directory that is a symlink?

[Paul Slootman]

On Fri 03 Mar 2017, Steve Dondley wrote:

> I'm trying to rsync a directory from a server to my local machine that has
> a symbolic link to a directory I don't want to download. I have an
> "exclude" option to exclude the symlink which works fine. However, if I add
> a --copy-links option to the command, it appears to override my "exclude"
> directive and the contents of the symlinked directory gets downloaded
> anyway.

You don't say explicitly, is the target of the symlink inside or
outside the source directory?


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Ownership and permissions when syncing directory contents

[Paul Slootman]

On Fri 20 Jan 2017, John Lane wrote:
> 
> When you rsync a directory, say `$rsync mydir/ ...` with the trailing
> slash, the destination directory is changed to the ownership,
> permissions and timestamp of `mydir`.

> $ rsync -a a/ b/ c
> $ ls -ld c
> drwxr-xr-x 2 john users 4096 Jun 15  1985 c
> 
> The directory 'c' now has the timestamp of the file 'a'
> 
> Furthermore, if the copy is done with root, the ownership and
> permissions of the file are also propagated to the directory. Note the
> destination directory gains the properties of the first source directory
> with a trailing slash.

That is as documented. You are requesting that the directory is synced
to c, preserving all attributes (due to -a) so rsync does that.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: make "rsync -N" == "rsync --numeric-ids" ?

[Paul Slootman]

On Fri 18 Nov 2016, Harald Dunkel wrote:
> 
> "--numeric-ids" is a lot of text and easy to be forgotten or
> misspelled.
> 
> Since it is a highly important option for making backups of
> remote systems via rsync I wonder if "-N" could be introduced
> as an abbreviation for "--numeric-ids"? "-N" is not in use yet,
> afaics.

Here's a "me too"; I remember actually trying to use -N a couple of
times instead of --numeric-ids because it's the natural thing to use...


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Paul Slootman]

On Sat 29 Oct 2016, Samuel Williams wrote:

> I'm not proposing some additional characters to split on, but quite
> the opposite, to handle the backslash escaped spaces correctly and NOT
> split. Rest assured, there is no bug with the original escaping. For
> your edification:
> 
> $ echo \I\'\m\ \a\ \s\t\r\i\n\g
> I'm a string

The point is that the original escaping DOUBLE escapes an equals sign:
 foo\\\=bar
It shouldn't, there's no reason to.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: O_NOATIME ?

[Paul Slootman]

On Wed 26 Oct 2016, devz...@web.de wrote:
> 
> since we are using rsync for backing up millions of files in a virtual 
> environment, and most of the virtual machines run on SSD cached storage, i`d 
> be curious how that negatively impacts lifetime of the SSD`s when we do rsync 
> run every night for backup
> 
> my question:
> does rsync normal file comparison run to determine if anything has changed 
> change atime of any files ?
> 
> for me it seems, stat/lstat calls of rsync do NOT modify atime, but i`m not 
> sure under which conditions atime is changed. 

Most filesystems on modern linux systems should be mounted with the
relatime option.

The atime will then only be updated if either the mtime or ctime is
newer than the atime, or if the atime is older than a defined interval.

Note that simply using stat will not update the atime, as the *file*
itself has not been accessed, only its metadata.

There's also the nodiratime option that can be useful, as directories
are read to find what files exist in those directories; and it's
seldomly useful to maintain the atime of a directory.

You can also use noatime as a mount option, but then be sure that no
application uses the atimes of files; e.g. something like mutt use the
atime and mtime to determine whether there is a mail file with unread
mail.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: error code 255 - not mentioned in manpage?

[Paul Slootman]

On Tue 25 Oct 2016, devz...@web.de wrote:
> 
> is there a reason why error code 255 is not mentioned in the manpage 
> and wouldn`t it make sense to add "255 Unexplained Error" there 
> for completeness ?

It wouldn't be unexplained then anymore, would it? :-)


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: -e escape rule

[Paul Slootman]

On Thu 20 Oct 2016, Samuel Williams wrote:
> 
> I'm using Ruby's Shellwords module, which generates a string from an
> array, suitable for shell evaluation.
> 
> Ruby's implementation prefers escaping whitespace with a backslash
> rather than quotes. However, this appears to cause some kind of issue
> in Rsync when it computes argv from -e option.
> 
> Here is an example command generated by some Ruby code:
> 
> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
> ../../latest/etc/ /etc/
> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

There's no reason to escape an "=" sign in the above command. Try:

rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ 
ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

Or even:

rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ 
-o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ 
bac...@example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/

I'd probably create an entry for this host in ~/.ssh/config :

Host example.backup.server.com
User backup
IdentityFile /etc/synco/id_rsa
ConnectTimeout 60
BatchMode=yes

and then just use:

rsync --archive --stats --link-dest ../../latest/etc/ /etc/ 
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/


If you're dynamically doing this, you can pass a config file with -F.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync: connection unexpectedly closed

[Paul Slootman]

On Wed 12 Oct 2016, Kip Warner wrote:
> 
> I think the key insight was in the strace log which showed the select()
> call was timed out. If I knew what type of file descriptor it was being
> fed, I might have a clue. It might have been a socket or something on
> disk. I don't know.

You can use lsof -p $pid to show what files that process has opened.
On linux you can also use 'ls -l /proc/$pid/fd'.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync: connection unexpectedly closed

[Paul Slootman]

On Mon 10 Oct 2016, Kip Warner wrote:
> 
> The server the data is being uploaded to with the strace running on it
> has rsync version:
> 
> $ rsync --version
> rsync  version 3.0.9  protocol version 30
> 
> The client reported:
> 
> $ rsync --version
> rsync  version 3.1.1  protocol version 31

As always it's best to first upgrade to the current version (3.1.3) if
at all possible, as there's always the chance that the cause of your
problems has already been fixed.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: RFE to allow dry-run against read only target

[Paul Slootman]

On Fri 01 Jul 2016, Stuart Anderson wrote:

> I would like to request that rsync -n to an rsync server target that is read 
> only be supported rather than generating the following error,
> ERROR: module is read only
> 
> The motivation is to allow the generation of a list of files against an rsync 
> target that will be used by another application. Since dry-run mode does not 
> require write access it would be convenient if the “module is read only” 
> error message was only thrown if/when a write attempt is made and not assume 
> write access is needed.

Of course, one could argue that the point of the -n option is to test if
it will work when the -n option is removed. That would not be the case
here.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --partial not working?

[Paul Slootman]

On Fri 24 Jun 2016, Kevin Korb wrote:

> Again, --partial only means don't delete the incomplete file if rsync is
> aborted.  Normally rsync will delete the incomplete file so you don't
> have bogus files laying around.
> 
> When you rsync to or from a network mount to rsync that is a local copy.
>  To use rsync over the network either your source or your target would
> be hostname:/path (for rsync over ssh) or hostname::module (for an
> rsyncd server).
> 
> With a local copy rsync forces --whole-file because that is a simple
> read the file from one place and write it to the other place.  If you

If you are 100% sure that the source file has not changed in the
meantime, you could use -P --append as that tells rsync that the
destination file is identical to the source file as far as the data
exists. So rsync will just append the remaining data to the file without
bothering to check it.

According to a test I did --append isn't overridden by --whole-file so
it should work.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Memory consumption for rsync -axv --delete

[Paul Slootman]

On Fri 25 Mar 2016, John Long wrote:
> 
> I have been using rsync for many years and never had any kind of problem.
> Lately I am running out of RAM trying to do an incremental backup to a box
> that only has 2G of RAM. The entire directory structure I'm mirroring is
> about 200G of files. A minority of subdirectories have many files.
> 
> Is there a way to do an incremental backup with --delete option that does
> not use as much memory? Is there a way to tell rsync to use a tempfile
> instead of RAM for keeping tracking of whatever it does?

No to the last question; you could consider adding (more) swapspace to
the system, which is effectively like using a tempfile.

> And would it be useful to add ignores for the subdirectories I know have
> many files and back them up separately? Is --delete safe to use in this
> case, as in does --delete with --ignore somedir/ not delete files in other
> target dirs that are not in the ignore path?

There's no --ignore, you probably mean --exclude.
I don't really understand what you're asking in your last question...
Why should --exclude somedir/ affect what --delete does elsewhere?
--delete will still delete stuff elsewhere if necessary.
Also look at the description of --delete and --delete-excluded, if you
have any questions about what's in the manpage then feel free to ask
those here; but for now I get the impression you haven't spent much time
reading the manpage.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [PATCH] Consider nanoseconds when quick-checking for unchanged files

[Paul Slootman]

On Wed 20 Jan 2016, Wayne Davison wrote:

> equal. One possible improvement would be to skip the nanosecond check if
> the destination file has a nanosecond value of 0.  That could possibly be
> improved if we figure out if a particular device ID supports nanoseconds
> somehow.  I have a potential heuristic in mind that I can code up and see
> how it works.

It would be very handy if that could be extended to take into account
FAT filesystems, i.e. automagically set --modify-window=2 if it's
detected that you're writing to a FAT filesystem. It's basically the
same problem as detecting that you're writing to a filesystem that
doesn't support nanoseconds IMHO.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [PATCH] Consider nanoseconds when quick-checking for unchanged files

[Paul Slootman]

On Wed 20 Jan 2016, Andrey Gursky wrote:
> 
> I was just about to implement the same, since nanoseconds are taken
> into account when transferring, thus making it obvious not to ignore

Really? I thought the protocol only transmits seconds.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

minor patch for manpage

[Paul Slootman]

The description for --compress-level=NUM does not give any indication
what values are permitted for NUM.

Paul

--- ./rsync.yo.orig 2015-12-30 11:47:24.180646652 +0100
+++ ./rsync.yo  2015-12-30 11:47:27.477198899 +0100
@@ -1907,7 +1907,9 @@
 that will not be compressed.
 
 dit(bf(--compress-level=NUM)) Explicitly set the compression level to use
-(see bf(--compress)) instead of letting it default.  If NUM is non-zero,
+(see bf(--compress)) instead of letting it default.  Allowed values for NUM
+are between 0 and 9; default when bf(--compress) option is specified is 6.
+If NUM is non-zero,
 the bf(--compress) option is implied.
 
 dit(bf(--skip-compress=LIST)) Override the list of file suffixes that will

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

strange behaviour when using (conflicting) options -q --progress

[Paul Slootman]

-q seems to override -v completely, but when combined with --progress,
a single newline is output when there are no updates transferred; but if
a file *was* updated nothing at all is output.

It seems that there might be some short-circuited code when nothing is
trasferred, but that a check for quiet mode is skipped somehow.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: strange behaviour when using (conflicting) options -q --progress

[Paul Slootman]

On Wed 30 Dec 2015, Paul Slootman wrote:

> -q seems to override -v completely, but when combined with --progress,
> a single newline is output when there are no updates transferred; but if
> a file *was* updated nothing at all is output.
> 
> It seems that there might be some short-circuited code when nothing is
> trasferred, but that a check for quiet mode is skipped somehow.

The bug report in Debian below may be related:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749165
Here the issue is basically that a newline is *missing* after --progress
output has been generated. The output basically is:

 0 files...^M 400 files...^M 2500 files...^M 4600 files...^Ma1/
a2/

The a1/ part overwrites the " 4600 files..." part partially.

perhaps the extraneous newline in my first part is related to this
missing newline here?


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: What is the default compression level for local synchronisation?

[Paul Slootman]

On Mon 05 Oct 2015, Charles wrote:
> 
> The question I want answering is "Does rsync need telling not to
> compress when SOURCE and DEST are both local or does it automatically
> choose a sane compression setting?"

If you don't ask rsync to compress, it won't compress...
That's different from telling rsync not to compress.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --link-dest and --files-from lead by a change list from some file system audit tool (Was: Re: cut-off time for rsync ?)

[Paul Slootman]

On Mon 13 Jul 2015, Andrew Gideon wrote:
 
 On the other hand, I do confess that I am sometimes miffed at the waste 
 involved in a small change to a very large file.  Rsync is smart about 
 moving minimal data, but it still stores an entire new copy of the file.
 
 What's needed is a file system that can do what hard links do, but at the 
 file page level.  I imagine that this would work using the same Copy On 
 Write logic used in managing memory pages after a fork().

btrfs has support for this: you make a backup, then create a btrfs
snapshot of the filesystem (or directory), then the next time you make a
new backup with rsync, use --inplace so that just changed parts of the
file are written to the same blocks and btrfs will take care of the
copy-on-write part.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync Failed /proc/kcore 128 TiB NTFS HDD Makes Scary sounds even when not mounted

[Paul Slootman]

On Thu 30 Apr 2015, Kezhawe wrote:

 I can still readwrite on the external hdd and on my laptop (which I
 have my gentoo there) I can't do anything on / because its filled up
 with /proc/kcore after rsync failed

You need to ignore /proc/kcore as (a) that's a virtual file that doesn't
reside on your disk and (b) it's always that large as it's the
representation of the kernel's address space.

As your root filesystem is full, something else has filled it.
Are you sure /run/media/narin/My Book/Gentoo Rsync Backup/ (curious
quoting BTW) was in fact mounted? If not, that's simply on your root
filesystem and you've copied your data to the root filesystem.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Two same rsync commands give different results on two debian mirrors.

[Paul Slootman]

On Tue 14 Apr 2015, Hongyi Zhao wrote:
 On Tue, 14 Apr 2015 09:32:10 +0200, Paul Slootman wrote:
 
  On Tue 14 Apr 2015, Hongyi Zhao wrote:
  
  See the following commands:
  
  ___
  
  $ rsync -rmnv -f +_README -f -_* ftp.is.co.za::mirror/ftp.debian.org/
  ftp.us.debian.org/debian
 
 Please see the above line including the part of line-wrapping.  I've used 

You should ensure your messages don't mangle the command lines you want
to show. You could have inserted a \ and a newline before the hostname:

$ rsync -rmnv -f +_README -f -_* \
  ftp.is.co.za::mirror/ftp.debian.org/ftp.us.debian.org/debian


 In fact, the issue I meet is as follows, see the following for detail: 
 
 1- If I append the `/' in the end of the tree, I'll succeed:

That's right, by appending the / you start at the contents of the given
directory, not with the directory itself. Without the / rsync needs to
recurse into the directory, but you've excluded '*' in your filter so it
can't reach the README.

 But for the case when use of the following rsync server, both case will 
 succeed:
 
 werner@debian:~$ rsync -rmnv -f +_README -f -_* ftp.cn.debian.org::debian 

This is because the ::debian part is not a directory, it's a module name
and hence the starting point is the contents of the module's directory,
which contains README so that can be found.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Two same rsync commands give different results on two debian mirrors.

[Paul Slootman]

On Tue 14 Apr 2015, Hongyi Zhao wrote:
 
 See the following commands:
 
 ___
 
 $ rsync -rmnv -f +_README -f -_* ftp.is.co.za::mirror/ftp.debian.org/
 ftp.us.debian.org/debian
 
 [snipped]
 
 receiving file list ... done
 
 sent 23 bytes  received 9 bytes  1.42 bytes/sec
 total size is 0  speedup is 0.00 (DRY RUN)
 
 $ rsync -rmnv -f +_README -f -_* ftp.cn.debian.org::debian
 
 [snipped]
 
 receiving file list ... done
 drwxr-xr-x  4,096 2015/04/14 11:34:19 .
 -rw-rw-r--  1,370 2015/01/10 18:21:28 README
 
 sent 47 bytes  received 1,389 bytes  574.40 bytes/sec
 total size is 1,370  speedup is 0.95 (DRY RUN)
 _
 
 Why the same rsync commands give different results on the above two debian 
 mirrors?  The first one failed to find anything even the README exists, 
 while the second one succeed for the job.

The contents of those locations is entirely different, which becomes
obvious if you simply run:

$ rsync ftp.is.co.za::mirror/ftp.debian.org/
$ rsync ftp.cn.debian.org::debian

If you want the same tree, use
ftp.is.co.za::mirror/ftp.debian.org/ftp.us.debian.org/debian/ for the
first location.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: The complicated filter rule used by me worked for one Debian mirror and not for the other.

[Paul Slootman]

On Tue 14 Apr 2015, Hongyi Zhao wrote:

 I write a complex filter rules as follows:
 
 rsync -amvHPRSB131072 -n --delete --delete-excluded \
 -f +_dists/jessie/**binary-all/Packages.gz \
 -f +_dists/jessie/Release* \
 -f +_dists/jessie/**binary-amd64/Packages.gz \
 -f +_dists/jessie/**installer-amd64/*** \
 -f +_dists/jessie/**binary-i386/Packages.gz \
 -f +_dists/jessie/**installer-i386/*** \
 -f +_dists/***/ \
 -f -_* ftp.cn.debian.org::debian/
 
 When I run the above commands, it will list the corresponding files that 
 will be retrieved corresponding to these rules.
 
 But, when use it to the other debian mirror as follows:
 
 rsync -amvHPRSB131072 -n --delete --delete-excluded \
 -f +_dists/jessie/**binary-all/Packages.gz \
 -f +_dists/jessie/Release* \
 -f +_dists/jessie/**binary-amd64/Packages.gz \
 -f +_dists/jessie/**installer-amd64/*** \
 -f +_dists/jessie/**binary-i386/Packages.gz \
 -f +_dists/jessie/**installer-i386/*** \
 -f +_dists/***/ \
 -f -_* \
 ftp.is.co.za::mirror/ftp.debian.org/ftp.us.debian.org/debian/
 
 This time, it will give nothing.

ftp.cn.debian.org is running a version of rsync that supports protocol
version 30, whereas ftp.is.co.za is running an older version that only
supports up to 29.  I haven't used such complex filter rules yet (I've
been sticking to the --include / --exclude stuff) but I wouldn't be
surprised if some of the filter rules don't work on the older rsync.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync 3.0.9 segmentation fault

[Paul Slootman]

On Tue 07 Apr 2015, Aron Rotteveel wrote:

 Anyone have any other ideas I could try to debug this issue? :)

You could try using the latest version of rsync? 3.1.1 was released some
time ago.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync slow exclude folder

[Paul Slootman]

On Sat 28 Feb 2015, ? ?? wrote:
 
 rsync  version 3.0.9  protocol version 30
 
 rsync -a --exclude=tmp/* /home/ /backup/home/
 
 It is necessary that the contents of the folder tmp copy, but the folder tmp 
 in
 backup was created.
 
 If the folder /home/tmp/ is many millions of files, rsync
 think of this folder can be seen through the lsof -p PID
 
 If you do so: rsync -a --exclude=tmp/ /home/ /backup/home/
 then backup is done very quickly, but tmp folder in the backup does not
 will be created.
 
 How to make so that rsync long thought over such folders, but
 This created a backup of them empty?

I think the problem is that you only excluded files in tmp, not also
directories. So if you have a directory /home/tmp/subdir/ that directory
will still be created on the destination side. That is why rsync has to
check every entry in the tmp directory: to see if it is a directory.

Try --exclude=tmp/** to exclude everything under tmp, including
directories.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync to multiple destinations

[Paul Slootman]

On Thu 29 Jan 2015, Johan Kröckel wrote:

 I am looking for a way to start one rsync command with multiple destinations.
 My use case:
 I have one (slow) usb drive which should be synced to two other
 harddrives. It would be great if rsync could read one file which is to
 be copied to both hard drives only once, so the slow usb connection is
 used more efficiently.

Running two parallel rsync jobs, one to one disk and the other to the
second disk will generally lead to the processes running in sync with
each other, so that data will only be read from the source disk once,
the read by the other rsync process will be satisfied from the buffer
cache.
At least that's been my experience...


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync splits filenames, creates special characters where none are, weird permissions

[Paul Slootman]

On Wed 07 Jan 2015, Lenz Weber wrote:

 Where the local destination /data/snapshots is an NFS volume mounted with the 
 flags
 (rw,noatime,addr=192.168.1.XX)
 and the source is a symlink to a zfs snapshot - that looks like this:
 /var/backups/mail - 
 /tank/mail/.zfs/snapshot/zfs-auto-snap_hourly-2015-01-07-1417

Why not skip the NFS part and run rsync to the destination over the
network? Rsync is written to minimize network traffic at the cost of
local IO, and if you're doing NFS then that local IO is really also
network traffic.  You also eliminate one potential source of problems in
that case.

 as far as I can tell, both systems work with UTF8 just fine (source is Ubuntu 
 14.04 and target is Debian Lenny)
 
 Now there seems to be a problem while gathering or transferring the file list,
 as rsync tries to create files/folders that share a part with real files on 
 the source,
 but with additional characters, sometimes cut off, without the preceding 
 parent folder et cetera.

How often? Every file? 10% 1%? ...

 The source file names in this case look like this:
 
 /var/backups/mail/redacted-domain/catchall/Maildir/.Sent/cur/1313508314.M654736P32713V0801I000B03CC_6.redacted-hostname,S=42352:2,S
 /var/backups/mail/redacted-domain/info/Maildir/.Sent/cur/1313508314.M654736P32713V0801I000B03CC_6.redacted-hostname,S=42352:2,S
 
 but rsync fails on files like this, that clearly do not exist:
 
 skipping non-regular file 
 2713V0801I000B03CC_6.redacted-hostname,S=423\#001\#305\#001O\#233\#240é
 skipping non-regular file 
 2713V0801I000B03CC_6.redacted-hostname,S=42352:2,S
 skipping non-regular file 83E13498714.M297793P23544V000
 skipping non-regular file Ø \#201
 skipping non-regular file 
 redacted-domain/catchall/Maildir/.Sent/cur/1301490998.M622842P6671V0801I00280BD9_0.redacted-hostname\#004
 skipping non-regular file 
 2713V0801I000B03CC_6.redacted-hostname,S=pedition/courierimapkeywords/:list

Is this reproducible, i.e. a second run (after cleaning up the mess it
left behind) creates these same files again, or others?

My first thought is that this combination of factors is triggering some
sort of memory problems which is corrupting the filenames. It may also
be useful to do a run with --checksum to catch any data corruption (or
to see if it finds mismatches where there shouldn't).

If this can be narrowed down to a fairly small transfer which goes wrong
reproducibly, then using strace -f on rsync (with -o strace-output.txt)
then perhaps you can see whether the errors already occur when reading
the files or not.


I have not heard of rsync performing this way, so I strongly suspect
some hardware problem.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Differing behaviour on consecutive runs

[Paul Slootman]

On Wed 17 Dec 2014, David Jennings wrote:

 Hi all. I have a strange issue with sync'ing from a USB-attached ext4 file
 system on one machine, to a USB-attached ext3 file system on another.

 The sync works fine (a few thousand files). If I run it again straight away,
 or a few minutes later, rsync reports no action required.
 
 But if I wait for a few hours and run it again it copies all the files
 again.
 
 I'm trying to determine what's making it think the files are not up-to-date.
 I have all the debug flags turned on. Am I missing something that will let
 me know what rsync is thinking?

Use -i (or --itemize-changes), that's more helpful in determining _why_
a file is being updated.

 I've just noticed that set modtime differs between the two runs. If I do
 an ls now on that directory I get:

The times of a directory are hardly useful, writing a file or renaming a
file in that directory will update the modification time of the
directory. Check file timestamps if you're trying to track something
down, but first try it with -i (and leave out all the debugging, that
might just obscure the -i output).



Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Bug-report:rsync may hung if time jumps backwards

[Paul Slootman]

On Wed 03 Dec 2014, yhu2 wrote:
 On 11/28/2014 09:41 AM, yhu2 wrote:
 
 thanks your reply, could you please send off official fix?
 
 or could you please  tell me which release will include this patch. any
 comments would be appreciated!

The git repository is browseable, the patch for this can be found at
https://git.samba.org/?p=rsync.git;a=commitdiff;h=5546dab32970955e77ef7a5886bcd8fb765a25bf


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync keeps copying everything to the server

[Paul Slootman]

On Thu 02 Oct 2014, Juan Carlos Valle Sandoval wrote:

 I have changed the script to:
 
 rsync.exe -ai --size-only --progress
 --log-file=/cygdrive/v/notes/archive/RSyncLogFile.txt
 /cygdrive/d/%username%/notes/archive /cygdrive/v/notes/

You could still try the other suggestion of --modify-window=1
instead of --size-only. Now if a file is updated to e.g. correct a
spelling mistake srync to rsync then that files won't be backed up.

 The output is:
 
 2014/10/02 10:25:49 [5088] f.st.. archive/file1.nsf
 2014/10/02 10:25:54 [5088] sent 4073663746 bytes  received 41 bytes  total
 size 9589227520
 
 The result:
 
 Only the modified file was backed up. However the whole file was copied,
 not just the new things. :-(

Why do you say this?
The output reports (paraphrased) sent 4GB, total size 9.5GB so unless
you're not showing the whole output then it only sent half the file.

Use --stats to get more info on the whole transfer.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync keeps copying everything to the server

[Paul Slootman]

On Sat 04 Oct 2014, Paul Slootman wrote:

  Only the modified file was backed up. However the whole file was copied,
  not just the new things. :-(
 
 Why do you say this?

Oops, didn't read Kevin's followup and didn't notice the rsync over two
local filesystem paths.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Backup scripts - recycling old backup directories (Kevin Korb)

[Paul Slootman]

On Mon 15 Sep 2014, Kevin Korb wrote:
 
 I would never operate in a manner that only has 5-6 days of old
 backups.  The backups that I am deleting are more than a year old.

I keep the Sunday backups for a month, the 1st of the month backups for
a year.

The other daily backups are expired after 10-20 days depending on
importance etc.



Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Concern: rsync failing to find some attributes in a file transfer?

[Paul Slootman]

On Sat 26 Jul 2014, Kevin Korb wrote:
 
  I just moved my home partition to a new harddisk w/more space.
 
 Home Partition?  Are we in 1995?  Why would you have a partition
 mounted anywhere other than /boot ?

Didn't we just have this discussion already recently?
There are valid reasons to have separate filesystems.
My /home is encrypted, the rest isn't. I have a separate XFS filesystem
as I find that the best option for handling really large files.

The relevancy of the separate filesystem to the question isn't clear to
me either, so why bring it up?


  my $rcmd = [$Rsync]; push( @$rcmd, qw( --8-bit-output --acls
  --archive --hard-links --human-readable --no-inc-recursive
  --one-file-system --prune-empty-dirs --whole-file --xattrs ), 
  --compare-dest=$base_lvh-fs_mp/.);

Transferring with --compare-dest? I thought that the data was being
moved from one filesystem to another, that seldomly calls for usage of
--compare-dest.  It seems to me that the perl script being used is meant
for another purpose, and it's being used inappropriately here.
Why not just use rsync directly? That way maybe we here on the mailing
list can make sense of what's actually happening. Otherwise take it up
with the author of that script.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: SIGHUP signal

[Paul Slootman]

On Sat 15 Feb 2014, Perry Hutchison wrote:
 Hiroyuki Ikegami ike...@mixallow.net wrote:
  2014-02-15 7:39 GMT+09:00 Grozdan neutri...@gmail.com:
   Yesterday, I changed my rsyncd.conf file to add one more module to it.
   Then I sent a kill -HUP $pid signal to rsync running in daemon mode,
   but what gives? It just died so I had to start it up again. I though
   sending a HUP would just make it reload its config file, no?
 
  Many softwares catches SIGHUP as a trigger to reload configurations.
  But it is a kind of software design. If the programmer does not want to
  write signal handling code, the programs received HUP just dies.
 
 This really ought to go on the to-do list.  It is a very
 longstanding convention that a daemon should reinitialize itself
 (for some reasonable definition of reinitialize) upon receiving
 a SIGHUP.

Why should the code be modified to help those that don't read
the docs properly?

From the rsyncd.conf manpage:

   Note that you should not send the rsync daemon a HUP signal to force it
   to reread the rsyncd.conf file. The file is re-read on each client con‐
   nection.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: problem with rsync-daemon via ssh

[Paul Slootman]

On Fri 14 Feb 2014, Matthias Schniedermeyer wrote:
 
 You mixed the options from remote shell with rsync daemon.
 Rsync is used either as 'rsync over SSH'(/remote shell) OR daemon-mode.

Matthias,
Ik recommend you check out the USING RSYNC-DAEMON FEATURES VIA A
REMOTE-SHELL CONNECTION part of the manpage.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: /usr/bin/ssh not found when rsync is executed within rsnapshot

[Paul Slootman]

On Mon 10 Feb 2014, Lorenz wrote:

 grep -v # /etc/rsnapshot | grep [a-z]
 i.e. the /etc/rsnapshot minus the comments and the empty lines:

I'd recommend using 'grep .' to find non-empty lines... shorter and more
accurate :-)

 rsync_long_args   -ev --rsync-path=/home/backupuser/rsync-wrapper.sh

-e is the short version of --rsh so I don't know what you're trying to
do here... use the 'v' command instead of (the default) ssh? Probably not.

 /usr/bin/rsync -av -ev --rsync-path=/home/backupuser/rsync-wrapper.sh \
 --rsh=/usr/bin/ssh -i /home/backupuser/.ssh/id_rsa backupuser@debx40:/ \
 /media/extfp/Backup/rsnapshot/test/hourly.0/debx40/
 rsync: Failed to exec /usr/bin/ssh -i /home/backupuser/.ssh/id_rsa: No such 
 file or directory (2)

Besides the extraneous -e option this should work.
The error message is a bit misleading though.

Make sure that there are no wrong whitespace characters in there.
I've fallen into the trap of copypasting commands / configs from
websites and having them fail mysteriously, until I noticed I could not
left-shift those lines in vim with  . Those leading spaces were not
spaces but no-break spaces, hex value 0x80.

So check your config / scripts with LANG=C cat -v /etc/rsnapshot etc.

 What could be the reason? How could i debug this?

I often use strace -f -e execve command ... and / or
strace -f -e execve command ... in such cases to see what
it is really trying to run.

Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: List subdirectories only n level deep on the remote server when use the -a option of rsync.

[Paul Slootman]

On Sun 02 Feb 2014, Hongyi Zhao wrote:
 
 But I've tried the following methodes based on your suggestion and get
 different results:
 
 
 werner@debian:~$ rsync -a --exclude '/*/*/*/*'
 rsync://ftp.cn.debian.org/debian | wc -l
 1798
 werner@debian:~$ rsync -a --exclude '/*/*/*/**'
 rsync://ftp.cn.debian.org/debian | wc -l
 1798
 werner@debian:~$ rsync -a --exclude '/*/*/*/***'
 rsync://ftp.cn.debian.org/debian | wc -l
 1631
 
 
 As you can see, the first two give same results, but the third, i.e.,
 the one you gives, give a different result.   Why does this happen?

From the manpage:

* a trailing dir_name/*** will match both the directory (as if
  dir_name/ had been specified) and everything in the directory
  (as if dir_name/**  had been  specified).  This behavior was
  added in version 2.6.7.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: List subdirectories only n level deep on the remote server when use the -a option of rsync.

[Paul Slootman]

On Sun 02 Feb 2014, Hongyi Zhao wrote:
 
 I want to list the contents on the remote rsync server only to n, say,
 2 level deep.
 
 In my case the the server is rsync://ftp.cn.debian.org, and I want to
 list all of the stuff in the following subdirectory only to 2 level
 deep:
 
 rsync://ftp.cn.debian.org/debian
 
 Any hints on this?

rsync -a --exclude '/*/*/*/***' rsync://ftp.cn.debian.org/debian


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Selective --delete

[Paul Slootman]

On Tue 07 Jan 2014, Ashley M. Kirchner wrote:
 On Tue, Jan 7, 2014 at 9:44 AM, Paul Slootman paul+rs...@wurtel.net wrote:
 
  rsync will not touch files that have been --exclude'ed; unless you also
  specify --delete-excluded . So just specify those LIVE-only files in
  your exclude list.
 
 Uh, that hasn't been my experience.  I just ran this and it happily deleted
 the events folder.  I did verify that it gets excluded during the sync
 portion as files that either exist or don't exist on either servers did not
 get synced or deleted, but it did, at the end, delete the entire folder and
 contents.

$ cd /tmp
$ mkdir rsync-src rsync-dst
$ mkdir rsync-dst/events
$ rsync -ai --delete --exclude /events/ rsync-src/ rsync-dst/
.d..t.. ./
$ ls -ld rsync-dst/events
drwxr-xr-x 2 paul paul 4096 Jan  8 17:49 rsync-dst/events

Your experience differs from mine...


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Selective --delete

[Paul Slootman]

On Tue 07 Jan 2014, Ashley M. Kirchner wrote:
 
 My issue is with the --delete part.  While I want the procedure to delete
 extraneous files from the LIVE server when they are removed from the DEV
 one, there are a few files on LIVE that do not and will never exist on the
 DEV server, however they are required on the LIVE one.  Is there a way to
 have rsync NOT delete those?  Is there an equivalent exclude option for the

rsync will not touch files that have been --exclude'ed; unless you also
specify --delete-excluded . So just specify those LIVE-only files in
your exclude list.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --itemize-changes not recursive (not printing created files inside created directories)

[Paul Slootman]

On Wed 18 Dec 2013, Kevin Korb wrote:

 Also, rsync's -c is rather dumb as it computes checksums for files
 that have different sizes so they can't possibly be the same and it
 computes checksums for files that only exist on one end and therefore
 has nothing to compare them to.

The list of files on the source is generated and transferred to the
destination before rsync knows that the destination file is different.

To make rsync checksum only the files with same size would mean changing
the filesystem scan to a two-pass thing (send the list of filenames plus
their sizes, wait for the destination to tell you what files need
checksumming, do that and send the filenames again, now with checksum
data), and retransferring file metadata again. Remember that rsync is
basically a tool optimized for very fast local IO and much lower network
bandwidth, so such a two-pass feature could possibly greatly increase
the network traffic overhead (i.e. traffic not related to file data
itself).

That said, I'm sure that if someone builds a patch to implement
something like that via yet another commandline option, no one would
complain :-)


Note that transferred files are always checksummed anyway, as
described at the end of the -c option manpage description.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Moving/merging a filesystem back into /

[Paul Slootman]

On Tue 03 Dec 2013, Kevin Korb wrote:
 
 Sure.  Simply put, you can just change ext[23] to ext4 in fstab and it
 will just work.  Faster even.

Just be very sure that you have ext4 support in your kernel, otherwise
you have a problem. ext4 should be listed in /proc/filesystems .
Of course, the root filesystem is mounted before /etc/fstab can be
read... :-)  ignoring initrd for now.

  There is a lot of FUD about reiserfs.
 
 Reiserfs can take a hard crash without needing an fsck just like
 ext[34] can.  This is what journaling is for.  I am talking about when
 fsck -p aborts and says you need to run fsck manually.  In ext[34]
 this is a minor inconvenience and you can usually even run fsck with
 - -y to just say yes to any prompts.  With reiserfs if you ever have a
 problem at this level you are screwed.

Not necessarily, but with reiser it's documented that reiser expects
_no_ storage failure whatsoever, so some form of RAID with redundancy is
a must.  You _are_ screwed if you happen to have an image of a reiser
filesystem on your reiser filesystem, as the fsck will see the reiser
markers and merge that image into your real filesystem.

I've avoided reiser since Hans got jailed for murder (but that's another
topic :-)

 I have seen it happen.  It is rare that reiserfs needs an fsck but if
 it does the fsck is destructive.

I've seen a reiser fsck eat the filesystem about 5 in 100 times.
(I used to help administer a server farm with 200 servers, running on
reiser at the time.)

 but the same is not true on Gentoo.  I used to use XFS for my media
 storage (it is great on large files) but I had to give it up because
 it had a bunch of weird problems interacting with NFS.

I used XFS with NFS for my dreambox (DVB satellite receiver), storing
multi-GB recordings. Of course there was only one NFS client. That
worked great. It's especially good at large files. That said, ext4 has
become quite amazing and even if it does fall mostly over, chances of
extracting files from the broken fs are excellent.

btrfs is promising to be quite cool as well, I love the snapshot system.
However having had some bad experiences with corruption I'm hesitant to
use it for anything critical for now. That _was_ a number of years ago,
on a 8TB filesystem which got hammered pretty badly with daily backups
(each being snapshotted and deleted after some time), so not really
representative of normal usage.

Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: segfault in 3.1.1dev

[Paul Slootman]

On Fri 01 Nov 2013, Carlos Carvalho wrote:

 Happened again:
 
 Oct 31 15:55:49 sagres kernel: denied resource overstep by requesting 8392704 
 for RLIMIT_STACK against limit 8388608 for /usr/local/bin/rsync[rsync:80954] 
 uid/euid:1011/1011 gid/egid:1011/1011, parent 
 /usr/local/bin/rsync[rsync:78330] uid/euid:1011/1011 gid/egid:1011/1011
 Oct 31 15:55:49 sagres kernel: Segmentation fault occurred at 
 03a8d3ddee88 in /usr/local/bin/rsync[rsync:80954]

Looks to me that the problem is the first line, the segfault is
triggered by a failed stack extension.
Now a stack size of 8MB should be enough for everybody... so why is
rsync using so much stack.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync-3.1.0 --chown, --usermap, and --groupmap ignored?

[Paul Slootman]

On Sun 13 Oct 2013, Jed Brown wrote:

 The man page describes options --chown, --usermap, and --groupmap, but
 these seem to be silently ignored beyond validating that the user and
 groups do indeed exist.  Is the following supposed to work?

No, because you're not running the rsync command as the superuser,
and not using the --owner / --group options.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: multiple secrets file

[Paul Slootman]

On Mon 30 Sep 2013, thomas veymont wrote:
 
 ie. any modules declared in mymodules1/ will use users/passwords in
 secret1.inc whereas mymodules2/ modules will use secret2.inc.
 
 is it the right way to implement this ?

Well, the secrets parameter is listed under MODULE PARAMETERS in the
rsyncd.conf manpage(*), so that should be fine.

(*) in the 3.0.9 version at least, only just noticed 3.1.0 being out

Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: Re: need help with an rsync patch

[Paul Slootman]

On Tue 13 Aug 2013, Sherin A wrote:

 But if a user create  a
 hard link to /etc/shadow from his home dir , and he request a restore ,
 then he can read the shadow files and decrypt it .

If he can make a HARD link to the shadow file, then he can already read
it - and worse.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: Re: need help with an rsync patch

[Paul Slootman]

On Tue 13 Aug 2013, Matthias Schniedermeyer wrote:
 On 13.08.2013 09:52, Paul Slootman wrote:
  On Tue 13 Aug 2013, Sherin A wrote:
  
   But if a user create  a
   hard link to /etc/shadow from his home dir , and he request a restore ,
   then he can read the shadow files and decrypt it .
  
  If he can make a HARD link to the shadow file, then he can already read
  it - and worse.
 
 No.

My mistake for assuming that people run current linux kernels...

/proc/sys/fs/protected_hardlinks (since Linux 3.6)
When  the value in this file is 0, no restrictions are placed on
the creation of hard links (i.e., this is the historical  behav‐
iour  before  Linux  3.6).   When the value in this file is 1, a
hard link can be created to a target file only  if  one  of  the
following conditions is true:

I would suggest that upgrading the kernel is a better solution for the
OP than patching rsync.  If your backup strategy involves backuping up
files as root to a medium that is readable by everyone so that the link
in the user's home directory is restorable as the user, then there are
more problems waiting to happen besides this...


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: Re: need help with an rsync patch

[Paul Slootman]

On Tue 13 Aug 2013, Matthias Schniedermeyer wrote:
 
 BUT there is no direct vulnerability in that, only processes after that 
 (like backup/rsync) can make a vulnerability out of it.

... which is what I already wrote.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: Re: need help with an rsync patch

[Paul Slootman]

On Tue 13 Aug 2013, Matthias Schniedermeyer wrote:
 
 I read your sentence differently:
 
  If he can make a HARD link to the shadow file, then he can already 
  read it - and worse.
 
 My understanding of your sentence says:
 The ability to hardlink, means that anyone can read any file they can 
 make a hardlink to.

Then I didn't express myself clearly enough. Again, keep in mind I was
thinking from the perspective of a linux 3.6 and up kernel without any
sys tweaks.

 Having access to the directory entry is not the same as having access to 
 the inode. User/group/permission is on the inode NOT the 
 directory-entry.

I have access to the inode when I do an ls -l of the file :-P
perhaps you mean modification permissions. Then again, when
hardlinking, I'm changing the link count which is stored in the inode... :)


I'm done here... coming back to the OP's problem: if the backup is made
by root, then a user should not be allowed to access all parts of that
backup. The security problem is there, and not in rsync.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Platform variations with --exclude-from

[Paul Slootman]

On Mon 27 May 2013, Joseph L. Casale wrote:

 On Linux, an rsync command and exclude_file contents of:
 
 # cat exclude_file
 /etc/alsa
 # rsync -a --delete --delete-excluded --exclude-from=exclude_file /etc 
 server::module
 
 properly excludes /etc/alsa but not any file within /etc's directories that 
 is named alsa.

Here the exclude file's contents matches the path you've given:
root is /, path is etc so /etc/alsa matches.


 On Windows I don't seem to be able to reliably emulate this:
 
 C:\Scripts\Backup\rsyncdtype rsyncd_exclude
 /cygdrive/d/$RECYCLE.BIN
 /cygdrive/d/Exclude
 /cygdrive/d/Inetpub
 /cygdrive/d/System Volume Information
 C:\Program Files (x86)\cwRsync\binrsync -a --delete --delete-excluded 
 --exclude-from=exclude_file /cygdrive/d server::module

Here the exclude files are also anchored at the filesystem root,
but the transfer root is /cygdrive/

If you change the excludes to /d/$RECYCLE.BIN etc. (i.e. remove the
leading /cygdrive) then it should work.

That you're seeing this on windows is irrevelant, a similar setup on
linux (e.g. transferring /etc/network and excluding
/etc/network/interfaces) will also fail to do what you want.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync behavior on copy-on-write filesystems

[Paul Slootman]

On Tue 21 May 2013, Allen Supynuk wrote:
 
 ## 1) Start with an empty filesystem
 
 $ df -h .

Note that you need to be using btrfs filesystem df .
for reliable numbers; the normal df does not take into account
background cleanups etc.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: exclude a pattern but only in the top level

[Paul Slootman]

On Wed 15 May 2013, Brian K. White wrote:

 I did in the case when it was only one pattern, but that was just a
 simplified example.
 
 The actual job involves too many include and exclude patterns to use
 --include  --exclude, or even --include-from and --exclude-from,
 because the patterns are generated on the fly by a script from
 values supplied on the script commandline or hardcoded at the top,

I think you're missing the point.

--exclude-from=filename will do the same.
-f . filename specifies a rule, and that rule tells rsync to read the
file.  --exclude-from=filename tells rsync directly to read the file.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: exclude a pattern but only in the top level

[Paul Slootman]

On Wed 15 May 2013, Brian K. White wrote:

 Consider the following directory structure
 
 /foo/aaa/*/*
 /foo/bbb/*/*
 /foo/ccc/*/*
 
 I want to sync all of /foo,
 but exclude /foo/aaa

 rsync -avz /foo ${DEST}::root

Firstly, I always recommend that with directory transfers you add a
trailing slash to the source, so your command becomes:

rsync -avz /foo/ ${DEST}::root/foo/

Then a filter would be:

- /aaa/
+ /*

Note the leading slash, as your aaa directory is now in the root of the
source. The second line is not really needed in this scenario IMHO

To be honest I'm wondering about your usage of -f . filter,
I always do --exclude-from=filter


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: exclude a pattern but only in the top level

[Paul Slootman]

On Wed 15 May 2013, Brian K. White wrote:

  rsync -avz /foo/ ${DEST}::root/foo/
 
 This syntax does work in his case, and is easier to read, because it
 ends up using the exact same specification /foo/ and /foo/ for
 both source and dest, but the syntax I had was also correct. I no
 longer remember why I always do the way I posted, but I've been
 doing it that way for decades (if you count rcp before rsync).

My reasoning is that if you're transferring a directory, make that
obvious by specifying a trailing slash. But use whatever you're happy
with, just don't complain if things get confusing (which it looked like
was the problem).


 The -f . filename syntax is correct it's right in the manual and
 I've been using it for ages.

Well, -f specifies a filter rule. You're using -f to specify a filter
rule to tell rsync to merge in a file with additional filter rules,
which is a bit of a roundabout way. Why not tell rsync directly to read
a file with include/exclude rules by using --exclude-from


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Compilation error during rsync-3.0.9

[Paul Slootman]

On Tue 30 Apr 2013, Gauri Senad wrote:
 
 I am compiling rsync-3.0.9 in my build system (which is for 2.6.10 linux
 kernel. I am using windriver build system), compilation is failing due to
 size of “uint32” and “uint16” was computed wrongly during configuration.

Hmm, it took some googling to see that windriver should be parsed as
wind-river and not win-driver :-)

 checking size of int... (cached) 4
 checking size of long... (cached) 4
 checking size of long long... (cached) 8
 checking size of short... (cached) 2
 checking size of int16_t... 2
 *checking size of uint16_t... (cached) 0*
 checking size of int32_t... 4
 *checking size of uint32_t... (cached) 0*
 checking size of int64_t... 8
 checking size of off_t... (cached) 8
 checking size of off64_t... 8
 checking size of time_t... 4
 checking for inline... (cached) inline

 As I am new to this kind of build system I am confused, which file is
 referred for checking the values of variable
 “ac_cv_sizeof_uint32_t/checking size of uint16_t”
 ? As far as my study during this error debugging it seems the value is
 looked from the cache file.

There should be no cache file present the first time you run configure.
If there is, wipe the directory clean and unpack the rsync sources
again, and check that there is no cache file at that point.
Then rerun configure.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: file corruption

[Paul Slootman]

On Mon 29 Apr 2013, Kevin Korb wrote:
 
 Simply put, if you have something that is screwing with your file data
 without touching your time stamps then you have been infected with a
 rootkit.  You should be thanking rsync for not backing up your

As he's running QNX I don't expect he's infected.

I've seen some windows programs update files but reset the mtime to the
original time, this may be the issue if he's running samba, for example.
We'd need to know what the differences are between the files. If they
are single bit changes then memory or system bus may be corrupting data.
If there are localized real changes then an application is more likely
to be the culprit.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Error 28 and exclusion

[Paul Slootman]

On Mon 11 Feb 2013, scott.geo...@parker.com wrote:
 
 Secondly, I am receiving an error 28 (ENOSPC) on the transfer of a large 
 file.  I make sure the destination file system is double what the source 
 for the possibility of needing to keep copies of both during transfer, but 
 I assumed --delete-before would help me avoid that. 

If a file still exists at the destination but is changed on the source,
then there will be at a point in time two copies of the file at the
destination: the original file and the newly copied file from the
source, under its temporary name. --delete-whatever doesn't come into it.

Only way to avoid that would be --inplace but that might be detrimental
to the speedup, as moved data might not be able to be used as it might
be written over during the transfer. Also the file's contents are not
correct during the transfer (by default the file will have either the
old or the new contents, but not a mix of the two).


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync - using a --files-from list to cut out scanning. How to handle deletions? (fwd)

[Paul Slootman]

On Fri 18 Jan 2013, Robert Bell wrote:
 
 If a file exists in the target directory when using --link-dest rsync
 modifies the link rather than replacing it which means you don't have
 history for files that have been replaced rather than added or deleted.
 Thanks for your astute observation about updating hard-linked
 files: you had me worried for a while.
 
 Fortunately, we are using the --whole-file option in our production
 backups, since the target of our backups is an HSM system (SGI's DMF),
 and we don't want rsync to start comparing files (and thus triggering a
 recall).  With this option, if a file is changed between the source and
 a target which contains a hard-linked version of the file, then the
 rsync update replaces the file in the target, not overwrites it and all
 its hard-linked cousins. Whew!

Rsync will not update an existing file in-place unless you use the
--inplace option. So --whole-file is irrelevant for this.
Rsync (without --inplace) will always create a new (temporary) file,
using the existing data (without --whole-file) to enable the delta diff
speedup algorithm. Once the temp file is successfully created, it's
renamed to the original name, deleting the existing link. So any
hardlinked data will remain untouched.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync without POSIX ACLs

[Paul Slootman]

On Thu 20 Dec 2012, Ryan  John wrote:
 
 The umask would be preferable to using --chmod. Is that possible?

Don't you just want to use -a instead of -r in your options to rsync?
That will include -p that will preserve permissions from the source.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync when using --whole-file

[Paul Slootman]

On Wed 19 Dec 2012, VanL wrote:

 I have a question about what happens at the code level when I use
 --whole-file. I know that it turns off the rolling checksum. I also
 understand that it only checks the file's mtime and size to identify
 whether there should be some transfer. Two questions:
 
 1) Could anyone give me a pointer to the correct file so that I can
 read what happens when --whole-file is used?

The manpage... If you're hard core you can read the source code :-)

 2) When using --whole-file, does rsync use any kind of hash function
 after transferring the file to make sure that the originating and
 newly copied file are the same?

From the manpage, admittedly mixed in with the --checksum option
description:

  Note that rsync always verifies that each transferred file was
  correctly reconstructed on the receiving side by checking a whole-file
  checksum that is generated as the file is transferred, but that
  automatic after-the-transfer verification has nothing to do with this
  option’s before-the-transfer Does this file need to be updated?
  check.

For the --whole-file case this won't be very meaningful as the data is
checksummed while writing to the file, which should always match the
checksum calculated while reading the source file...

For absolute security you can do a second run with --checksum.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync without POSIX ACLs

[Paul Slootman]

On Thu 20 Dec 2012, Ryan  John wrote:
 On Thu 20 Dec 2012, Ryan  John wrote:
 
  The umask would be preferable to using --chmod. Is that possible?
 
  Don't you just want to use -a instead of -r in your options to rsync?
  That will include -p that will preserve permissions from the source.
 
 That's the problem, the source permissions are - , which I don't 
 want/can't use in the destination.

In that case --chmod is what you want to turn bits on.
umask just defines what bits to turn OFF.

(Ignoring the fact that I'd say that having source perms - will
prevent rsync from reading the files in the first place...)


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --list-only ordering

[Paul Slootman]

On Mon 17 Dec 2012, Chris Dennis wrote:
 
 I've noticed an apparent inconsistency in the ordering of output
 from the --list-only option.
 
 For example:
   $ ls
   d1  d2  d2-x  d3  f1  f2  f2-x  f3
   $ rsync --list-only .
   drwxr-xr-x4096 2012/12/17 15:18:05 .
   -rw-r--r--   0 2012/12/17 15:17:52 f1
   -rw-r--r--   0 2012/12/17 15:17:52 f2
   -rw-r--r--   0 2012/12/17 15:17:52 f2-x
   -rw-r--r--   0 2012/12/17 15:17:52 f3
   drwxr-xr-x4096 2012/12/17 15:17:40 d1
   drwxr-xr-x4096 2012/12/17 15:17:40 d2-x
   drwxr-xr-x4096 2012/12/17 15:17:40 d2
   drwxr-xr-x4096 2012/12/17 15:17:40 d3
 
 Note that the files are listed in a 'sensible' order, with f2-x
 coming after f2.  But for the directories, d2-x comes before d2.

I wouldn't be surprised if rsync appends a slash to the directory names
internally; '-' sorts before '/' in ASCII.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync using huge traffic

[Paul Slootman]

On Mon 03 Dec 2012, Ben Oswald wrote:
 
 I've tested your advise for a few days now but is seems like the
 problem isn't solved. The traffice is now lower, about 15GB per backup,
 but still 4 times bigger than the whole data on the server. Is there an
 other option of rsync I can give a try.

I've never seen rsync transfer significantly more data over the wire
than the size of the data to transfer, so I can't help but think that
rsync is not at fault.

   The server I backup has 4GB of data and I use the following command
   to backup this data.  /usr/bin/rsync -aze 'ssh
   -i /root/.ssh/backup.key -l backupuser' --rsync-path='sudo rsync'
   --delete --exclude-from=ex.list $SRC $TRG

Check how many (large) files are hard-linked on the source.
If you have one 2GB file that has 7 hardlinks, then you will be
transferring that file 7 times as you don't use the -H option.

Secondly, you might try without the -z option.

Try it again with --stats to see what rsync's idea is of the amount of
data transferred etc.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync in daemon mode, no lock file generated

[Paul Slootman]

On Fri 14 Sep 2012, Dariusz Dolecki wrote:

 We are running rsync in daemon mode (::) (two colons), and in
 /etc.rsyncd.conf there is a lock file specified:
 log file = /var/adm/rsyncd.log
 pid file = /var/run/rsyncd.pid
 lock file = /var/run/rsync.lock
 
 
 But I do not see the lock file...

Are you having problems that you think is related to this?
Do you have max connections set?



Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Huge rsyncd.log file - what do I grep for to debug rscync failures

[Paul Slootman]

On Thu 06 Sep 2012, Kevin Korb wrote:
 
 Connection reset by peer essentially means that the program you were
 talking to went away (crashed).  Therefore whatever is at or near the

.. or a stateful firewall somewhere lost the state of the connection.
Sometimes happens when a connection is idle for too long, especially if
the firewall isn't very good or configured wrong (too few connection
tracking entries).


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Time rsYnc Machine (tym)

[Paul Slootman]

On Thu 09 Aug 2012, Linda Walsh wrote:
 
 Anyway, thanks for the history update.   I have a feeling rsync is afraid to 
 use
 memory -- and really, it should try to use alot of memory to optimize 
 transfers,

I have had rsync fail after using up 8GB memory + 4GB swap, so I'm very
happy it does its best to minimize memory usage.

 Turning off partial-send's (--whole-file) using 8-bit-io) seem to really help
 speed things up on a same-system copy... In doing a full sync with a backup,

--whole-file is the default when source and destination are both on the
same system -- at least, if you don't specify hostnames...

 of a 6G HD, (I used --drop-cache, --inplace and --del as well)  Doing an
 archive diff with --acls --xattrs + --hardlinks rysync averaged 125MB/s
 for the actual IO... (about 30% of the disk)...

OK, and now on a 6*TB* HD :-) This is on one of my backup servers:
/dev/sdc   39T   33T  5.9T  85% /backup


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsyncing huge file, timeout on rsyncd

[Paul Slootman]

On Thu 02 Aug 2012, Libor Klepáč wrote:

 Ok, i'l try setting timeout, but according to documentation, default is 
 timeout=0 which means no timeout.

IIRC setting the timeout explicitly on both ends (if you're using an
rsync daemon) to the same value will cause some sort of heartbeat
activity. Note also that an inferior firewall may also forget the
connection if there is no traffic for a longer period of time.

 Maybe setting --contimeout to some large value should help?

No, I'm quite sure that has nothing to do with this; you _had_ a
connection, which timed out. --contimeout is for the timeout for making
the initial connection.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: change UID+GID on target system?

[Paul Slootman]

On Mon 18 Jun 2012, Uwe Brauer wrote:
  On Mon, 18 Jun 2012 15:33:09 +0200, Paul Slootman wrote:
 
 This makes no sense.
 
 
 There is no UID option for mounting a jfs filesystem;
 that's only appropriate for e.g. VFAT where there is no concept of UID.
 After mounting a jfs filesystem the files on that filesystem will have
 the stored UID.
 
 That is what I meant, however it seems that there is a patch
 for the ext(2/3/4) file systems which allows you to mount a
 file system under a different  UID.

Do you have that patch installed in your kernel?

 And as far as I understand ext(3/4) are similar to jfs, so I
 don't understand what you mean it does not make sense.

jfs is totally NOT like ext[234].

So
- the mythical patch is most probably not installed
- even if it was, it does nothing for your jfs mount

Hence it makes no sense to expect this to work.


 So you could mount an USB system so that all the files have
 UID 1001 instead of the original 1000.

I can't imagine anyone making something that supports that.


Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html