Re: rsync whole file transfers extremely slow over SSH - but only in a particular virtual guest

[Paul Slootman via rsync]

On Wed 15 May 2024, Graham Leggett via rsync wrote:
> 
> Then we check the disk underneath rsync:
> 
> [root@arnie images]# dd if=/dev/urandom of=random.img count=1024 bs=10M 
> status=progress
> 1604321280 bytes (1.6 GB, 1.5 GiB) copied, 16 s, 100 MB/s^C
> 159+0 records in
> 159+0 records out
> 1667235840 bytes (1.7 GB, 1.6 GiB) copied, 16.7261 s, 99.7 MB/s. < fast 
> enough

I would try this again with the block size that rsync is using, which
will be way less than 10MB. It could be that the VM is limited in the
number of IOPS, which is slowing rsync down.

If that is the problem, using --whole-file might help as that stops
rsync "wasting" IOPS on reading the existing files, and may help in de
IO block size.

Using 'top' while rsync is running may help to see if rsync is IO bound,
the "wa" (wait IO) column will have a large percentage then.

You can also run strace to profile rsync to see where most wall
clock time is spent: strace -w -c
You'll have to do this on each rsync process.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Everything working as expected, so shouldn't ERROR be WARNING

[Paul Slootman via rsync]

On Thu 18 Jan 2024, Roger Price via rsync wrote:

> I am backing up a user's directories from local machine titan to remote
> machine maria. On the remote machine maria file /etc/rsyncd.conf contains
> 
>  [rprice-home]
>  ...
>  exclude = *.dvi
> 
> I start the backup by using this command on the local machine titan:
> 
>  rprice@titan ~ rsync -av --dry-run /mnt/home/rprice 
> rsync://rprice@maria/rprice-home
> 
> I get the messages
> 
>  sending incremental file list
>  ERROR: daemon refused to receive file "rprice/demo.dvi"
>  ...
> 
> I understand that the remote daemon has refused file demo.dvi because I
> specifically requested that dvi files not be transferred.  I choose that
> myself in a regular configuration file.  So shouldn't it be a WARNING rather
> than an ERROR?  I would expect to see

In this case you're in control of both ends of the transfer, so you know
that *.dvi files won't be transferred.

However, it could be that this rsync command is being run by someone who
expects rsync to do what they asked it to do, i.e. transfer the entire
contents of .../rprice to the remote server, and the client rsync can't
fulfil that request; hence the error.

If you don't want *.dvi files to be transferred, then you should add
--exclude '*.dvi'
to the invocation.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[Paul Slootman via rsync]

On Fri 22 Sep 2023, Kevin Korb via rsync wrote:

> 444 {} +' to make read only files for rsync to want to chmod, then used cp
> -al to make several duplicate trees using hard linked files.  An rm -rf on
> one such tree took .97 seconds while an rsync deletion took 1.25 seconds.

Be sure to drop the caches before such tests every time:
echo 3 > /proc/sys/vm/drop_caches


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Why try to update (some) permissions which are the same?

[Paul Slootman via rsync]

On Sun 03 Sep 2023, Perry Hutchison via rsync wrote:

> On the source system:
> 
> $ rsync --version
> rsync  version 2.6.8  protocol version 29

> On the destination system:

> $ rsync --version
> rsync  version 3.0.7  protocol version 30

The current version is 3.2.7, especially 2.6.8 is quite ancient.
You may want to upgrade before going bug hunting, chances are your
problem has already been fixed.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: What could cause rsync to kill ssh?

[Paul Slootman via rsync]

On Sat 03 Jun 2023, Maurice R Volaski via rsync wrote:

> I have an rsync script that it is copying one computer (over ssh) to a shared 
> CIFS mount on Gentoo Linux, kernel 6.3.4. The script runs for a while and 
> then at some point quits knocking my ssh session offline on all terminals and 
> it blocks ssh from being able to connect again. Even restarting sshd doesn’t 
> help. Rsync has apparently killed it. I have to reboot.

Note there's no such thing as an rsync script. You probably mean you
have a shell script that runs rsync at some point.

Is the script copying from the system it's running on, to the Gentoo
Linux system? Is the CIFS mount actually mounted on the Gentoo Linux, or
is the Gentoo Linux system serving the CIFS mount which actually is
mounted on the "one computer"? In that case it would be much better to
directly rsync to the filesystem on the Gentoo system.

Re: the ssh stopping working:
To me this would suggest that there's an out-of-memory situation going
on, and sshd is being killed because of this. However that would not
explain why restarting it doesn't work.
What exactly do you mean when you say restarting sshd doesn't help?
Does it not stay running, or is the daemon in fact running but not
accepting connections?
It's the age-old question: "it doesn't work" -- "_how_ is it not working?"

Does dmesg give any useful information? Or perhaps journalctl?
Usually the clues are in plain signt if you check logs.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Permission denied errors

[Paul Slootman via rsync]

On Mon 14 Nov 2022, Wes Render via rsync wrote:

> Hello, I'm running an rsync like this:
> 
> rsync -avSHP --delete-after 
> --log-file=/opt/mirrorsync/centos_mirror/rsync-1.log 
> --exclude-from=/opt/mirrorsync/centos_mirror/excludelist.txt 
> rsync://centos.mirror.rafal.ca/CentOS/ /data/centos
> 
> I'm running as a user that has full permissions on /data/ and /data/centos, 
> but then I get the following errors, when the rsync runs the second time:
> 
> 2022/11/14 10:13:54 [68738] receiving file list
> 2022/11/14 10:13:56 [68738] rsync: opendir 
> "/7.9.2009/cloud/x86_64/openstack-train/repodata/.~tmp~" (in CentOS) failed: 
> Permission denied (13)
> 2022/11/14 10:13:56 [68738] rsync: opendir 
> "/7.9.2009/updates/x86_64/repodata/.~tmp~" (in CentOS) failed: Permission 
> denied (13)
> 2022/11/14 10:13:56 [68738] 198081 files to consider
> 2022/11/14 10:13:57 [68751] sent 62 bytes  received 15,071,081 bytes  
> 6,028,457.20 bytes/sec
> 2022/11/14 10:13:57 [68751] total size is 149,535,538,669  speedup is 9,921.98
> 2022/11/14 10:13:57 [68738] rsync error: some files/attrs were not 
> transferred (see previous errors) (code 23) at main.c(1819) [generator=3.2.3]
> 
> The strange thing is, if I login as this user, and do an: ls -la 
> /data/centos//7.9.2009/cloud/x86_64/openstack-train/repodata/.~tmp~

The error message is saying it can't open that file on the source;
testing access on your local (destination) system is irrelevant.

$ rsync 
'rsync://centos.mirror.rafal.ca/CentOS/7.9.2009/updates/x86_64/repodata/.~tmp~'
drwx--  4,096 2022/11/10 22:39:21 .~tmp~

$ rsync 
'rsync://centos.mirror.rafal.ca/CentOS/7.9.2009/updates/x86_64/repodata/.~tmp~/'
rsync: change_dir "/7.9.2009/updates/x86_64/repodata/.~tmp~" (in CentOS) 
failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 
23) at main.c(1819) [Receiver=3.2.3]


You might want to --exclude '.~tmp~'


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --daemon complains parsing nonexistant /etc/rsyncd.conf

[Paul Slootman via rsync]

On Sat 17 Sep 2022, Colton Lewis via rsync wrote:

> This is on a system where /etc/rsyncd.conf does not exist and goes away if 
> /etc/rsyncd.conf is an empty file.
> 
> Version: rsync  version 3.2.5  protocol version 31
> Command: rsync --daemon
> What happens: The program outputs "Failed to parse config file: 
> /etc/rsyncd.conf"
> What I expect: The program should run with the same default values it does 
> when /etc/rsyncd.conf is an empty file.

So what functionality does rsync as a daemon have if there is an empty
rsyncd.conf file?

IMHO rsync is correct in refusing to run with a missing rsyncd.conf.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Does rsync verify its writes?

[Paul Slootman via rsync]

On Tue 12 Jul 2022, Kevin Korb via rsync wrote:

> Rsync does not verify writes.  --checksum doesn't verify anything. Sounds
> like you want a file verification tool.  The simplest would be md5sum.

Running rsync --checksum directly after transferring your files will
verify that the files are written correctly (if the source hasn't
changed in the meantime). That might help to give some peace of mind.


Paul

> On 7/12/22 02:31, Mark Filipak via rsync wrote:
> > Hello. Does rsync verify its writes?
> > 
> > Re, 'info rsync'.
> > 
> > Maybe I just being stupid, but there's no mention of verification in the
> > 'DESCRIPTION' section, so despite the words in the 'OPTIONS' section,
> > '-c, --checksum' topic (which I may be misinterpreting), I assume rsync
> > does not verify except for the checksum directive.
> > 
> > I admit that I'm paranoid. ;-) ...Please clarify.
> > 
> > Regards, and Thanks,
> > Mark Filipak.
> > 
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   https://sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync Users and Groups

[Paul Slootman via rsync]

On Fri 24 Jun 2022, Kevin Korb via rsync wrote:

> Nope.  Rsync groups are not groups of users they are just @users with their
> own password.  I believe the @ just designates that you intend multiple
> people to have that password and use that username.

I think I have to disagree here.

The manpage for rsyncd.conf states:

   In addition to username matching, you can specify groupname matching
   via a '@' prefix. When using groupname matching, the authenticating
   username must be a real user on the system, or it will be assumed to
   be a member of no groups. For example, specifying "@rsync" will match
   the authenticating user if the named user is a member of the rsync
   group.

So the user used by the client rsync should exist on the system and
belong to the specified system group for any permissions that are given
to that group in rsyncd.conf to be applicable. The password will still
need to be set in the secrets file, no system passwords are used.

Check the rsyncd.conf manpage, heading "auth users".


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: exclude include path problems

[Paul Slootman via rsync]

On Mon 07 Feb 2022, Edwardo Garcia via rsync wrote:
> 
> Lets call module mirror
> this results in about 200 or more directories (projects), but I only want
> one of them, lets call it foo, the problem is foo/  has about 50
> directories, but the one and only one we want is bar, but bar also has
> dozens of directories - but we do want these ones.
> 
> so  we want server:mirrors/foo/bar/*
> 
> How do we place our rsync command?
> 
> we have tried all kinds of   --include=foo/bar/  --exclude=*
> servername:mirrors/
> we tried an include exclude include exclude  too, infact every conceivable

Why not just do:

rsync -a server:mirrors/foo/bar/ /local/path/to/foo/bar/


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Confused as to why rsync thinks time, owner and group of many files differ

[Paul Slootman via rsync]

On Thu 03 Feb 2022, Andy Smith via rsync wrote:

> sudo rsync -iPva \
> --inplace \
> --numeric-ids \
> --delete \
> /data/backup/rsnapshot/daily.0/cacti/ \
> root@koff:/data/backup/rsnapshot/daily.0/cacti/
> 
> ...
>5,258 100%5.78kB/s0:00:00 (xfr#1276, to-chk=1/43437)

Could you try the transfer like this?:

sudo rsync -ia \
--debug=OWN,TIME \
--inplace \
--numeric-ids \
--delete \
/data/backup/rsnapshot/daily.0/cacti/var/www/index.html \
root@koff:/data/backup/rsnapshot/daily.0/cacti/var/www/

That should give detailed information about ownership and modification
times, limiting the transfer to just that index.html file to limit the
amount of output.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: trailing spaces in exclude-from file

[Paul Slootman via rsync]

On Mon 24 Jan 2022, Jürgen Bausa via rsync wrote:
> 
> However, that's just a proposal. But the behavior of trailing spaces is 
> something I guess should be corrected.

Is it?
How would you otherwise specify a space that you *do* intend to be
relevant?


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to manage root<-->root rsync keeping permissions?

[Paul Slootman via rsync]

On Tue 03 Aug 2021, Chris Green via rsync wrote:

> Is there a way to copy (for example) the /etc hierarchy from one
> system to another preserving root ownership of files and without
> revealing root passwords all over the place?

Best way is to run an rsync daemon on the source system, and be sure to
use "uid = 0" so that the daemon reads the source as root.

> So, it's easy for the sending end to be run as root as it's going to be
> run by a script in /etc/cron.daily, so it can access all the files in
> /etc even if only readable by root.

Hmm I prefer to use "pull" mechanisms as that's more secure (harder to
screw up the destination).

So create a /etc/rsyncd.conf file with the appropriate config, something
like:

[etc]
  path = /etc
  read only = yes
  hosts allow = another-system
  uid = 0

If using systemd then enable and start the daemon:

systemctl enable rsync.service
systemctl start rsync.service

Then on another-system as root run rsync:

rsync -a one-system::etc/ /backups/etc/

I usually also use -H for hard links, but /etc usually won't have those.

You can also use an rsync password to make this a bit more secure so
that not everyone on another-system can read all of /etc from
one-system. Details in the manpage.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: '--address' option on client side.

[Paul Slootman via rsync]

On Fri 26 Mar 2021, Harry Mangalam via rsync wrote:
> 
> I'm trying to improve a parallel rsync wrapper called parsyncfp (pfp)  in
> response to a user request.  He wants rsync to emit data on multiple
> interfaces (one interface per rsync instance). From the man page it seems
> like the '--address' option would do that and in fact using it as such does
> not result in an error, but it also does not result in both interfaces
> being used, either from pfp or when launched directly from different shells.
> 
> My route (working from home) shows the 2 wlan interfaces up with
> different IP #s:
> wlp3s0: flags=4163  mtu 1500
>inet 192.168.1.223  netmask 255.255.255.0  broadcast 192.168.1.255
> ...
> 
> wlx9cefd5fb0bb5: flags=4163  mtu 1500
>inet 192.168.1.186  netmask 255.255.255.0  broadcast 192.168.1.255

As both interfaces are on the same network, the kernel will use one
interface to transmit data to that network.

> and route shows:
> $ route
> 
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric RefUse
> Iface
> default router.asus.com 0.0.0.0 UG60100
> wlx9cefd5fb0bb5
> default router.asus.com 0.0.0.0 UG60200

Here you see wlx9cefd5fb0bb5 has a lower metric, hence it will
preferentially be used.

You need to dive deeper into linux policy based routing, to force
traffic e.g. from a particular IP address out over a certain interface.

This is totally outside the scope of rsync; there's nothing rsync can do
to influence this. You need the 'ip rule' and 'ip route' commands.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync support in authprogs - feedback requested

[Paul Slootman via rsync]

On Thu 18 Feb 2021, Bri Hatch via rsync wrote:
> 
> We use authprogs for more than just rsync though, and want more granularity
> than rrsync can support. If you force rrsync for the ssh key via
> command="rrsync" then that key may only be used to run rsync, you can't
> also allow additional commands. From a CI/CD perspective it may be useful
> to have the client side rsync some files, restart some services, and not
> need to use separate keys for each.

I use post-xfer scripts defined in rsyncd.conf to do useful things after
transferring files. That works well.

But I do see that there could be a use for rsync support in authprogs.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is there any way to restore/create hardlinks lost in incremental backups?

[Paul Slootman via rsync]

On Thu 10 Dec 2020, Chris Green via rsync wrote:
> 
> Occasionally, because I've moved things around or because I've done
> something else that breaks things, the hard links aren't created as
> they should be and I get a very space consuming backup increment.
> 
> Is there any easy way that one can restore hard links in the *middle*
> of a series?  For example say I have:-
> 
> day1/pictures
> day2/pictures
> day3/pictures
> day4/pictures
> day5/pictures
> 
> and I notice that day4/pictures is using as much space as
> day1/pictures but all the others are relatively small, i.e.
> day2 day3 and day5 have correctly hard linked to the previous day but
> day4 hasn't.
> 
> It needs a tool that can scan day4, check a file is identical with the
> one in day3 then hardlink it without losing the link from day5.

If you have these files that are hardlinked:

day1/pictures/1.jpg
day2/pictures/1.jpg
day3/pictures/1.jpg

And these are hardlinked, but to a different inode:

day4/pictures/1.jpg
day5/pictures/1.jpg

then there is no way of linking the second group to the first in one
step; you will have to individually link day3/pictures/1.jpg to
day4/pictures/1.jpg and then day3/pictures/1.jpg (or
day4/pictures/1.jpg) to day5/pictures/1.jpg.

It's not like a group of directory entries that are hardlinked to one
inode are some sort of actual group; they just happen to be directory
entries that point to the same inode number. There is no other relation
between those directory entries.

So you will have to incrementally process each next day against the
previous day.


If I make a significant change in such a directory structure (e.g.
renaming a directory) I try to remember to do the same thing on the
backup which some say is wrong, but it saves a lot of space, like you
discovered :)


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: feature request: exclude from path

[Paul Slootman via rsync]

On Mon 03 Aug 2020, Matt Stevens via rsync wrote:

> So I've gotten excluding paths to work as a standalone command. When I paste
> this into a script however, it ignores the exclusions. Any advice?
> 
> rsync -aXvr --times --links
> --exclude={'*.vdi','*.vmdk','*.ova','*.qcow2','.config/discord/'}
> /home/path/ user@nas:/NAS/HOME/destination/
> 
> Are there supposed to be some kind of brackets around this?

Using these brackets with bash causes the --exclude to be repeated for
each value:

$ echo TEST: --exclude={'*.vdi','*.vmdk','*.ova','*.qcow2','.config/discord/'}
TEST: --exclude=*.vdi --exclude=*.vmdk --exclude=*.ova --exclude=*.qcow2 
--exclude=.config/discord/

I suspect you've pasted this into a shell script which does not start
with #!/bin/bash but perhaps #!/bin/sh so that the script is not run by
bash but e.g. ash or dash that don't do the braces expansion.

It's always best (especially in a script) to write it out, don't rely on
the shell for such things.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Removing folder at destination

[Paul Slootman via rsync]

On Thu 14 Nov 2019, Freddie Valdez via rsync wrote:
> 
> Rsync 2.6.9

Wow, that's ancient. Released 06 Nov 2006

> Mac OS interacting with Windows servers.
> 
> My Rsync command: rsync -rvz --exclude-from=exclude.txt source destination.
> 
> I copy large files from server A to server B excluding multiple directories
> which rsync quickly and beautifully executes.
> What I end up with at destination is these folders.
>   01_us_eng...
>  2_ASSETS
>   3_web
>4_print
> I manually then move the web/print folders into the 01_us_eng... folder and
> then I manually delete the assets folder.

Why not exclude 2_ASSETS from the first run, and then do a second run to
rsync the _contents_ of 2_ASSETS to the target 01_us_eng directory?

Add /01_us_eng/2_ASSETS/ to the exclude.txt file.

rsync -rvz --exclude-from=exclude.txt source destination
rsync -rvz source/01_us_eng/2_ASSETS/ destination/01_us_eng/

(perhaps add some variation of the exclude.txt file if you're excluding
stuff under the 2_ASSETS directory)

> My humble question to samba.org is this, can I add an rsync command to move
> folders 3 and 4 into 01... and delete the 2_ASSETS folder so I dont have to
> manually do this 200 times each day?

You can't use rsync to perform remote rename operations, which is what
you're essentially asking for.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: hardlinking missing files from src to a dest: didn't work way I thought it would.

[Paul Slootman via rsync]

On Thu 14 Nov 2019, Pierre Bernhardt via rsync wrote:
> Am 14.11.19 um 10:54 schrieb Paul Slootman via rsync:
> > You need to specify the source directory as the link-dest directory.
> 
> Hi, I tried it also because it's an old question which has never worked
> for me. Instead it creates copies and not hard links:
> 
> 
> pierre@in94:~/tmp$ ls -li a b
> a:
> insgesamt 8
> 257315 -rw-r--r-- 1 pierre pierre 4 Nov 14 10:53 1
> 257316 -rw-r--r-- 1 pierre pierre 6 Nov 14 10:53 2
> 
> b:
> insgesamt 0
> pierre@in94:~/tmp$ rsync -av --link-dest=a a/ b/
> sending incremental file list
> --link-dest arg does not exist: a

There's your clue.
>From the manpage:

If DIR is a relative path, it is relative to the destination
directory.

So it's looking for b/a as the link-dest directory.

Use a full pathname for --link-dest to remove all uncertainty.
E.g.:

rsync -av --link-dest=$(pwd)/a a/ b/

In this case, as the destination is also in same current directory, you
could use:

rsync -av --link-dest=../a a/ b/


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: hardlinking missing files from src to a dest: didn't work way I thought it would.

[Paul Slootman via rsync]

On Thu 14 Nov 2019, L A Walsh via rsync wrote:

> Have a directory with a bunch rpms in it, mostly x86_64.
> 
> Have another directory with a bunch, mostly 'noarch'.
> 
> Some of the noarch files are already in the x86_64 dir
> and don't want to overwrite them.  They are on the same
> physical disk, so really, just want the new 'noarch' files
> hardlinked into the destination.
> 
> sitting in the noarch dir, I tried:
> rsync -auv --ignore-existing  \
>   --link-dest=/tumbleweed/. . /tumbleweed/.

This is not going to do anything useful, as you're telling it to look in
/tumbleweed/ for files that are to be placed in /tumbleweed/ i.e. the
exact same location. It's not going to do

ln /tumbleweed/foo/bar /tumbleweed/foo/bar

which is effectively what you're telling it to do.

You need to specify the source directory as the link-dest directory.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Seemingly impossible bug: -v not always listing every copied file

[Paul Slootman via rsync]

On Wed 30 Oct 2019, raf via rsync wrote:
> 
> I have a task that rsyncs files from a list of
> candidate files (--files-from=). It's verbose (-v) and

It would be helpful to show the complete rsync command line.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: checksum feature request

[Paul Slootman via rsync]

On Tue 01 Oct 2019, Bill Wichser via rsync wrote:
> 
> Attached is the patch we applied.  Since xxhash is in the distro, a
> dependency would be required for this RPM.  If nothing else, perhaps the
> developers should just take a look as this could benefit many.

"The distro" is a bit vague for a tool like rsync that runs on many
versions of Unix and linux, and even windows.

The problem is (AFAIK) that this would need a protocol version bump so
that the checksum algorithm to be used can be decided upon by both ends
of the transfer, it's not as simple as simply replacing the current
algorithm: that would make it impossible to rsync to / from an older
version of rsync.

It's an interesting idea, although I wonder how many users would
actually profit from this. CPU is generally fast enough to handle what
the IO subsystem can read for most people, I imagine.

Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync rewrites all blocks of large files although it uses delta transfer

[Paul Slootman via rsync]

On Thu 14 Feb 2019, Delian Krustev via rsync wrote:
> On Wednesday, February 13, 2019 6:25:59 PM EET Remi Gauvin 
>  wrote:
> > If the --inplace delta is as large as the filesize, then the
> > structure/location of the data has changed enough that the whole file
> > would have to be written out in any case.
> 
> This is not the case.
> If you see my original post you would have noticed that the delta transfer 
> finds only about 20 MB of differences within the almost 2G datafile.

I think you're missing the point of Remi's message.

Say the original file is:

ABCDEFGHIJ

The new file is:

XABCDEFGHI

Then the delta is just 10%, but the entire file needs to be rewritten as
the structure is changed.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: linux rsync <-> SSHDroid has started becoming unreliable after an upgrade of Fedora 28 to 29

[Paul Slootman via rsync]

On Sun 03 Feb 2019, Philip Rhoades via rsync wrote:
> 
> For some years I have been using rsync quite happily to send / retrieve
> files to / from SSHDroid Pro but recently I have started having a problem
> when transferring large numbers of file - I am pretty sure it started after
> upgrading from Fedora x86_64 28 to 29 - but I am not 100% sure.  Below is

[...]

> Corrupted MAC on input.
> ssh_dispatch_run_fatal: Connection to 192.168.1.100 port 22: message
> authentication code incorrect

ssh's communication gets disrupted somehow, and stops the connection,
thus causing rsync to fail.

This is a problem with ssh, not with rsync. Try enabling ssh debug
options, and try using different ssh ciphers.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: bug: xattr filter rule treated as file filter rule on the remote side

[Paul Slootman via rsync]

On Sat 28 Apr 2018, Andras Nagy via rsync wrote:

> Summary: an xattr filter rule (e.g. --filter='-x! user.*’, which is suggested 
> by the documentation) is treated as a file filter rule on the remote side.

I think that you're missing the point that filter rules affect the list
of files to be transferred. This implies that it always applies to the
*sending* side, as that is the side that builds the list to be sent.

It does not matter if the sending side is local or remote, so using that
in the description of the problem is wrong.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: What does this mean? select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)

[Paul Slootman via rsync]

On Mon 22 Jan 2018, Kevin Korb via rsync wrote:

> From man 2 select:
> int select(int nfds, fd_set *readfds, fd_set *writefds,
> fd_set *exceptfds, struct timeval *timeout);
> 
> So, it is waiting for file descriptor #1 to become available with a 60
> second timeout which it is hitting.

Actually:

> > select(1, [0], [], NULL, {60, 0})

... it's waiting for file descriptor 0 to become readable.
The 1st argument 1 is "the highest-numbered file descriptor in any of
the three sets, plus 1".


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 12568] Integer overflow still exists in xattrs.c, leading to buffer overflow

[Paul Slootman via rsync]

On Sun 08 Oct 2017, just subscribed for rsync-qa from bugzilla via rsync wrote:
> 
> --- Comment #1 from Wayne Davison  ---
> I've committed a fix for this into git. Many thanks for pointing this out.
> Sorry for how slow I've been lately.

Hey, I'm just happy you're still around :)


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync got stuck

[Paul Slootman via rsync]

On Tue 05 Sep 2017, Vangelis Katsikaros via rsync wrote:
> On 08/30/2017 05:39 PM, Paul Slootman wrote:
> > On Wed 23 Aug 2017, Vangelis Katsikaros via rsync wrote:
> > 
> >> abc   3797  3796  0 01:12 ?00:03:14 /usr/bin/rsync --compress 
> >> --compress-level=9 --bwlimit=512k --recursive --delay-updates --quiet 
> >> --update --exclude=/.* /SRC_PATH/ DEST_IP:/DEST_PATH/
> > 
> > Try running rsync without the --compress option, that has been the
> > source of problems in the past.
> > If you do need compression you could add that at the ssh level.
> 
> Thanks Paul! I've given it a go, and I'll see how it goes.
> 
> For reference, regarding the --compress options do you have any specific 
> bug(s) in mind or is this something you have noticed in practice?

I've never encountered it but there have been reports of --compress
causing such hangs which go away when --compress is removed.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to set absolute path for rsync?

[Paul Slootman via rsync]

On Tue 15 Aug 2017, Joe Qiao via rsync wrote:

> Thanks so much for the quick reply, Kevin!
> 
> I tried with ssh and --partial-dir, it looks the partial file still will be
> stored in local dir, but not in /tmp.
> 
> 
> Every 1.0s: ls -al /home/joe/rsync/ /tmp/
> Tue Aug 15 17:29:30 2017
> 
> /home/joe/rsync/:
> total 408840
> drwxr-xr-x  2 root root  4096 Aug 15 17:29 .
> drwxr-xr-x 32 joe  joe   4096 Aug 15 15:01 ..
> *-rw---  1 root root 418643968 Aug 15 17:29 .flash_image.14WoMV

No, you misunderstand the meaning of "partial" here.

--partial-dir is where the partially transferred file is stored if rsync
is interrupted with --partial (I'm not sure if --partial-dir implies
--partial, it probably does).

It does NOT mean where the temporary files are stored during transfer.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Clarifications on getting debug information when rsync freezes

[Paul Slootman via rsync]

On Wed 16 Aug 2017, Vangelis Katsikaros via rsync wrote:
> 
> I am having a problem with rsync freezing and I would like to collect the 
> proper information while the problem happens. However, I would like to ask 
> some clarifications.
> 
> rsync-debug
> ===
> 
> I see references of using rsync-debug but I cannot figure out how to use it 
> *while* the rsync is stuck. If I understand from various replies in this 
> email list it must be given as an argument to rsync beforehand?
> 
> If I do not want to change something in my current rsync setup, would it be 
> ok to do sth like this on the destination machine:
> 
> # 1. find the ssh PID from the source IP
> destination_machine $ sudo netstat -atlp | grep "192.168.23"
> tcp0 36 192.168.40.23:ssh   192.168.23.40:49187 
> ESTABLISHED 915/sshd: abc [priv
> 
> # 2. find all related processes
> destination_machine $ sudo pstree --show-pids 915
> sshd(915)───sshd(1079)───rsync(1082)───rsync(1085)
> 
> # 3. then strace these all these

Well, you'll miss anything leading up to the hang, but doing it this may
show something significant, but perhaps not.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync got stuck

[Paul Slootman via rsync]

On Wed 23 Aug 2017, Vangelis Katsikaros via rsync wrote:

> abc   3797  3796  0 01:12 ?00:03:14 /usr/bin/rsync --compress 
> --compress-level=9 --bwlimit=512k --recursive --delay-updates --quiet 
> --update --exclude=/.* /SRC_PATH/ DEST_IP:/DEST_PATH/

Try running rsync without the --compress option, that has been the
source of problems in the past.
If you do need compression you could add that at the ssh level.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Bug: rsync erroneously changes modification time

[Paul Slootman via rsync]

On Mon 12 Jun 2017, max.power--- via rsync wrote:

> How exactly does rsync determine that the copy has the incorrect timestamp
> and not the source file?

The source by definition is correct.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html