Re: Diff between using "-i" and not using "-i"

[- via rsync]

Hi,

thanks for the information. Shall I file a bug report at
https://github.com/RsyncProject/rsync/issues ?

Yours

lopiuh

Am Mo., 3. Juni 2024 um 15:57 Uhr schrieb Paul Slootman via rsync
:
>
> On Mon 03 Jun 2024, Kevin Korb via rsync wrote:
>
> > It appears that xattr changes (that is what the x mens) never made it into
> > the verbose output.  I would call that a bug but I would rather it be fixed
> > by making -v include -i.
>
> I think -v only shows when file data is transferred (or files /
> directories created / deleted).
> Just a permission change is also not shown with -v, but is with -i.
>
>
> Paul
>
> --

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is it possible to use rsync daemon running on 64-bit machine from a 32-bit machine?

[Chris Green via rsync]

Paul Slootman via rsync  wrote:
> On Sun 02 Jun 2024, Chris Green via rsync wrote:
> 
> > I have an rsync daemon running on a 64-bit (x86_64) system which I
> > successfully use for backups from several other 64-bit systems on my
> > LAN.
> > 
> > I want to use it for backups from a BeagleBone Black (32-bit, armv7l)
> > but it fails as follows:-
> > 
> > root@bbb:~# rsync -a /etc chris@backup::bbb
> > Password: 
> > pre-xfer exec returned failure (256)
> > rsync error: requested action not supported (code 4) at
> > clientserver.c(1171) [Receiver=3.2.7]
> > rsync: [sender] read error: Connection reset by peer (104)
> > root@bbb:~# 
> > 
> > Is it simply not possible to do what I'm trying to do or is there some
> > way to tell the rsync daemon to work with connections from 32-bit
> > clients?
> 
> There should be no problem communicating between 32-bit and 64-bit
> systems with rsync.
> 
> Make sure that the versions of rsync aren't too different, you may run
> into problems with rsync options not being supported on the old one.
> 
> The error message says that there was a problem with the pre-xfer exec.
> Check what that's doing (in the daemon's rsyncd.conf), most probably
> that is the problem.
> 
Yes, I saw that the error is in the 'pre-xfer exec' but it's exactly
the same as works with all the 64-bit clients.  I.e. it's the same
rsyncd and configuration on the server.

 after a lot of digging around I found the problem.  The first
pass of the pre-xfer exec generates a warning error message which was
failing because it was trying to send it to a system that no longer
exists.  The 64-bit/32-bit difference was a red herring, it was just
that the 64-bit systems had been doing backups for a while so didn't
generate any error messages (and so no mail failures).

Thanks for making me dig further! :-)

-- 
Chris Green
·


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is it possible to use rsync daemon running on 64-bit machine from a 32-bit machine?

[Paul Slootman via rsync]

On Sun 02 Jun 2024, Chris Green via rsync wrote:

> I have an rsync daemon running on a 64-bit (x86_64) system which I
> successfully use for backups from several other 64-bit systems on my
> LAN.
> 
> I want to use it for backups from a BeagleBone Black (32-bit, armv7l)
> but it fails as follows:-
> 
> root@bbb:~# rsync -a /etc chris@backup::bbb
> Password: 
> pre-xfer exec returned failure (256)
> rsync error: requested action not supported (code 4) at
> clientserver.c(1171) [Receiver=3.2.7]
> rsync: [sender] read error: Connection reset by peer (104)
> root@bbb:~# 
> 
> Is it simply not possible to do what I'm trying to do or is there some
> way to tell the rsync daemon to work with connections from 32-bit
> clients?

There should be no problem communicating between 32-bit and 64-bit
systems with rsync.

Make sure that the versions of rsync aren't too different, you may run
into problems with rsync options not being supported on the old one.

The error message says that there was a problem with the pre-xfer exec.
Check what that's doing (in the daemon's rsyncd.conf), most probably
that is the problem.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Is it possible to use rsync daemon running on 64-bit machine from a 32-bit machine?

[Chris Green via rsync]

I have an rsync daemon running on a 64-bit (x86_64) system which I
successfully use for backups from several other 64-bit systems on my
LAN.

I want to use it for backups from a BeagleBone Black (32-bit, armv7l)
but it fails as follows:-

root@bbb:~# rsync -a /etc chris@backup::bbb
Password: 
pre-xfer exec returned failure (256)
rsync error: requested action not supported (code 4) at
clientserver.c(1171) [Receiver=3.2.7]
rsync: [sender] read error: Connection reset by peer (104)
root@bbb:~# 

Is it simply not possible to do what I'm trying to do or is there some
way to tell the rsync daemon to work with connections from 32-bit
clients?

-- 
Chris Green

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Diff between using "-i" and not using "-i"

[Paul Slootman via rsync]

On Mon 03 Jun 2024, Kevin Korb via rsync wrote:

> It appears that xattr changes (that is what the x mens) never made it into
> the verbose output.  I would call that a bug but I would rather it be fixed
> by making -v include -i.

I think -v only shows when file data is transferred (or files /
directories created / deleted).
Just a permission change is also not shown with -v, but is with -i.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Diff between using "-i" and not using "-i"

[Kevin Korb via rsync]

It appears that xattr changes (that is what the x mens) never made it 
into the verbose output.  I would call that a bug but I would rather it 
be fixed by making -v include -i.


On 6/3/24 05:22, - via rsync wrote:

Hi,

Version: sync  version 3.3.0  protocol version 31

If I do a 

I get:
=
sending incremental file list
var/log/journal/

sent 263,31K bytes  received 1,14K bytes  528,90K bytes/sec
total size is 294,92M  speedup is 1.115,21 (DRY RUN)
=

including option <-i>:

rsync -avhHXASDin --del  /mnt/ /foo/mnt/

I get:
=
sending incremental file list
.fx usr/bin/ping
.d...a. var/log/journal/

sent 263,31K bytes  received 1,14K bytes  528,90K bytes/sec
total size is 294,92M  speedup is 1.115,21 (DRY RUN)
=

How is that, that "usr/bin/ping" is not included / included in
results? That is no feature, is it?

Yours lopiuh



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Diff between using "-i" and not using "-i"

[- via rsync]

Hi,

Version: sync  version 3.3.0  protocol version 31

If I do a 

I get:
=
sending incremental file list
var/log/journal/

sent 263,31K bytes  received 1,14K bytes  528,90K bytes/sec
total size is 294,92M  speedup is 1.115,21 (DRY RUN)
=

including option <-i>:

rsync -avhHXASDin --del  /mnt/ /foo/mnt/

I get:
=
sending incremental file list
.fx usr/bin/ping
.d...a. var/log/journal/

sent 263,31K bytes  received 1,14K bytes  528,90K bytes/sec
total size is 294,92M  speedup is 1.115,21 (DRY RUN)
=

How is that, that "usr/bin/ping" is not included / included in
results? That is no feature, is it?

Yours lopiuh

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync whole file transfers extremely slow over SSH - but only in a particular virtual guest

[Paul Slootman via rsync]

On Wed 15 May 2024, Graham Leggett via rsync wrote:
> 
> Then we check the disk underneath rsync:
> 
> [root@arnie images]# dd if=/dev/urandom of=random.img count=1024 bs=10M 
> status=progress
> 1604321280 bytes (1.6 GB, 1.5 GiB) copied, 16 s, 100 MB/s^C
> 159+0 records in
> 159+0 records out
> 1667235840 bytes (1.7 GB, 1.6 GiB) copied, 16.7261 s, 99.7 MB/s. < fast 
> enough

I would try this again with the block size that rsync is using, which
will be way less than 10MB. It could be that the VM is limited in the
number of IOPS, which is slowing rsync down.

If that is the problem, using --whole-file might help as that stops
rsync "wasting" IOPS on reading the existing files, and may help in de
IO block size.

Using 'top' while rsync is running may help to see if rsync is IO bound,
the "wa" (wait IO) column will have a large percentage then.

You can also run strace to profile rsync to see where most wall
clock time is spent: strace -w -c
You'll have to do this on each rsync process.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync whole file transfers extremely slow over SSH - but only in a particular virtual guest

[Graham Leggett via rsync]

Hi all,

I am trying to get to the bottom of a strange rsync performance problem.

On a specific guest OS, and only on this guest OS, rsync is giving modem-like 
transfer speeds. This happens on delta transfers, and whole file transfers.


[root@arnie ~]# rsync -avz --progress --sparse 
arnie.example.com:/home/backup/example/cuttysark.example.com/var-lib-libvirt-images-snapshot/
 /home/backup/example/cuttysark.example.com/var-lib-libvirt-images-snapshot/
receiving incremental file list
./
.timestamp
  0 100%0.00kB/s0:00:00 (xfr#1, to-chk=23/25)
3cx.example.com.img
  8,589,934,592 100%  771.32kB/s3:01:15 (xfr#2, to-chk=22/25). < s l o w
machine.example.img
 14,458,254,200  29%  208.50kB/s   45:06:35  ^Crsync error: received SIGINT, 
SIGTERM, or SIGHUP (code 20) at rsync.c(644) [generator=3.1.3]
rsync: [generator] write error: Broken pipe (32) < s  l  o  w  e  r


[root@arnie images]# rsync --verbose --progress --compress --sparse 
--copy-devices --partial --whole-file --inplace 
blackadder.example.com:/dev/dm-8 test.img
dm-8
 47,825,017 -2147483648%  587.01kB/s   ??:??:??   < s l o w


The above transfer started off for a few seconds at full speed, but then 
suddenly dropped to treacle slow.


First, we check the network underneath rsync. We're tunnelling rsync through 
ssh, so we test bandwidth through ssh with perf over and ssh tunnel, and we get 
expected speed for this link.

[root@arnie ~]# iperf -c [::1]:5001 -V

Client connecting to ::1, TCP port 5001
TCP window size: 2.50 MByte (default)

[  1] local ::1 port 42324 connected with ::1 port 5001
^C[ ID] Interval   Transfer Bandwidth
[  1] 0.00-19.28 sec  11.9 MBytes  5.19 Mbits/sec. < fast enough


Then we check the disk underneath rsync:

[root@arnie images]# dd if=/dev/urandom of=random.img count=1024 bs=10M 
status=progress
1604321280 bytes (1.6 GB, 1.5 GiB) copied, 16 s, 100 MB/s^C
159+0 records in
159+0 records out
1667235840 bytes (1.7 GB, 1.6 GiB) copied, 16.7261 s, 99.7 MB/s. < fast 
enough


Then the RAM under the guest:

[root@arnie ~]# free -h
  totalusedfree  shared  buff/cache   available
Mem:  3.6Gi   2.0Gi   394Mi   3.0Mi   1.2Gi   
1.3Gi. < 1.3GB seems fine?
Swap: 923Mi89Mi   834Mi


Then we check rsync running on the hypervisor of this guest:


[root@emma images]# rsync --verbose --progress --compress --sparse 
--copy-devices --partial --whole-file --inplace 
blackadder.example.com:/dev/dm-8 test.img
dm-8
173,316,344 -2147483648%6.07MB/s   ??:??:??  < fast enough


On an identically sized guest in a different datacenter:

[root@arnie images]# rsync --verbose --progress --compress --sparse 
--copy-devices --partial --whole-file --inplace 
blackadder.example.com:/dev/dm-8 
/home/backup/example/blackadder.example.com/vg001-var_lib_libvirt_images-snapshot.img
dm-8
170,053,588 -2147483648%   10.64MB/s   ??:??:??  < fast enough


>From what I can see:

- The guest OS networking is fine, including bandwidth over ssh.
- The hypervisor rsync works fine.
- A guest OS in another datacenter works fine.
- CPU is not being maxed out, either on the source or target. iowait is not 
showing anything significant. RAM seems modest.
- rsync on the guest is v3.1.3 on Rocky8.


I am somewhat stuck.

Google is of no help, it's all "it might be this, it might be that", but I've 
eliminated everything I have found and still the problem remains.

Has anyone encountered anything like this before?

Regards,
Graham
--


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync as a de-duplication-only tool, using --link-dest

[Kevin Korb via rsync]

I don't believe that what you are asking for can be done with rsync.  At 
first thought you can't mix --ignore-existing with --ignore-non-existing 
as that would ignore everything.  Something would have to at least exist 
and not be ignored for rsync to link to it.


Anyway, for a laugh, I asked chatgpt to make something to do this. 
After I got my laugh I cleaned up some of the silly stuff it did and 
came up with this:


#!/bin/bash

# Define the directories to compare
dir1="$1"
dir2="$2"

# Recursively list all files in both directories
files1=$(find "$dir1" -type f)

# Loop through files in first directory
for file1 in $files1; do
# Get relative path of file1
rel_path="${file1#$dir1}"
file2="$dir2$rel_path"

# Check if file exists in the second directory
if [ -f "$file2" ]; then
# Get metadata of both files
metadata1=$(stat -c "%Y%s" "$file1")
metadata2=$(stat -c "%Y%s" "$file2")

# Compare metadata
if [ "$metadata1" -eq "$metadata2" ]; then
# Delete file1 and create a hard link to file2
# rm "$file1"
# ln "$file2" "$file1"
echo "Hard linked: $file2 to $File1"
# else
# echo "Different: $file1"
fi
fi
done

Note that I only tested it a little bit which is why anything actually 
destructive is commented.


On 5/1/24 19:34, B via rsync wrote:
Recently I was thinking about --link-dest= and if it was possible to use 
rsync to de-duplicate two nearly-identical directory structures.


Normally I would use a tool like hardlink, jdupes, or rdfind, but in 
this case the files are huge and numerous, so hashing them would take 
forever. I did a test run and these tools mostly choked to death after a 
few hours.


These directories were made using rsync in the first place, so I know 
the files are duplicate and I would be willing to use rsync's 
quick-check (path/filename, mtime, size) to assume uniqueness of the files.


My objective is to hard-link files with the same relative path/filename, 
mtime, and size. Nothing more. Files which are different should not be 
touched. Files which exist in the destination but not the source should 
not be deleted. Files which exist in the source but not the destination 
should not be transferred.


The problem is that I don't want to create any new files in the 
destination. That's the sticking point.


I thought maybe I could do something wacky like 'rsync -a 
--ignore-existing --ignore-non-existing --link-dest="../new/" old/ new', 
but that doesn't work. The existing files get ignored and nothing is 
linked.


Is there a way to do this with rsync?





--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync as a de-duplication-only tool, using --link-dest

[B via rsync]

Recently I was thinking about --link-dest= and if it was possible to use 
rsync to de-duplicate two nearly-identical directory structures.


Normally I would use a tool like hardlink, jdupes, or rdfind, but in 
this case the files are huge and numerous, so hashing them would take 
forever. I did a test run and these tools mostly choked to death after a 
few hours.


These directories were made using rsync in the first place, so I know 
the files are duplicate and I would be willing to use rsync's 
quick-check (path/filename, mtime, size) to assume uniqueness of the files.


My objective is to hard-link files with the same relative path/filename, 
mtime, and size. Nothing more. Files which are different should not be 
touched. Files which exist in the destination but not the source should 
not be deleted. Files which exist in the source but not the destination 
should not be transferred.


The problem is that I don't want to create any new files in the 
destination. That's the sticking point.


I thought maybe I could do something wacky like 'rsync -a 
--ignore-existing --ignore-non-existing --link-dest="../new/" old/ new', 
but that doesn't work. The existing files get ignored and nothing is linked.


Is there a way to do this with rsync?



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  4592aa77 More tweaks for Actions.
  from  8bc363cc Separate the builds and make Cygwin always run.

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit 4592aa770d51d5e83845b032feea1de441f03ee7
Author: Wayne Davison 
Date:   Wed Apr 10 13:12:52 2024 -0700

More tweaks for Actions.

- When a .github/workflows/*.yml file changes, skip running unaffected
  builds.
- We need git to be installed for git-version.h generation.

---

Summary of changes:
 .github/workflows/cygwin-build.yml  | 6 ++
 .github/workflows/freebsd-build.yml | 8 +++-
 .github/workflows/macos-build.yml   | 6 ++
 .github/workflows/solaris-build.yml | 8 +++-
 .github/workflows/ubuntu-build.yml  | 6 ++
 5 files changed, 32 insertions(+), 2 deletions(-)


Changeset truncated at 500 lines:

diff --git a/.github/workflows/cygwin-build.yml 
b/.github/workflows/cygwin-build.yml
index a9635f2e..c6afb118 100644
--- a/.github/workflows/cygwin-build.yml
+++ b/.github/workflows/cygwin-build.yml
@@ -3,8 +3,14 @@ name: Test rsync on Cygwin
 on:
   push:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/cygwin-build.yml'
   pull_request:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/cygwin-build.yml'
   schedule:
 - cron: '42 8 * * *'
 
diff --git a/.github/workflows/freebsd-build.yml 
b/.github/workflows/freebsd-build.yml
index d82b160b..1ac22388 100644
--- a/.github/workflows/freebsd-build.yml
+++ b/.github/workflows/freebsd-build.yml
@@ -3,8 +3,14 @@ name: Test rsync on FreeBSD
 on:
   push:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/freebsd-build.yml'
   pull_request:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/freebsd-build.yml'
   schedule:
 - cron: '42 8 * * *'
 
@@ -22,7 +28,7 @@ jobs:
   with:
 usesh: true
 prepare: |
-  pkg install -y bash autotools m4 devel/xxhash zstd liblz4 python3 
archivers/liblz4
+  pkg install -y bash autotools m4 devel/xxhash zstd liblz4 python3 
archivers/liblz4 git
 run: |
   freebsd-version
   ./configure --with-rrsync -disable-zstd --disable-md2man 
--disable-xxhash --disable-lz4
diff --git a/.github/workflows/macos-build.yml 
b/.github/workflows/macos-build.yml
index bb85bb00..5471bf53 100644
--- a/.github/workflows/macos-build.yml
+++ b/.github/workflows/macos-build.yml
@@ -3,8 +3,14 @@ name: Test rsync on macOS
 on:
   push:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/macos-build.yml'
   pull_request:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/macos-build.yml'
   schedule:
 - cron: '42 8 * * *'
 
diff --git a/.github/workflows/solaris-build.yml 
b/.github/workflows/solaris-build.yml
index 557a5781..231fbd4a 100644
--- a/.github/workflows/solaris-build.yml
+++ b/.github/workflows/solaris-build.yml
@@ -3,8 +3,14 @@ name: Test rsync on Solaris
 on:
   push:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/solaris-build.yml'
   pull_request:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/solaris-build.yml'
   schedule:
 - cron: '42 8 * * *'
 
@@ -22,7 +28,7 @@ jobs:
   with:
 usesh: true
 prepare: |
-  pkg install bash automake gnu-m4 pkg://solaris/runtime/python-35 
autoconf gcc
+  pkg install bash automake gnu-m4 pkg://solaris/runtime/python-35 
autoconf gcc git
 run: |
   uname -a
   ./configure --with-rrsync -disable-zstd --disable-md2man 
--disable-xxhash --disable-lz4
diff --git a/.github/workflows/ubuntu-build.yml 
b/.github/workflows/ubuntu-build.yml
index 60dc8d5f..1db9a482 100644
--- a/.github/workflows/ubuntu-build.yml
+++ b/.github/workflows/ubuntu-build.yml
@@ -3,8 +3,14 @@ name: Test rsync on Ubuntu
 on:
   push:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/ubuntu-build.yml'
   pull_request:
 branches: [ master ]
+paths-ignore:
+  - '.github/workflows/*.yml'
+  - '!.github/workflows/ubuntu-build.yml'
   schedule:
 - cron: '42 8 * * *'
 


-- 
The rsync repository.

___
rsync-cvs mailing list
rsync-cvs@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-cvs

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  8bc363cc Separate the builds and make Cygwin always run.
   via  a9a31557 Work around pkg install issue.
  from  fcc79836 Get fetch-depth:0 right.

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit 8bc363cc9fcc57a352135e244497469ce9b2c4f8
Author: Wayne Davison 
Date:   Wed Apr 10 13:02:34 2024 -0700

Separate the builds and make Cygwin always run.

commit a9a315575631015dbee5eb74a986bf0c784830b4
Author: Wayne Davison 
Date:   Wed Apr 10 12:39:53 2024 -0700

Work around pkg install issue.

The xxhash, lz4, and zstd libraries aren't getting installed on FreeBSD.
[buildall]

---

Summary of changes:
 .github/workflows/build.yml | 126 
 .github/workflows/cygwin-build.yml  |  50 ++
 .github/workflows/freebsd-build.yml |  10 ++-
 .github/workflows/macos-build.yml   |  47 ++
 .github/workflows/solaris-build.yml |  18 +-
 .github/workflows/ubuntu-build.yml  |  50 ++
 6 files changed, 171 insertions(+), 130 deletions(-)
 delete mode 100644 .github/workflows/build.yml
 create mode 100644 .github/workflows/cygwin-build.yml
 create mode 100644 .github/workflows/macos-build.yml
 create mode 100644 .github/workflows/ubuntu-build.yml


Changeset truncated at 500 lines:

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
deleted file mode 100644
index f407dab7..
--- a/.github/workflows/build.yml
+++ /dev/null
@@ -1,126 +0,0 @@
-name: build
-
-on:
-  push:
-branches: [ master ]
-  pull_request:
-branches: [ master ]
-  schedule:
-- cron: '42 8 * * *'
-
-jobs:
-
-  ubuntu-build:
-runs-on: ubuntu-20.04
-steps:
-- uses: actions/checkout@v4
-  with:
-fetch-depth: 0
-- name: prep
-  run: |
-sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev 
libzstd-dev libxxhash-dev python3-cmarkgfm openssl
-echo "/usr/local/bin" >>$GITHUB_PATH
-- name: configure
-  run: ./configure --with-rrsync
-- name: make
-  run: make
-- name: install
-  run: sudo make install
-- name: info
-  run: rsync --version
-- name: check
-  run: sudo RSYNC_EXPECT_SKIPPED=crtimes make check
-- name: check30
-  run: sudo RSYNC_EXPECT_SKIPPED=crtimes make check30
-- name: check29
-  run: sudo RSYNC_EXPECT_SKIPPED=crtimes make check29
-- name: ssl file list
-  run: rsync-ssl --no-motd download.samba.org::rsyncftp/ || true
-- name: save artifact
-  uses: actions/upload-artifact@v3
-  with:
-name: ubuntu-bin
-path: |
-  rsync
-  rsync-ssl
-  rsync.1
-  rsync-ssl.1
-  rsyncd.conf.5
-  rrsync.1
-  rrsync
-
-  macos-build:
-runs-on: macos-latest
-steps:
-- uses: actions/checkout@v4
-  with:
-fetch-depth: 0
-- name: prep
-  run: |
-brew install automake openssl xxhash zstd lz4
-sudo pip3 install commonmark
-echo "/usr/local/bin" >>$GITHUB_PATH
-- name: configure
-  run: CPPFLAGS=-I/usr/local/opt/openssl/include/ 
LDFLAGS=-L/usr/local/opt/openssl/lib/ ./configure --with-rrsync
-- name: make
-  run: make
-- name: install
-  run: sudo make install
-- name: info
-  run: rsync --version
-- name: check
-  run: sudo 
RSYNC_EXPECT_SKIPPED=acls-default,chmod-temp-dir,chown-fake,devices-fake,dir-sgid,protected-regular,xattrs-hlink,xattrs
 make check
-- name: ssl file list
-  run: rsync-ssl --no-motd download.samba.org::rsyncftp/ || true
-- name: save artifact
-  uses: actions/upload-artifact@v3
-  with:
-name: macos-bin
-path: |
-  rsync
-  rsync-ssl
-  rsync.1
-  rsync-ssl.1
-  rsyncd.conf.5
-  rrsync.1
-  rrsync
-
-  cygwin-build:
-runs-on: windows-2022
-if: (github.event_name == 'schedule' || 
contains(github.event.head_commit.message, '[buildall]'))
-steps:
-- uses: actions/checkout@v4
-  with:
-fetch-depth: 0
-- name: cygwin
-  run: choco install -y --no-progress cygwin cyg-get
-- name: prep
-  run: |
-cyg-get make autoconf automake gcc-core attr libattr-devel python39 
python39-pip libzstd-devel liblz4-devel libssl-devel libxxhash0 libxxhash-devel
-echo "C:/tools/cygwin/bin" >>$Env:GITHUB_PATH
-- name: commonmark
-  run: bash -c 'python3 -mpip install --user commonmark'
-- name: configure
-  run: bash -c './configure --with-rrsync'
-- name: make
-  run: bash -c 'make'
-- name: install
-  run: bash -c 'make install'
-- name: info
-  run: bash -c '/usr/local/bin/rsync --version'
-- name: check
-  run: bash -c 

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  fcc79836 Get fetch-depth:0 right.
  from  804411b7 Get rid of gensend target & cached git version.

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit fcc79836b8f99bb9993b26de3a03cee14daf5ddf
Author: Wayne Davison 
Date:   Wed Apr 10 12:30:05 2024 -0700

Get fetch-depth:0 right.

---

Summary of changes:
 .github/workflows/build.yml | 9 ++---
 .github/workflows/freebsd-build.yml | 3 ++-
 .github/workflows/solaris-build.yml | 3 ++-
 3 files changed, 10 insertions(+), 5 deletions(-)


Changeset truncated at 500 lines:

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 09a8ce24..f407dab7 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -14,7 +14,8 @@ jobs:
 runs-on: ubuntu-20.04
 steps:
 - uses: actions/checkout@v4
-  fetch-depth: 0
+  with:
+fetch-depth: 0
 - name: prep
   run: |
 sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev 
libzstd-dev libxxhash-dev python3-cmarkgfm openssl
@@ -52,7 +53,8 @@ jobs:
 runs-on: macos-latest
 steps:
 - uses: actions/checkout@v4
-  fetch-depth: 0
+  with:
+fetch-depth: 0
 - name: prep
   run: |
 brew install automake openssl xxhash zstd lz4
@@ -88,7 +90,8 @@ jobs:
 if: (github.event_name == 'schedule' || 
contains(github.event.head_commit.message, '[buildall]'))
 steps:
 - uses: actions/checkout@v4
-  fetch-depth: 0
+  with:
+fetch-depth: 0
 - name: cygwin
   run: choco install -y --no-progress cygwin cyg-get
 - name: prep
diff --git a/.github/workflows/freebsd-build.yml 
b/.github/workflows/freebsd-build.yml
index 0fb5adb0..2c0061ee 100644
--- a/.github/workflows/freebsd-build.yml
+++ b/.github/workflows/freebsd-build.yml
@@ -12,7 +12,8 @@ jobs:
 name: Test rsync on FreeBSD
 steps:
 - uses: actions/checkout@v4
-  fetch-depth: 0
+  with:
+fetch-depth: 0
 - name: Test in FreeBSD
   id: test
   uses: vmactions/freebsd-vm@v1
diff --git a/.github/workflows/solaris-build.yml 
b/.github/workflows/solaris-build.yml
index feb4ad0b..7de3d35e 100644
--- a/.github/workflows/solaris-build.yml
+++ b/.github/workflows/solaris-build.yml
@@ -12,7 +12,8 @@ jobs:
 name: Test rsync on Solaris
 steps:
 - uses: actions/checkout@v4
-  fetch-depth: 0
+  with:
+fetch-depth: 0
 - name: Test in Solaris
   id: test
   uses: vmactions/solaris-vm@v1


-- 
The rsync repository.

___
rsync-cvs mailing list
rsync-cvs@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-cvs

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  804411b7 Get rid of gensend target & cached git version.
   via  0b1b2a3f Get the "dev" suffix right.
   via  50bdf968 Remove duplicate paragraph.
   via  3f2a38b0 CI: added Solaris build
  from  5510255f Tweak maintainer messaging.

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit 804411b7fd006d6a399a2b917b508259a752cfed
Author: Wayne Davison 
Date:   Wed Apr 10 12:15:49 2024 -0700

Get rid of gensend target & cached git version.

- Change the developer flow to not require updating the git-version repo
  that the builds used to download a git-version.h file. The Actions now
  do a full repo fetch so that the .h file can be generated via the git
  history.
- Get rid of the gensend Makefile target that was used for the above.
- Get rid of the pre-push git hook file that called "Make gensend".
- Change the FreeBSD build to save an artifact with its built binaries.

[buildall]

commit 0b1b2a3ff4b01b7ad6be5defe49172f04a0f7082
Author: Wayne Davison 
Date:   Wed Apr 10 11:53:07 2024 -0700

Get the "dev" suffix right.

commit 50bdf9685dc823ef9930c6ca862ac1588f815970
Author: Wayne Davison 
Date:   Wed Apr 10 11:51:59 2024 -0700

Remove duplicate paragraph.

commit 3f2a38b01184cae9a931280b534acf5a3dae2e94
Author: Charalampos Mitrodimas 
Date:   Mon Apr 8 11:40:02 2024 +0300

CI: added Solaris build

Signed-off-by: Charalampos Mitrodimas 

---

Summary of changes:
 .github/workflows/build.yml  | 16 
 .github/workflows/freebsd-build.yml  | 13 +++--
 .../workflows/{freebsd-build.yml => solaris-build.yml}   | 15 ---
 Makefile.in  |  8 
 README.md|  3 ---
 packaging/auto-Makefile  |  2 +-
 packaging/pkglib.py  | 11 ---
 packaging/pre-push   | 16 
 packaging/release-rsync  |  2 --
 version.h|  2 +-
 10 files changed, 29 insertions(+), 59 deletions(-)
 copy .github/workflows/{freebsd-build.yml => solaris-build.yml} (62%)
 delete mode 100755 packaging/pre-push


Changeset truncated at 500 lines:

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 9273e11a..09a8ce24 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -13,11 +13,11 @@ jobs:
   ubuntu-build:
 runs-on: ubuntu-20.04
 steps:
-- uses: actions/checkout@v3
+- uses: actions/checkout@v4
+  fetch-depth: 0
 - name: prep
   run: |
-sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev 
libzstd-dev libxxhash-dev python3-cmarkgfm openssl wget
-wget -O git-version.h 
https://gist.githubusercontent.com/WayneD/c11243fa374fc64d4e42f2855c8e3827/raw/rsync-git-version.h
+sudo apt-get install acl libacl1-dev attr libattr1-dev liblz4-dev 
libzstd-dev libxxhash-dev python3-cmarkgfm openssl
 echo "/usr/local/bin" >>$GITHUB_PATH
 - name: configure
   run: ./configure --with-rrsync
@@ -51,12 +51,12 @@ jobs:
   macos-build:
 runs-on: macos-latest
 steps:
-- uses: actions/checkout@v3
+- uses: actions/checkout@v4
+  fetch-depth: 0
 - name: prep
   run: |
-brew install automake openssl xxhash zstd lz4 wget
+brew install automake openssl xxhash zstd lz4
 sudo pip3 install commonmark
-wget -O git-version.h 
https://gist.githubusercontent.com/WayneD/c11243fa374fc64d4e42f2855c8e3827/raw/rsync-git-version.h
 echo "/usr/local/bin" >>$GITHUB_PATH
 - name: configure
   run: CPPFLAGS=-I/usr/local/opt/openssl/include/ 
LDFLAGS=-L/usr/local/opt/openssl/lib/ ./configure --with-rrsync
@@ -87,13 +87,13 @@ jobs:
 runs-on: windows-2022
 if: (github.event_name == 'schedule' || 
contains(github.event.head_commit.message, '[buildall]'))
 steps:
-- uses: actions/checkout@v3
+- uses: actions/checkout@v4
+  fetch-depth: 0
 - name: cygwin
   run: choco install -y --no-progress cygwin cyg-get
 - name: prep
   run: |
 cyg-get make autoconf automake gcc-core attr libattr-devel python39 
python39-pip libzstd-devel liblz4-devel libssl-devel libxxhash0 libxxhash-devel
-curl.exe -o git-version.h 
https://gist.githubusercontent.com/WayneD/c11243fa374fc64d4e42f2855c8e3827/raw/rsync-git-version.h
 echo "C:/tools/cygwin/bin" >>$Env:GITHUB_PATH
 - name: commonmark
   run: bash -c 'python3 -mpip install --user commonmark'
diff --git a/.github/workflows/freebsd-build.yml 

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  5510255f Tweak maintainer messaging.
   via  56a039b0 Changes for 3.3.1dev.
  from  7bc3be2b CI: fixed rules for when to trigger

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit 5510255f120ba13a1874086a1ef7ea8a7bf66570
Author: Wayne Davison 
Date:   Mon Apr 8 13:16:12 2024 -0700

Tweak maintainer messaging.

commit 56a039b04a186678e8e150a95529f23d93e2ca56
Author: Wayne Davison 
Date:   Mon Apr 8 13:14:59 2024 -0700

Changes for 3.3.1dev.

---

Summary of changes:
 NEWS.md  | 15 +++
 README.md|  8 +---
 rsync.1.md   |  3 +--
 rsyncd.conf.5.md |  3 +--
 version.h|  2 +-
 5 files changed, 23 insertions(+), 8 deletions(-)


Changeset truncated at 500 lines:

diff --git a/NEWS.md b/NEWS.md
index 846ed0ac..1a88a70d 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,3 +1,17 @@
+# NEWS for rsync 3.3.1 (UNRELEASED)
+
+## Changes in this version:
+
+### BUG FIXES:
+
+- Fixed the included popt to avoid a memory error on modern gcc versions.
+
+### INTERNAL:
+
+ - Updated included popt to version 1.19.
+
+--
+
 # NEWS for rsync 3.3.0 (6 Apr 2024)
 
 ## Changes in this version:
@@ -4762,6 +4776,7 @@
 
 | RELEASE DATE | VER.   | DATE OF COMMIT\* | PROTOCOL|
 |--||--|-|
+| ?? Apr 2024  | 3.3.1  |  | 31  |
 | 06 Apr 2024  | 3.3.0  |  | 31  |
 | 20 Oct 2022  | 3.2.7  |  | 31  |
 | 09 Sep 2022  | 3.2.6  |  | 31  |
diff --git a/README.md b/README.md
index f9689972..6bedf2e1 100644
--- a/README.md
+++ b/README.md
@@ -132,9 +132,11 @@ source.
 COPYRIGHT
 -
 
-Rsync was originally written by Andrew Tridgell and is currently
-maintained by Wayne Davison.  It has been improved by many developers
-from around the world.
+Rsync was originally written by Andrew Tridgell and has been improved by many
+developers from around the world.
+
+Rsync was originally written by Andrew Tridgell and Paul Mackerras.  Many
+people from around the world have helped to maintain and improve it.
 
 Rsync may be used, modified and redistributed only under the terms of
 the GNU General Public License, found in the file [COPYING][9] in this
diff --git a/rsync.1.md b/rsync.1.md
index afaf1de8..4407a013 100644
--- a/rsync.1.md
+++ b/rsync.1.md
@@ -4838,8 +4838,7 @@ David Bell.  I've probably missed some people, my 
apologies if I have.
 ## AUTHOR
 
 Rsync was originally written by Andrew Tridgell and Paul Mackerras.  Many
-people have later contributed to it. It is currently maintained by Wayne
-Davison.
+people from around the world have helped to maintain and improve it.
 
 Mailing lists for support and development are available at
 .
diff --git a/rsyncd.conf.5.md b/rsyncd.conf.5.md
index 2ba7..2f257659 100644
--- a/rsyncd.conf.5.md
+++ b/rsyncd.conf.5.md
@@ -1273,8 +1273,7 @@ Thanks to Karsten Thygesen for his many suggestions and 
documentation!
 ## AUTHOR
 
 Rsync was originally written by Andrew Tridgell and Paul Mackerras.  Many
-people have later contributed to it. It is currently maintained by Wayne
-Davison.
+people from around the world have helped to maintain and improve it.
 
 Mailing lists for support and development are available at
 .
diff --git a/version.h b/version.h
index b162146e..47d5dbe1 100644
--- a/version.h
+++ b/version.h
@@ -1,2 +1,2 @@
-#define RSYNC_VERSION "3.3.0"
+#define RSYNC_VERSION "3.3.1pre"
 #define MAINTAINER_TZ_OFFSET -7.0


-- 
The rsync repository.

___
rsync-cvs mailing list
rsync-cvs@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-cvs

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  7bc3be2b CI: fixed rules for when to trigger
   via  411c4789 support: added install_deps_ubuntu.sh
   via  231b239f check for stpcpy
   via  4c8683c8 update to popt 1.19
   via  85c906f9 Silence unused var warning
   via  35f5a21a hint that a proxy can handle plain and ssl stream at the 
same time
   via  99673f93 CI: added FreeBSD build
   via  9505ac59 removed old cirrus CI
   via  0dd25d47 configure.ac: fix failing IPv6 check due to missing return 
type
  from  ae3e13ba Update github links.

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit 7bc3be2b9e32cf95ce9d67c6227a8e5367d0b8ad
Author: Andrew Tridgell 
Date:   Mon Apr 8 15:35:42 2024 +1000

CI: fixed rules for when to trigger

commit 411c4789dfb2561d3400a5f8282f5f1bab76eeae
Author: Andrew Tridgell 
Date:   Mon Apr 8 10:16:31 2024 +1000

support: added install_deps_ubuntu.sh

convenient way to bootstrap quickly

commit 231b239f304fb2daa1240eec567880d225b7f730
Author: Andrew Tridgell 
Date:   Mon Apr 8 13:40:58 2024 +1000

check for stpcpy

needed for popt on macos

commit 4c8683c8753f2493ef13c3425fb5847960a6e305
Author: Andrew Tridgell 
Date:   Mon Apr 8 12:45:59 2024 +1000

update to popt 1.19

commit 85c906f96425195f8ba8646fbb37142206399ba0
Author: Rose <83477269+ataridre...@users.noreply.github.com>
Date:   Wed May 3 09:50:31 2023 -0400

Silence unused var warning

recv_ida_entries still needs to be called regardless, so we cannot take 
that out. Let's just quiet the compiler instead.

commit 35f5a21a16ca2689710e0503464bb736d136edd6
Author: Christian Hesse 
Date:   Wed Apr 5 13:08:02 2023 +0200

hint that a proxy can handle plain and ssl stream at the same time

commit 99673f937f36ab90a160a1b8c1e37645893d7b77
Author: Andrew Tridgell 
Date:   Sun Apr 7 07:11:31 2024 +1000

CI: added FreeBSD build

commit 9505ac59455278b54708b7c2790d93eb3d0b960b
Author: Andrew Tridgell 
Date:   Sun Apr 7 07:11:47 2024 +1000

removed old cirrus CI

commit 0dd25d4752520ed405315f1d2a8454fd507631bb
Author: Ivan Babrou 
Date:   Mon Jan 1 19:31:01 2024 -0800

configure.ac: fix failing IPv6 check due to missing return type

Fixing this warning escalated to an error, resuting in no IPv6 support:

```
configure.sh:7679: checking whether to enable ipv6
configure.sh:7718: clang -o conftest -g -O2 -DHAVE_CONFIG_H -Wall -W   
conftest.c  >&5
conftest.c:73:1: error: type specifier missing, defaults to 'int'; ISO C99 
and later do not support implicit int [-Wimplicit-int]
main()
^
int
1 error generated.
configure.sh:7718: $? = 1
configure.sh: program exited with status 1
```

---

Summary of changes:
 .cirrus.yml |   23 -
 .github/workflows/build.yml |2 -
 .github/workflows/freebsd-build.yml |   27 +
 INSTALL.md  |2 +
 Makefile.in |2 +-
 acls.c  |1 +
 configure.ac|6 +-
 popt/lookup3.c  |  959 +++
 popt/popt.c | 1447 +++
 popt/popt.h |  475 ++--
 popt/poptconfig.c   |  526 ++---
 popt/popthelp.c |  786 ++-
 popt/poptint.c  |  194 +
 popt/poptint.h  |  118 ++-
 popt/poptparse.c|  100 +--
 popt/system.h   |  148 +---
 rsyncd.conf.5.md|3 +
 support/install_deps_ubuntu.sh  |   11 +
 18 files changed, 3411 insertions(+), 1419 deletions(-)
 delete mode 100644 .cirrus.yml
 create mode 100644 .github/workflows/freebsd-build.yml
 create mode 100644 popt/lookup3.c
 create mode 100644 popt/poptint.c
 create mode 100755 support/install_deps_ubuntu.sh


Changeset truncated at 500 lines:

diff --git a/.cirrus.yml b/.cirrus.yml
deleted file mode 100644
index 33e2685e..
--- a/.cirrus.yml
+++ /dev/null
@@ -1,23 +0,0 @@
-freebsd_task:
-  name: FreeBSD
-  freebsd_instance:
-image_family: freebsd-13-1
-  env:
-PATH: /usr/local/bin:$PATH
-  prep_script:
-- dd if=/dev/zero of=/tmp/zpool bs=1M count=1024
-- zpool create -m `pwd`/testtmp zpool /tmp/zpool
-- pkg install -y bash autotools m4 xxhash zstd liblz4 wget
-- wget -O git-version.h 
https://gist.githubusercontent.com/WayneD/c11243fa374fc64d4e42f2855c8e3827/raw/rsync-git-version.h
-  configure_script:
-- CPPFLAGS=-I/usr/local/include/ LDFLAGS=-L/usr/local/lib/ ./configure 
--disable-md2man
-  make_script:
-- make
-  install_script:
-- make install
-  info_script:
-- rsync --version
-  test_script:
-- 

RE: Rsync 3.3.0 released

[Randall S. Becker via rsync]

Hi Wayne,

 

Just an FYI: RSync 3.3.0 built for HPE NonStop x86 and ia64 is now available on 
the ITUGLIB website (my team). We have supported that community and platform 
for many years. I am unsure how best to notify the RSync team about this.

 

Regards,

Randall

 

From: rsync  On Behalf Of Wayne Davison via rsync
Sent: Saturday, April 6, 2024 1:09 PM
To: rsync-annou...@lists.samba.org; rsync 
Subject: Rsync 3.3.0 released

 

I have released rsync version 3.3.0. This is a bug fix release, with the 
increased version bump being a delayed reaction to some of the recent larger 
changes that have happened.

 

To see a summary of all the recent changes, visit this link:

https://rsync.samba.org/ftp/rsync/NEWS#3.3.0

You can download the source tar file and its signature from here:

https://download.samba.org/pub/rsync/src-previews/rsync-3.3.0.tar.gz
https://download.samba.org/pub/rsync/src-previews/rsync-3.3.0.tar.gz.asc

See the website for other downloads, including diffs, patches, etc.:

https://rsync.samba.org/

 

The github repos have moved to a new RsyncProject organization. Because various 
life events have been monopolizing my time, I reached out to Tridge (the 
original author) and he has graciously agreed to get back into rsync work, 
along with Paul Mackerras, who was also an early contributor to rsync. This new 
team will be working mainly on maintenance tasks, and not so much on new 
features. If you want to get involved, feel free to reach out on the new 
discord RsyncProject channels.

 

For rsync on github and discord:

https://github.com/RsyncProject/rsync

https://discord.com/channels/1225946406041288736/1225946406041288739

..wayne..

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  ae3e13ba Update github links.
  from  6c8ca91c Preparing for release of 3.3.0 [buildall]

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit ae3e13ba99d0d6c1727ca7930b0eab5f60122ae0
Author: Wayne Davison 
Date:   Sat Apr 6 10:33:42 2024 -0700

Update github links.

---

Summary of changes:
 README.md   | 6 +++---
 configure.ac| 2 +-
 rsync.1.md  | 2 +-
 rsyncd.conf.5.md| 2 +-
 support/rrsync.1.md | 2 +-
 5 files changed, 7 insertions(+), 7 deletions(-)


Changeset truncated at 500 lines:

diff --git a/README.md b/README.md
index a86c7710..f9689972 100644
--- a/README.md
+++ b/README.md
@@ -34,7 +34,7 @@ If you need to build rsync yourself, check out the 
[INSTALL][1] page for
 information on what libraries and packages you can use to get the maximum
 features in your build.
 
-[1]: https://github.com/WayneD/rsync/blob/master/INSTALL.md
+[1]: https://github.com/RsyncProject/rsync/blob/master/INSTALL.md
 
 SETUP
 -
@@ -120,7 +120,7 @@ If you want to get the very latest version of rsync direct 
from the
 source code repository, then you will need to use git.  The git repo
 is hosted [on GitHub][6] and [on Samba's site][7].
 
-[6]: https://github.com/WayneD/rsync
+[6]: https://github.com/RsyncProject/rsync
 [7]: https://git.samba.org/?p=rsync.git;a=summary
 
 See [the download page][8] for full details on all the ways to grab the
@@ -140,5 +140,5 @@ Rsync may be used, modified and redistributed only under 
the terms of
 the GNU General Public License, found in the file [COPYING][9] in this
 distribution, or at [the Free Software Foundation][10].
 
-[9]: https://github.com/WayneD/rsync/blob/master/COPYING
+[9]: https://github.com/RsyncProject/rsync/blob/master/COPYING
 [10]: https://www.fsf.org/licenses/gpl.html
diff --git a/configure.ac b/configure.ac
index ccad7f13..0d868571 100644
--- a/configure.ac
+++ b/configure.ac
@@ -573,7 +573,7 @@ if test x"$no_lib" != x; then
 echo ""
 echo "See the INSTALL file for hints on how to install the missing 
libraries and/or"
 echo "how to generate (or fetch) manpages:"
-echo "https://github.com/WayneD/rsync/blob/master/INSTALL.md;
+echo "https://github.com/RsyncProject/rsync/blob/master/INSTALL.md;
 echo ""
 echo "To disable one or more features, the relevant configure options are:"
 for lib in $no_lib; do
diff --git a/rsync.1.md b/rsync.1.md
index 2ae6f481..afaf1de8 100644
--- a/rsync.1.md
+++ b/rsync.1.md
@@ -4818,7 +4818,7 @@ An rsync web site is available at 
.  The site
 includes an FAQ-O-Matic which may cover questions unanswered by this manual
 page.
 
-The rsync github project is .
+The rsync github project is .
 
 We would be delighted to hear from you if you like this program.  Please
 contact the mailing-list at .
diff --git a/rsyncd.conf.5.md b/rsyncd.conf.5.md
index cd10e659..ec976bac 100644
--- a/rsyncd.conf.5.md
+++ b/rsyncd.conf.5.md
@@ -1260,7 +1260,7 @@ Rsync is distributed under the GNU General Public 
License.  See the file
 [COPYING](COPYING) for details.
 
 An rsync web site is available at  and its github
-project is .
+project is .
 
 ## THANKS
 
diff --git a/support/rrsync.1.md b/support/rrsync.1.md
index 24892900..5f33930e 100644
--- a/support/rrsync.1.md
+++ b/support/rrsync.1.md
@@ -163,7 +163,7 @@ rsync is distributed under the GNU General Public License.  
See the file
 [COPYING](COPYING) for details.
 
 An rsync web site is available at  and its github
-project is .
+project is .
 
 ## AUTHOR
 


-- 
The rsync repository.

___
rsync-cvs mailing list
rsync-cvs@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-cvs

[rsync-announce] Rsync 3.3.0 released

[Wayne Davison via rsync-announce]

I have released rsync version 3.3.0. This is a bug fix release, with the
increased version bump being a delayed reaction to some of the recent
larger changes that have happened.

To see a summary of all the recent changes, visit this link:

https://rsync.samba.org/ftp/rsync/NEWS#3.3.0

You can download the source tar file and its signature from here:

https://download.samba.org/pub/rsync/src-previews/rsync-3.3.0.tar.gz
https://download.samba.org/pub/rsync/src-previews/rsync-3.3.0.tar.gz.asc

See the website for other downloads, including diffs, patches, etc.:

https://rsync.samba.org/

The github repos have moved to a new RsyncProject organization. Because
various life events have been monopolizing my time, I reached out to Tridge
(the original author) and he has graciously agreed to get back into rsync
work, along with Paul Mackerras, who was also an early contributor to
rsync. This new team will be working mainly on maintenance tasks, and not
so much on new features. If you want to get involved, feel free to reach
out on the new discord RsyncProject channels.

For rsync on github and discord:

https://github.com/RsyncProject/rsync
https://discord.com/channels/1225946406041288736/1225946406041288739

..wayne..
___
rsync-announce mailing list
rsync-announce@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-announce

Rsync 3.3.0 released

[Wayne Davison via rsync]

I have released rsync version 3.3.0. This is a bug fix release, with the
increased version bump being a delayed reaction to some of the recent
larger changes that have happened.

To see a summary of all the recent changes, visit this link:

https://rsync.samba.org/ftp/rsync/NEWS#3.3.0

You can download the source tar file and its signature from here:

https://download.samba.org/pub/rsync/src-previews/rsync-3.3.0.tar.gz
https://download.samba.org/pub/rsync/src-previews/rsync-3.3.0.tar.gz.asc

See the website for other downloads, including diffs, patches, etc.:

https://rsync.samba.org/

The github repos have moved to a new RsyncProject organization. Because
various life events have been monopolizing my time, I reached out to Tridge
(the original author) and he has graciously agreed to get back into rsync
work, along with Paul Mackerras, who was also an early contributor to
rsync. This new team will be working mainly on maintenance tasks, and not
so much on new features. If you want to get involved, feel free to reach
out on the new discord RsyncProject channels.

For rsync on github and discord:

https://github.com/RsyncProject/rsync
https://discord.com/channels/1225946406041288736/1225946406041288739

..wayne..
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[SCM] The rsync repository. - annotated tag v3.3.0 created

[Rsync CVS commit messages]

The annotated tag, v3.3.0 has been created
at  df60f6aa84e3ba9f251dde205f3ed03726b37137 (tag)
   tagging  6c8ca91c731b7bf2b081694bda85b7dadc2b7aff (commit)
  replaces  v3.3.0pre1
 tagged by  Wayne Davison
on  Sat Apr 6 09:38:26 2024 -0700

- Log -
Version 3.3.0.
-BEGIN PGP SIGNATURE-

iG8EABECAC8WIQQASMiwJtTJbw5YnC9shZ+xS5aoxQUCZhF6ghEcd2F5bmVkQHNh
bWJhLm9yZwAKCRBshZ+xS5aoxaDFAJ9PhspDNI6jUQrBjHGbHiCvUCznEwCg8xaJ
kBZO9GbiniI6ILawknW7kAI=
=86eP
-END PGP SIGNATURE-

Grant Gardner (1):
  typo in rsyncd.conf.5.md

Jiri Slaby (1):
  exclude: fix crashes with fortified strlcpy()

Wayne Davison (9):
  A couple spelling tweaks; tweak order.
  Mention updated config files.
  A couple more NEWS improvements.
  Fix old stats bug that counted devices as symlinks.
  Convert mnt-excl into python.
  Make `--max-alloc=0` safer.
  Mention latest changes in NEWS.
  Some year updates.
  Preparing for release of 3.3.0 [buildall]

zhangwenlong (1):
  update config.guess config.sub (#478)

---


-- 
The rsync repository.

___
rsync-cvs mailing list
rsync-cvs@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-cvs

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  6c8ca91c Preparing for release of 3.3.0 [buildall]
  from  079e74a3 Some year updates.

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit 6c8ca91c731b7bf2b081694bda85b7dadc2b7aff
Author: Wayne Davison 
Date:   Sat Apr 6 09:30:21 2024 -0700

Preparing for release of 3.3.0 [buildall]

---

Summary of changes:
 NEWS.md  |  4 ++--
 delete.c |  2 +-
 exclude.c|  2 +-
 packaging/lsb/rsync.spec | 10 +-
 util2.c  |  2 +-
 version.h|  2 +-
 6 files changed, 11 insertions(+), 11 deletions(-)


Changeset truncated at 500 lines:

diff --git a/NEWS.md b/NEWS.md
index da1e1852..846ed0ac 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,4 +1,4 @@
-# NEWS for rsync 3.3.0 (UNRELEASED)
+# NEWS for rsync 3.3.0 (6 Apr 2024)
 
 ## Changes in this version:
 
@@ -4762,7 +4762,7 @@
 
 | RELEASE DATE | VER.   | DATE OF COMMIT\* | PROTOCOL|
 |--||--|-|
-| ?? May 2023  | 3.3.0  |  | 31  |
+| 06 Apr 2024  | 3.3.0  |  | 31  |
 | 20 Oct 2022  | 3.2.7  |  | 31  |
 | 09 Sep 2022  | 3.2.6  |  | 31  |
 | 14 Aug 2022  | 3.2.5  |  | 31  |
diff --git a/delete.c b/delete.c
index dcb6a9af..89c1f8d6 100644
--- a/delete.c
+++ b/delete.c
@@ -4,7 +4,7 @@
  * Copyright (C) 1996-2000 Andrew Tridgell
  * Copyright (C) 1996 Paul Mackerras
  * Copyright (C) 2002 Martin Pool 
- * Copyright (C) 2003-2023 Wayne Davison
+ * Copyright (C) 2003-2024 Wayne Davison
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
diff --git a/exclude.c b/exclude.c
index 1a5de3b9..87edbcf7 100644
--- a/exclude.c
+++ b/exclude.c
@@ -4,7 +4,7 @@
  * Copyright (C) 1996-2001 Andrew Tridgell 
  * Copyright (C) 1996 Paul Mackerras
  * Copyright (C) 2002 Martin Pool
- * Copyright (C) 2003-2022 Wayne Davison
+ * Copyright (C) 2003-2024 Wayne Davison
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
diff --git a/packaging/lsb/rsync.spec b/packaging/lsb/rsync.spec
index 5df58b96..10385a39 100644
--- a/packaging/lsb/rsync.spec
+++ b/packaging/lsb/rsync.spec
@@ -1,9 +1,9 @@
 Summary: A fast, versatile, remote (and local) file-copying tool
 Name: rsync
 Version: 3.3.0
-%define fullversion %{version}pre1
-Release: 0.1.pre1
-%define srcdir src-previews
+%define fullversion %{version}
+Release: 1
+%define srcdir src
 Group: Applications/Internet
 License: GPL
 Source0: 
https://rsync.samba.org/ftp/rsync/%{srcdir}/rsync-%{fullversion}.tar.gz
@@ -79,8 +79,8 @@ rm -rf $RPM_BUILD_ROOT
 %dir /etc/rsync-ssl/certs
 
 %changelog
-* Sat Apr 29 2023 Wayne Davison 
-Released 3.3.0pre1.
+* Sat Apr 06 2024 Wayne Davison 
+Released 3.3.0.
 
 * Fri Mar 21 2008 Wayne Davison 
 Added installation of /etc/xinetd.d/rsync file and some commented-out
diff --git a/util2.c b/util2.c
index e398340e..b59bff0a 100644
--- a/util2.c
+++ b/util2.c
@@ -4,7 +4,7 @@
  * Copyright (C) 1996-2000 Andrew Tridgell
  * Copyright (C) 1996 Paul Mackerras
  * Copyright (C) 2001, 2002 Martin Pool 
- * Copyright (C) 2003-2023 Wayne Davison
+ * Copyright (C) 2003-2024 Wayne Davison
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
diff --git a/version.h b/version.h
index da4bb368..b162146e 100644
--- a/version.h
+++ b/version.h
@@ -1,2 +1,2 @@
-#define RSYNC_VERSION "3.3.0pre1"
+#define RSYNC_VERSION "3.3.0"
 #define MAINTAINER_TZ_OFFSET -7.0


-- 
The rsync repository.

___
rsync-cvs mailing list
rsync-cvs@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-cvs

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  079e74a3 Some year updates.
   via  abc3c746 Mention latest changes in NEWS.
   via  99ab5946 exclude: fix crashes with fortified strlcpy()
  from  a47ae6fa typo in rsyncd.conf.5.md

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit 079e74a30f3615ccd70864621dab6d8df0ae0122
Author: Wayne Davison 
Date:   Sat Apr 6 09:21:44 2024 -0700

Some year updates.

commit abc3c746527bb030db37010e03ef574ddc47fe36
Author: Wayne Davison 
Date:   Sat Apr 6 09:17:16 2024 -0700

Mention latest changes in NEWS.

commit 99ab59464bf44f18d668e373bc3d0f65190b87ac
Author: Jiri Slaby 
Date:   Fri Aug 18 08:26:20 2023 +0200

exclude: fix crashes with fortified strlcpy()

Fortified (-D_FORTIFY_SOURCE=2 for gcc) builds make strlcpy() crash when
its third parameter (size) is larger than the buffer:
  $ rsync -FFXHav '--filter=merge global-rsync-filter' Align-37-43/ xxx
  sending incremental file list
  *** buffer overflow detected ***: terminated

It's in the exclude code in setup_merge_file():
  strlcpy(y, save, MAXPATHLEN);

Note the 'y' pointer was incremented, so it no longer points to memory
with MAXPATHLEN "owned" bytes.

Fix it by remembering the number of copied bytes into the 'save' buffer
and use that instead of MAXPATHLEN which is clearly incorrect.

Fixes #511.

---

Summary of changes:
 NEWS.md   | 7 +++
 delete.c  | 2 +-
 exclude.c | 5 +++--
 latest-year.h | 2 +-
 util2.c   | 2 +-
 5 files changed, 13 insertions(+), 5 deletions(-)


Changeset truncated at 500 lines:

diff --git a/NEWS.md b/NEWS.md
index ca60c32c..da1e1852 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -10,6 +10,11 @@
 - Fixed an buffer overflow in the checksum2 code if SHA1 is being used for
   the checksum2 algorithm.
 
+- Fixed an issue when rsync is compiled using `_FORTIFY_SOURCE` so that the
+  extra tests don't complain about a strlcpy() limit value (which was too
+  large, even though it wasn't possible for the larger value to cause an
+  overflow).
+
 - Add a backtick to the list of characters that the filename quoting needs to
   escape using backslashes.
 
@@ -49,6 +54,8 @@
 - Changed the mapfrom & mapto perl scripts (in the support dir) into a single
   python script named idmap.  Converted a couple more perl scripts into python.
 
+- Changed the mnt-excl perl script (in the support dir) into a python script.
+
 ### DEVELOPER RELATED:
 
  - Updated config.guess (timestamp 2023-01-01) and config.sub (timestamp
diff --git a/delete.c b/delete.c
index 80766164..dcb6a9af 100644
--- a/delete.c
+++ b/delete.c
@@ -4,7 +4,7 @@
  * Copyright (C) 1996-2000 Andrew Tridgell
  * Copyright (C) 1996 Paul Mackerras
  * Copyright (C) 2002 Martin Pool 
- * Copyright (C) 2003-2020 Wayne Davison
+ * Copyright (C) 2003-2023 Wayne Davison
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
diff --git a/exclude.c b/exclude.c
index ffe55b16..1a5de3b9 100644
--- a/exclude.c
+++ b/exclude.c
@@ -720,7 +720,8 @@ static BOOL setup_merge_file(int mergelist_num, filter_rule 
*ex,
parent_dirscan = True;
while (*y) {
char save[MAXPATHLEN];
-   strlcpy(save, y, MAXPATHLEN);
+   /* copylen is strlen(y) which is < MAXPATHLEN. +1 for \0 */
+   size_t copylen = strlcpy(save, y, MAXPATHLEN) + 1;
*y = '\0';
dirbuf_len = y - dirbuf;
strlcpy(x, ex->pattern, MAXPATHLEN - (x - buf));
@@ -734,7 +735,7 @@ static BOOL setup_merge_file(int mergelist_num, filter_rule 
*ex,
lp->head = NULL;
}
lp->tail = NULL;
-   strlcpy(y, save, MAXPATHLEN);
+   strlcpy(y, save, copylen);
while ((*x++ = *y++) != '/') {}
}
parent_dirscan = False;
diff --git a/latest-year.h b/latest-year.h
index 0dcf3464..f978fb8b 100644
--- a/latest-year.h
+++ b/latest-year.h
@@ -1 +1 @@
-#define LATEST_YEAR "2023"
+#define LATEST_YEAR "2024"
diff --git a/util2.c b/util2.c
index 3b5a8f41..e398340e 100644
--- a/util2.c
+++ b/util2.c
@@ -4,7 +4,7 @@
  * Copyright (C) 1996-2000 Andrew Tridgell
  * Copyright (C) 1996 Paul Mackerras
  * Copyright (C) 2001, 2002 Martin Pool 
- * Copyright (C) 2003-2020 Wayne Davison
+ * Copyright (C) 2003-2023 Wayne Davison
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by


-- 
The rsync repository.

___
rsync-cvs mailing list
rsync-cvs@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-cvs

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  a47ae6fa typo in rsyncd.conf.5.md
  from  2f9b963a Make `--max-alloc=0` safer.

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit a47ae6fad901d94c1853fa27b56bc458a48bbee2
Author: Grant Gardner 
Date:   Sun Mar 17 14:00:16 2024 +1100

typo in rsyncd.conf.5.md

---

Summary of changes:
 rsyncd.conf.5.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


Changeset truncated at 500 lines:

diff --git a/rsyncd.conf.5.md b/rsyncd.conf.5.md
index 3d91cd93..cd10e659 100644
--- a/rsyncd.conf.5.md
+++ b/rsyncd.conf.5.md
@@ -1023,7 +1023,7 @@ in the values of parameters.  See that section for 
details.
 _not_ displayed if the script returns success.  The other programs cannot
 send any text to the user.  All output except for the `pre-xfer exec`
 stdout goes to the corresponding daemon's stdout/stderr, which is typically
-discarded.  See the `--no-detatch` option for a way to see the daemon's
+discarded.  See the `--no-detach` option for a way to see the daemon's
 output, which can assist with debugging.
 
 Note that the `early exec` command runs before any part of the transfer


-- 
The rsync repository.

___
rsync-cvs mailing list
rsync-cvs@lists.samba.org
https://lists.samba.org/mailman/listinfo/rsync-cvs

[SCM] The rsync repository. - branch master updated

[Rsync CVS commit messages]

The branch, master has been updated
   via  2f9b963a Make `--max-alloc=0` safer.
   via  3476caea Convert mnt-excl into python.
   via  6f3c5ecc Fix old stats bug that counted devices as symlinks.
   via  79fda353 A couple more NEWS improvements.
  from  cd769934 Mention updated config files.

https://git.samba.org/?p=rsync.git;a=shortlog;h=master


- Log -
commit 2f9b963abaa52e44891180fe6c0d1c2219f6686d
Author: Wayne Davison 
Date:   Tue Jun 27 09:01:15 2023 -0700

Make `--max-alloc=0` safer.

Always do size checking in my_alloc(), even for `--max-alloc=0`.

commit 3476caea3e10ec06b839d0e95b09c145dd3cbfaf
Author: Wayne Davison 
Date:   Mon May 22 08:29:15 2023 -0700

Convert mnt-excl into python.

commit 6f3c5eccee6cf4dead68b9f3fda8fc2ff90dc311
Author: Wayne Davison 
Date:   Tue May 16 22:44:54 2023 -0700

Fix old stats bug that counted devices as symlinks.

commit 79fda353425daba6b23753c8b1b01dc35ecaac7d
Author: Wayne Davison 
Date:   Thu May 4 08:56:10 2023 -0700

A couple more NEWS improvements.

---

Summary of changes:
 NEWS.md  |  9 ++---
 delete.c |  2 +-
 flist.c  |  2 +-
 options.c|  2 ++
 rsync.1.md   |  3 ++-
 support/mnt-excl | 48 +---
 util2.c  |  2 +-
 7 files changed, 42 insertions(+), 26 deletions(-)


Changeset truncated at 500 lines:

diff --git a/NEWS.md b/NEWS.md
index 2821a990..ca60c32c 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -13,8 +13,9 @@
 - Add a backtick to the list of characters that the filename quoting needs to
   escape using backslashes.
 
-- Fixed a string-comparison issue in the internal file-list code that affected
-  tr_TR.utf-8.
+- Fixed a string-comparison issue in the internal handling of `--progress` (a
+  locale such as tr_TR.utf-8 needed the internal triggering of `--info` options
+  to use upper-case flag names to ensure that they match).
 
 - Make sure that a local transfer marks the sender side as trusted.
 
@@ -25,10 +26,12 @@
   openssl library.
 
 - Fixed a problem in the daemon auth for older protocols (29 and before) if the
-  openssl library is being used to compute md4 checksums.
+  openssl library is being used to compute MD4 checksums.
 
 - Fixed `rsync -VV` on Cygwin -- it needed a flush of stdout.
 
+- Fixed an old stats bug that counted devices as symlinks.
+
 ### ENHANCEMENTS:
 
 - Enhanced rrsync with the `-no-overwrite` option that allows you to ensure
diff --git a/delete.c b/delete.c
index 4a294853..80766164 100644
--- a/delete.c
+++ b/delete.c
@@ -188,7 +188,7 @@ enum delret delete_item(char *fbuf, uint16 mode, uint16 
flags)
stats.deleted_symlinks++;
 #endif
else if (IS_DEVICE(mode))
-   stats.deleted_symlinks++;
+   stats.deleted_devices++;
else
stats.deleted_specials++;
}
diff --git a/flist.c b/flist.c
index 311bbcf1..464d556e 100644
--- a/flist.c
+++ b/flist.c
@@ -2659,7 +2659,7 @@ struct file_list *recv_file_list(int f, int dir_ndx)
} else if (S_ISLNK(file->mode))
stats.num_symlinks++;
else if (IS_DEVICE(file->mode))
-   stats.num_symlinks++;
+   stats.num_devices++;
else
stats.num_specials++;
 
diff --git a/options.c b/options.c
index 93bbe7b0..fd674754 100644
--- a/options.c
+++ b/options.c
@@ -1946,6 +1946,8 @@ int parse_arguments(int *argc_p, const char ***argv_p)
goto cleanup;
max_alloc = size;
}
+   if (!max_alloc)
+   max_alloc = SIZE_MAX;
 
if (old_style_args < 0) {
if (!am_server && protect_args <= 0 && (arg = 
getenv("RSYNC_OLD_ARGS")) != NULL && *arg) {
diff --git a/rsync.1.md b/rsync.1.md
index 894b3663..2ae6f481 100644
--- a/rsync.1.md
+++ b/rsync.1.md
@@ -2106,7 +2106,8 @@ expand it.
 See the [`--max-size`](#opt) option for a description of how SIZE can be
 specified.  The default suffix if none is given is bytes.
 
-Beginning in 3.2.3, a value of 0 specifies no limit.
+Beginning in 3.2.7, a value of 0 is an easy way to specify SIZE_MAX (the
+largest limit possible).
 
 You can set a default value using the environment variable
 [`RSYNC_MAX_ALLOC`](#) using the same SIZE values as supported by this
diff --git a/support/mnt-excl b/support/mnt-excl
index ed7b49ba..bc8b5bcd 100755
--- a/support/mnt-excl
+++ b/support/mnt-excl
@@ -1,4 +1,4 @@
-#!/usr/bin/env perl
+#!/usr/bin/env python3
 # This script takes a command-line arg of a source directory
 # that will be passed to rsync, and generates a set of excludes
 # that will exclude all mount 

ACLs vanish while transferring files

[Daniel Jordan via rsync]

Hi all,

I'm trying to transfer data from the old fileserver to the new cluster 
using rsync.
Depending on how I define the source path either the folder acls are 
transfered

or the file acls.

This command preserves the file acls:
# rsync -AavXz --numeric-ids --delete /winhome 
root@192.168.178.10:/glusterfs


with this command the folder acls are copied:
# rsync -AavXz --numeric-ids --delete /winhome/* 
root@192.168.178.10:/glusterfs/winhome/



Setup is the following:

OS: old: Debian 11, new: Debian 12

rsync Version on both hosts:

root@fs01:~# rsync --version
rsync  version 3.2.7  protocol version 31
Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
    hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs,
    xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes
Optimizations:
    SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none
Compress list:
    zstd lz4 zlibx zlib none
Daemon auth list:
    sha512 sha256 sha1 md5 md4

What i found out till now is that the rsync option -A should copy all 
acls, wether folder

or file, but it seems, that there are other things that have an impact.
What am i missing here? Any help is highly appreciated.


Best regards
Daniel

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 2294] Detect renamed files and handle by renaming instead of delete/re-send

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=2294

--- Comment #41 from Mihnea-Costin Grigore  ---
The discussion about file systems like ZFS/BTRFS/etc. and their various
snapshot mechanisms is off-topic relative to this feature request, since they
are very different technologies used for different purposes.

rsync is used commonly to synchronise at the *file level* between very
different operating systems -- from Linux to Windows, from macOS to Linux, etc.
It also has multiple features to allow filtering and selecting files within the
source and destination directories, thus only synchronising a subset of files.
We need a solution that works with the flexibility of rsync itself, and
snapshots (useful as they are) do not fit that at all.

This would be a very useful feature to have as part of rsync, made evident by
the many requests both here and on other discussion forums over the past almost
20 (!) years. Can we rather discuss what the blockers are for the existing
patches? What stops them from inclusion -- is it quality, functionality,
compatibility, test coverage, something else? Working to fix that and making
the already existing patches acceptable for inclusion would appear to be the
most constructive course of action, rather than deflecting the request.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync segfaults when openssl fips is enabled

[Shreenidhi Shedi via rsync]

This happens with rsync-3.2.4, upgraded to v3.2.7 and this is solved.
Thanks.

--
Shedi


On Tue, Mar 12, 2024 at 3:05 PM Shreenidhi Shedi <
shreenidhi.sh...@broadcom.com> wrote:

> Hi All,
>
> Any inputs on this issue?
>
> --
> Shedi
>
>
> On Wed, Feb 21, 2024 at 5:12 PM Shreenidhi Shedi <
> shreenidhi.sh...@broadcom.com> wrote:
>
>> Hi All,
>>
>> Copying the content from the GH issue as is.
>> Need your inputs on the same.
>> FWIW, the coredump files generated in linux have xattr values which are >
>> 32 bytes.
>>
>> https://github.com/WayneD/rsync/issues/569
>>
>> Hi All,
>>
>> System details:
>>
>> root@ph3dev [ ~ ]# rsync --version
>> rsync  version 3.2.4  protocol version 31
>> Capabilities:
>> 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
>> socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
>> hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, no ACLs,
>> xattrs, optional protect-args, iconv, prealloc, stop-at, no 
>> crtimesOptimizations:
>> SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
>> Checksum list:
>> xxh128 xxh3 xxh64 (xxhash) md5 md4 none
>> Compress list:
>> zstd lz4 zlibx zlib none
>>
>> OS:
>> Photon OS 3.0 with openssl fips enabled
>>
>> I'm using rsync like below:
>>
>> rsync -arXHpvxog  
>>
>> While doing so, if any file in src or dest location has a xattr value which 
>> is more than 32 bytes long, rsync segfaults.
>>
>> root@ph3dev [ ~ ]# getfattr -d -m '' -- /opt/x
>> getfattr: Removing leading '/' from absolute path names
>> # file: opt/x
>> user.bla1="1234567890abcdef1234567890abcdef1" > 33 bytes long
>>
>> After examining the coredump file, this happens at;
>>
>> rsync/xattrs.c
>> Line 277 in 2f9b963
>>
>> sum_init(xattr_sum_nni, checksum_seed);
>>
>> Here md5 is used by default and md5 usage is prohibited in fips mode.
>>
>> Is there any way to workaround this problem? Also, why not use xxhash for 
>> this operation like it's used by default during --checksum option.
>>
>> --
>> Shedi
>>
>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[PATCH] add option to skip files based on age/mtime

[J Raynor via rsync]

I've added the options --min-age=SECONDS and --max-age=SECONDS to allow 
rsync to skip files based on how recently they were modified.


Setting --min-age=30 (for example) would cause rsync to skip files that 
had been modified within the last 30 seconds.


Setting --max-age=7776000 would cause rsync to skip files that had been 
modified more than 90 days ago (7776000 == 60*60*24*90).


I realize some of this could be achieved with a find command and 
--exclude-from, but the same could be said for --min-size/--max-size. 
But it's more convenient to have these options, and they would allow for 
things --exclude-from wouldn't.  For example, a file might've been 
modified after generating the exclude file, so rsync would still 
transfer it even though that wasn't what was wanted.






diff --git a/generator.c b/generator.c
index 110db28f..22f0973f 100644
--- a/generator.c
+++ b/generator.c
@@ -70,6 +70,8 @@ extern int ignore_times;
 extern int size_only;
 extern OFF_T max_size;
 extern OFF_T min_size;
+extern int max_age;
+extern int min_age;
 extern int io_error;
 extern int flist_eof;
 extern int allowed_lull;
@@ -1706,6 +1708,23 @@ static void recv_generator(char *fname, struct 
file_struct *file, int ndx,

goto cleanup;
}

+   if (max_age > 0 && time(NULL)-file->modtime > max_age) {
+   if (INFO_GTE(SKIP, 1)) {
+   if (solo_file)
+   fname = f_name(file, NULL);
+   rprintf(FINFO, "%s is over max-age\n", fname);
+   }
+   goto cleanup;
+   }
+   if (min_age > 0 && time(NULL)-file->modtime < min_age) {
+   if (INFO_GTE(SKIP, 1)) {
+   if (solo_file)
+   fname = f_name(file, NULL);
+   rprintf(FINFO, "%s is under min-age\n", fname);
+   }
+   goto cleanup;
+   }
+
 	if (update_only > 0 && statret == 0 && file->modtime - sx.st.st_mtime 
< modify_window) {

if (INFO_GTE(SKIP, 1))
rprintf(FINFO, "%s is newer\n", fname);
@@ -2156,9 +2175,13 @@ void check_for_finished_files(int itemizing, enum 
logcode code, int check_redo)

if (check_redo && (ndx = get_redo_num()) != -1) {
OFF_T save_max_size = max_size;
OFF_T save_min_size = min_size;
+   int save_max_age = max_age;
+   int save_min_age = min_age;
csum_length = SUM_LENGTH;
max_size = -1;
min_size = -1;
+   max_age = 0;
+   min_age = 0;
ignore_existing = -ignore_existing;
ignore_non_existing = -ignore_non_existing;
update_only = -update_only;
@@ -2184,6 +2207,8 @@ void check_for_finished_files(int itemizing, enum 
logcode code, int check_redo)

csum_length = SHORT_SUM_LENGTH;
max_size = save_max_size;
min_size = save_min_size;
+   max_age = save_max_age;
+   min_age = save_min_age;
ignore_existing = -ignore_existing;
ignore_non_existing = -ignore_non_existing;
update_only = -update_only;
diff --git a/options.c b/options.c
index fd674754..7c7f31db 100644
--- a/options.c
+++ b/options.c
@@ -129,6 +129,8 @@ int need_messages_from_generator = 0;
 int max_delete = INT_MIN;
 OFF_T max_size = -1;
 OFF_T min_size = -1;
+int max_age = 0;
+int min_age = 0;
 int ignore_errors = 0;
 int modify_window = 0;
 int blocking_io = -1;
@@ -698,6 +700,8 @@ static struct poptOption long_options[] = {
   {"ignore-existing",  0,  POPT_ARG_NONE,   _existing, 0, 0, 0 },
   {"max-size", 0,  POPT_ARG_STRING, _size_arg, 
OPT_MAX_SIZE, 0, 0 },
   {"min-size", 0,  POPT_ARG_STRING, _size_arg, 
OPT_MIN_SIZE, 0, 0 },

+  {"max-age", 0,  POPT_ARG_INT, _age, 0, 0, 0 },
+  {"min-age", 0,  POPT_ARG_INT, _age, 0, 0, 0 },
   {"max-alloc",0,  POPT_ARG_STRING, _alloc_arg, 0, 0, 0 },
   {"sparse",  'S', POPT_ARG_VAL,_files, 1, 0, 0 },
   {"no-sparse",0,  POPT_ARG_VAL,_files, 0, 0, 0 },
@@ -2815,6 +2819,16 @@ void server_options(char **args, int *argc_p)
args[ac++] = safe_arg("--min-size", min_size_arg);
if (max_size >= 0)
args[ac++] = safe_arg("--max-size", max_size_arg);
+   if (min_age > 0) {
+   if (asprintf(, "--min-age=%d", min_age) < 0)
+   goto oom;
+   args[ac++] = arg;
+   }
+   if (max_age > 0) {
+   if (asprintf(, "--max-age=%d", max_age) < 0)
+   goto oom;
+  

Re: rsync Digest, Vol 255, Issue 2

[brent kimberley via rsync]

Shedi

I suggest upgrading rsync to a modern hash. For example,  Blake3.

On Tue, Mar 12, 2024, 8:00 a.m.  wrote:

> Send rsync mailing list submissions to
> rsync@lists.samba.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.samba.org/mailman/listinfo/rsync
> or, via email, send a message with subject or body 'help' to
> rsync-requ...@lists.samba.org
>
> You can reach the person managing the list at
> rsync-ow...@lists.samba.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of rsync digest..."
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> ---
> Today's Topics:
>
>1. Re: rsync segfaults when openssl fips is enabled
>   (Shreenidhi Shedi)
>
>
>
> -- Forwarded message --
> From: Shreenidhi Shedi 
> To: rsync@lists.samba.org
> Cc: Tapas Kundu 
> Bcc:
> Date: Tue, 12 Mar 2024 15:05:42 +0530
> Subject: Re: rsync segfaults when openssl fips is enabled
> Hi All,
>
> Any inputs on this issue?
>
> --
> Shedi
>
>
> On Wed, Feb 21, 2024 at 5:12 PM Shreenidhi Shedi <
> shreenidhi.sh...@broadcom.com> wrote:
>
>> Hi All,
>>
>> Copying the content from the GH issue as is.
>> Need your inputs on the same.
>> FWIW, the coredump files generated in linux have xattr values which are >
>> 32 bytes.
>>
>> https://github.com/WayneD/rsync/issues/569
>>
>> Hi All,
>>
>> System details:
>>
>> root@ph3dev [ ~ ]# rsync --version
>> rsync  version 3.2.4  protocol version 31
>> Capabilities:
>> 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
>> socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
>> hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, no ACLs,
>> xattrs, optional protect-args, iconv, prealloc, stop-at, no 
>> crtimesOptimizations:
>> SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
>> Checksum list:
>> xxh128 xxh3 xxh64 (xxhash) md5 md4 none
>> Compress list:
>> zstd lz4 zlibx zlib none
>>
>> OS:
>> Photon OS 3.0 with openssl fips enabled
>>
>> I'm using rsync like below:
>>
>> rsync -arXHpvxog  
>>
>> While doing so, if any file in src or dest location has a xattr value which 
>> is more than 32 bytes long, rsync segfaults.
>>
>> root@ph3dev [ ~ ]# getfattr -d -m '' -- /opt/x
>> getfattr: Removing leading '/' from absolute path names
>> # file: opt/x
>> user.bla1="1234567890abcdef1234567890abcdef1" > 33 bytes long
>>
>> After examining the coredump file, this happens at;
>>
>> rsync/xattrs.c
>> Line 277 in 2f9b963
>>
>> sum_init(xattr_sum_nni, checksum_seed);
>>
>> Here md5 is used by default and md5 usage is prohibited in fips mode.
>>
>> Is there any way to workaround this problem? Also, why not use xxhash for 
>> this operation like it's used by default during --checksum option.
>>
>> --
>> Shedi
>>
> ___
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> rsync mailing list
> rsync@lists.samba.org
> https://lists.samba.org/mailman/listinfo/rsync
>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync segfaults when openssl fips is enabled

[Shreenidhi Shedi via rsync]

Hi All,

Any inputs on this issue?

--
Shedi


On Wed, Feb 21, 2024 at 5:12 PM Shreenidhi Shedi <
shreenidhi.sh...@broadcom.com> wrote:

> Hi All,
>
> Copying the content from the GH issue as is.
> Need your inputs on the same.
> FWIW, the coredump files generated in linux have xattr values which are >
> 32 bytes.
>
> https://github.com/WayneD/rsync/issues/569
>
> Hi All,
>
> System details:
>
> root@ph3dev [ ~ ]# rsync --version
> rsync  version 3.2.4  protocol version 31
> Capabilities:
> 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
> socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
> hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, no ACLs,
> xattrs, optional protect-args, iconv, prealloc, stop-at, no 
> crtimesOptimizations:
> SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
> Checksum list:
> xxh128 xxh3 xxh64 (xxhash) md5 md4 none
> Compress list:
> zstd lz4 zlibx zlib none
>
> OS:
> Photon OS 3.0 with openssl fips enabled
>
> I'm using rsync like below:
>
> rsync -arXHpvxog  
>
> While doing so, if any file in src or dest location has a xattr value which 
> is more than 32 bytes long, rsync segfaults.
>
> root@ph3dev [ ~ ]# getfattr -d -m '' -- /opt/x
> getfattr: Removing leading '/' from absolute path names
> # file: opt/x
> user.bla1="1234567890abcdef1234567890abcdef1" > 33 bytes long
>
> After examining the coredump file, this happens at;
>
> rsync/xattrs.c
> Line 277 in 2f9b963
>
> sum_init(xattr_sum_nni, checksum_seed);
>
> Here md5 is used by default and md5 usage is prohibited in fips mode.
>
> Is there any way to workaround this problem? Also, why not use xxhash for 
> this operation like it's used by default during --checksum option.
>
> --
> Shedi
>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 6741] 'deleting' messages show up in improper places

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=6741

--- Comment #5 from Marc Aurèle La France  ---
Created attachment 18263
  --> https://bugzilla.samba.org/attachment.cgi?id=18263=edit
rsync stdout filter

Just something I've come up with to work around this issue.  Not perfect but
does the job.  See usage information in the comments.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync segfaults when openssl fips is enabled

[Shreenidhi Shedi via rsync]

Hi All,

Copying the content from the GH issue as is.
Need your inputs on the same.
FWIW, the coredump files generated in linux have xattr values which are >
32 bytes.

https://github.com/WayneD/rsync/issues/569

Hi All,

System details:

root@ph3dev [ ~ ]# rsync --version
rsync  version 3.2.4  protocol version 31
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, no ACLs,
xattrs, optional protect-args, iconv, prealloc, stop-at, no
crtimesOptimizations:
SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
zstd lz4 zlibx zlib none

OS:
Photon OS 3.0 with openssl fips enabled

I'm using rsync like below:

rsync -arXHpvxog  

While doing so, if any file in src or dest location has a xattr value
which is more than 32 bytes long, rsync segfaults.

root@ph3dev [ ~ ]# getfattr -d -m '' -- /opt/x
getfattr: Removing leading '/' from absolute path names
# file: opt/x
user.bla1="1234567890abcdef1234567890abcdef1" > 33 bytes long

After examining the coredump file, this happens at;

rsync/xattrs.c
Line 277 in 2f9b963

sum_init(xattr_sum_nni, checksum_seed);

Here md5 is used by default and md5 usage is prohibited in fips mode.

Is there any way to workaround this problem? Also, why not use xxhash
for this operation like it's used by default during --checksum option.

-- 
Shedi
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 15585] New: rsync ends still with error 22 when try to deleting many files

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=15585

Bug ID: 15585
   Summary: rsync ends still with error 22 when try to deleting
many files
   Product: rsync
   Version: 3.2.0
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: wa...@opencoder.net
  Reporter: f...@fkn-systems.de
QA Contact: rsync...@samba.org
  Target Milestone: ---

Problem:

When syncing Souce/Target and must deleting many files, rsync stop still
quiet with Error-Code:

 "22 - Error allocating core memory buffers"

Desc:

When more than ~5-10 million files to delete, rsync ends still quiet, in
case of deleting process. Rsync reads (with --delete-before) the
complete Source and breaks than after deleting many (>1000) files on
Target. Else, (withOUT --delete-before), he does some sync on Target and
some deletes, some sync ... and breaks than after delete some files.

In case of this, Backups are not synced, and may space overflow after
next backup run.

Use of ‐‐max‐delete=100.000.000 are not functional to solve this

Command:

 rsync -vaxHSPAX --delete [--delete-before] $SOURCE $TARGET
 # $SOURCE, $TARGET are real directorys or remote (ramote:dir...)

Versions: It tested fail with:

 * rsync  version 3.2.3  protocol version 31
 * rsync  version 3.2.7  protocol version 31

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

tests for clone-dest

[Joseph Maher via rsync]

WayneD's clone-dest patch seems to be working for me, it's available here:

https://github.com/WayneD/rsync-patches/blob/master/clone-dest.diff

I've written some very basic tests, which I've attached to this email, in 
case they are of use to anyone else.  They're also available here:


https://github.com/josephmaher/rsync

Notes:

Tests need to be run as root.

	clone-dest-btrs.test needs mkfs.btrfs (from btrfs-progs) and 
btrs-search-metadata (from python3-btrfs).


clone-dest-xfs.test needs mkfs.xfs and xfs_bmap (from xfsprogs).


Joseph#!/bin/sh

# Copyright (C) 2024 by Joseph Maher 
# This program is distributable under the terms of the GNU GPL (see COPYING)
#
# Check that the --clone-dest option makes reflinks as requested

. "$suitedir/rsync.fns"

test -f /sbin/mkfs.btrfs || test_skipped "Can't find mkfs.btrfs (only available 
on Linus with btrfs support)"
test -f /usr/bin/btrfs-search-metadata || test_skipped "Can't find 
btrfs-search-metatadata from python3-btrfs (only available on Linux)"

# make a btrfs filesystem and mount it
truncate -s 115M $scratchdir/btrfs.image
/sbin/mkfs.btrfs $scratchdir/btrfs.image > /dev/null
mkdir -p $scratchdir/mnt/
mount -o loop $scratchdir/btrfs.image $scratchdir/mnt/ || test_skipped "Can't 
mount btrfs image file, try running as root"

# set up some test files and rsync them
mkdir $scratchdir/mnt/1 $scratchdir/mnt/2 $scratchdir/mnt/3
# files should be at least 4K in size so they fill an extent block
dd if=/dev/urandom of=$scratchdir/mnt/1/a bs=4K count=1 status=none
# sometimes the extents get cached, sycn helps write them to disk
sync $scratchdir/mnt/1/a
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/1/b
sync $scratchdir/mnt/1/b
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/3/a
sync $scratchdir/mnt/3/a

clonedir=$(realpath $scratchdir/mnt/3)

checkit "$RSYNC -a --clone-dest='$clonedir' '$scratchdir/mnt/1/' 
'$scratchdir/mnt/2/'" "$scratchdir/mnt/1/" "$scratchdir/mnt/2/"
sync $scratchdir/mnt/2/a
sync $scratchdir/mnt/2/b

# check the extents are the same

get_extents() {
result=$(btrfs-search-metadata file $1 | grep disk_bytenr | sed 
's/.*disk_bytenr\ //' | sed 's/\ disk_num_bytes.*//')
if [ ! -n "$result" ]; then
echo "couldn't find extents for " $1
fi
echo "$result"
}

test "$(get_extents $scratchdir/mnt/2/a)" = "$(get_extents 
$scratchdir/mnt/3/a)" || test_fail "clone-dest files have different extents"

# clean up
umount $scratchdir/mnt/

# The script would have aborted on error, so getting here means we've won.
exit 0
#!/bin/sh

# Copyright (C) 2024 by Joseph Maher 
# This program is distributable under the terms of the GNU GPL (see COPYING)
#
# Check that the --clone-dest option makes reflinks as requested

. "$suitedir/rsync.fns"

test -f /sbin/mkfs.xfs || test_skipped "Can't find mkfs.xfs (only available on 
Linus with xfs support)"

# make a btrfs filesystem and mount it
truncate -s 300M $scratchdir/xfs.image
/sbin/mkfs.xfs $scratchdir/xfs.image > /dev/null
mkdir -p $scratchdir/mnt/
mount -o loop $scratchdir/xfs.image $scratchdir/mnt/ || test_skipped "Can't 
mount xfs image file, try running as root"

# set up some test files and rsync them
mkdir $scratchdir/mnt/1 $scratchdir/mnt/2 $scratchdir/mnt/3
# files should be at least 4K in size so they fill an extent block
dd if=/dev/urandom of=$scratchdir/mnt/1/a bs=4K count=1 status=none
# sometimes the extents get cached, sync helps write them to disk
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/1/b
cp --reflink=never $scratchdir/mnt/1/a $scratchdir/mnt/3/a

clonedir=$(realpath $scratchdir/mnt/3)

checkit "$RSYNC -a --clone-dest='$clonedir' '$scratchdir/mnt/1/' 
'$scratchdir/mnt/2/'" "$scratchdir/mnt/1/" "$scratchdir/mnt/2/"

# check the extents are the same

get_extents() {
echo "$(/sbin/xfs_bmap $1 | tail -n 1)"
}

test "$(get_extents $scratchdir/mnt/2/a)" = "$(get_extents 
$scratchdir/mnt/3/a)" || test_fail "clone-dest files have different extents"

# clean up
umount $scratchdir/mnt/

# The script would have aborted on error, so getting here means we've won.
exit 0
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync replacing symlinks without warning (resend)

[Kevin Korb via rsync]

I'm not really blaming the user.  If it were up to me, -v would include -i.

On 2/9/24 05:36, Andreas Gruenbacher wrote:

On Sun, Feb 4, 2024 at 7:20 PM Kevin Korb via rsync
 wrote:

rsync's -v is fairly useless.  Learn to use -i instead or in addition to.


Well, note that I didn't say anything about the lib/ directory in that
command; it's just that rsync decided to remove the symlink component
from the path lib/modules/.

Wouldn't it be more useful to make it obvious in the --verbose and
--dry-run output what rsync has actually decided to do instead of
blaming the user, which is what you're doing? Especially if the
behavior is all but self-explanatory?

Thanks,
Andreas


On 2/4/24 12:58, Andreas Gruenbacher via rsync wrote:

Hello,

when trying to rsync files between hosts, I ran into a surprising case
in which rsync replaces a symlink with a directory, with no indication
of any kind.

In the following reproducer, rsync is called as follows:

rsync --verbose --recursive --relative --delete a/./lib/modules b/

Directory b contains a 'lib' symlink pointing to 'usr/lib', and rsync
removes that and replaces it with a directory.

In my real-world use case, this caused '/lib' -> '/usr/lib' symlinks
to be replaced with '/lib' directories, which left the receiving test
machines in a fairly sad state.

I have since figured out that I can get rsync to behave as expected by
adding the --keep-dirlinks option, but ...

it's very unfortunate that when rsync does that kind of thing, it
leaves no indication in the 'rsync --dry-run' and 'rsync -v' output.
Could that please be fixed?

Thanks,
Andreas


#! /bin/sh

tmp=$(mktemp -dt ${0##*/}.XX)
trap 'cd /; rm -rf $tmp' EXIT
cd "$tmp"

umask 022

mkdir -p a/lib/modules
echo foo > a/lib/modules/foo

mkdir -p b/usr/lib/modules
ln -s usr/lib b/lib

show() {
  find "$@" | xargs stat -c "%F %N" | sort -k2
}

echo "from:"
show a

echo
echo "to:"
show b

echo
echo "rsync:"
rsync \
  --verbose \
  --recursive \
  --relative \
  --delete \
  a/./lib/modules \
  b/

echo
echo "to:"
show b

# SCRIPT OUTPUT with rsync 3.2.7:
# ==
=
# from:
# directory 'a'
# directory 'a/lib'
# directory 'a/lib/modules'
# regular file 'a/lib/modules/foo'
#
# to:
# directory 'b'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# symbolic link 'b/lib' -> 'usr/lib'
#
# rsync:
# sending incremental file list
# lib/
# lib/modules/
# lib/modules/foo
#
# sent 160 bytes  received 47 bytes  414.00 bytes/sec
# total size is 4  speedup is 0.02
#
# to:
# directory 'b'
# directory 'b/lib'
# directory 'b/lib/modules'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# regular file 'b/lib/modules/foo'




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync replacing symlinks without warning (resend)

[Andreas Gruenbacher via rsync]

On Sun, Feb 4, 2024 at 7:20 PM Kevin Korb via rsync
 wrote:
> rsync's -v is fairly useless.  Learn to use -i instead or in addition to.

Well, note that I didn't say anything about the lib/ directory in that
command; it's just that rsync decided to remove the symlink component
from the path lib/modules/.

Wouldn't it be more useful to make it obvious in the --verbose and
--dry-run output what rsync has actually decided to do instead of
blaming the user, which is what you're doing? Especially if the
behavior is all but self-explanatory?

Thanks,
Andreas

> On 2/4/24 12:58, Andreas Gruenbacher via rsync wrote:
> > Hello,
> >
> > when trying to rsync files between hosts, I ran into a surprising case
> > in which rsync replaces a symlink with a directory, with no indication
> > of any kind.
> >
> > In the following reproducer, rsync is called as follows:
> >
> >rsync --verbose --recursive --relative --delete a/./lib/modules b/
> >
> > Directory b contains a 'lib' symlink pointing to 'usr/lib', and rsync
> > removes that and replaces it with a directory.
> >
> > In my real-world use case, this caused '/lib' -> '/usr/lib' symlinks
> > to be replaced with '/lib' directories, which left the receiving test
> > machines in a fairly sad state.
> >
> > I have since figured out that I can get rsync to behave as expected by
> > adding the --keep-dirlinks option, but ...
> >
> > it's very unfortunate that when rsync does that kind of thing, it
> > leaves no indication in the 'rsync --dry-run' and 'rsync -v' output.
> > Could that please be fixed?
> >
> > Thanks,
> > Andreas
> >
> >
> > #! /bin/sh
> >
> > tmp=$(mktemp -dt ${0##*/}.XX)
> > trap 'cd /; rm -rf $tmp' EXIT
> > cd "$tmp"
> >
> > umask 022
> >
> > mkdir -p a/lib/modules
> > echo foo > a/lib/modules/foo
> >
> > mkdir -p b/usr/lib/modules
> > ln -s usr/lib b/lib
> >
> > show() {
> >  find "$@" | xargs stat -c "%F %N" | sort -k2
> > }
> >
> > echo "from:"
> > show a
> >
> > echo
> > echo "to:"
> > show b
> >
> > echo
> > echo "rsync:"
> > rsync \
> >  --verbose \
> >  --recursive \
> >  --relative \
> >  --delete \
> >  a/./lib/modules \
> >  b/
> >
> > echo
> > echo "to:"
> > show b
> >
> > # SCRIPT OUTPUT with rsync 3.2.7:
> > # ==
> > =
> > # from:
> > # directory 'a'
> > # directory 'a/lib'
> > # directory 'a/lib/modules'
> > # regular file 'a/lib/modules/foo'
> > #
> > # to:
> > # directory 'b'
> > # directory 'b/usr'
> > # directory 'b/usr/lib'
> > # directory 'b/usr/lib/modules'
> > # symbolic link 'b/lib' -> 'usr/lib'
> > #
> > # rsync:
> > # sending incremental file list
> > # lib/
> > # lib/modules/
> > # lib/modules/foo
> > #
> > # sent 160 bytes  received 47 bytes  414.00 bytes/sec
> > # total size is 4  speedup is 0.02
> > #
> > # to:
> > # directory 'b'
> > # directory 'b/lib'
> > # directory 'b/lib/modules'
> > # directory 'b/usr'
> > # directory 'b/usr/lib'
> > # directory 'b/usr/lib/modules'
> > # regular file 'b/lib/modules/foo'
> >
> >
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 2294] Detect renamed files and handle by renaming instead of delete/re-send

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=2294

--- Comment #39 from andy  ---
> This feature request is so old it has lost relavence because btrfs/zfs/etc 
> are more optimal backup solutions than rsync.

Funny I am doing exactly this, but I came to rsync looking for a backup for
when ZFS fails. Many consider zfs/btrfs/snapshots as "not a backup". There are
many things that can go wrong that you will need real backups to save you:
- accidentally deleted pool/datasets/snapshots
- bug in replication tool
- user error
- ZFS bug

It should be considered a certainty that one of those will happen at some
point, and the ZFS snapshots won't save you.

> With zfs one can even have encrypted backups without the backup server ever 
> seeing the key or un-encrypted data.

I love this idea, but in practice I'm finding there is significant risk with
the state of ZFS encryption. There are so many active bugs related to
encryption. I'm in the middle of implementing a replication system based on raw
encrypted snapshot replication between multiple systems, trusted and untrusted.
But the new bugs I've run into along the way, along with the previously known
ones, makes me really feel the need for a solid non-ZFS filesystem backup. And
also a low complexity tool, not dependent on complicated replication
tools/scripts.

In looking for rename/move solutions with rsync, one issue I can foresee with
inode tracking is that I find it is very common to cross filesystem boundaries.
Anything tracking inodes would need to track the device as well, though the
device number from the stat struct doesn't seem to be enough in the case of ZFS
to trace back to what filesystem it actually comes from. 

Reading the unison documentation, it seems that for linux they track the combo
of inode & last modification time to detect moves/renames. I wonder if some
kind of collision is possible under a rare multi-filesystem edge case. Inodes
aren't unique across multiple filesystems.

Restic is another option that handles moves/renames/dedup automatically, just
at the cost of CPU time for encrypting/hashing. Probably worth considering borg
and friends at that point.

Well, maybe the cost of rsync's inefficiency here is worth it's simplicity. But
it would be a great feature to have.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BUG? rsync ends without message by delete files

[Kevin Korb via rsync]

now it sounds like you have too many hard links for rsync to handle.

On 2/7/24 08:05, Franke via rsync wrote:

Am 06.02.24 um 23:20 schrieb Roland:

and then, it stops totally quiet.

you mean it simply exits without any message?



Yes rsync ends totally quit.



what's the return code ( echo $? )


After some more tests, the ERR-Code are:

  "22 - Error allocating core memory buffers"
(with --delete-before)


But, now there are only ~100.000 Files to delete, in some Tranches.


IMHO:

rsync read the Fileslist without problems, begins with deleting (and on
thi can decrease the list too) and fail than with a Momory-Error? :-o

Source- and Target-system have 64GB RAM, could be enough...

This seem for me as a Bug.


And sorry, but i cant make Issues on Github.



Gruss
Franke



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BUG? rsync ends without message by delete files

[Roland via rsync]

here is another report of this behaviour.
https://unix.stackexchange.com/questions/754923/rsync-just-stops

nothing appropriate in bugzilla, besides
https://bugzilla.samba.org/show_bug.cgi?id=13317

do you use zfs or is there full-space/quota condition while running?

if you can't resolve, please file a bugreport in bugzilla

roland


Am 06.02.24 um 22:18 schrieb Franke via rsync:

Hi Kevin,

Am 06.02.24 um 20:55 schrieb Kevin Korb:

The other likely cause is your $SOURCE being something that contains a *
or other wildcard.  If there is a wildcard in the source parameter then
the shell expands that wildcard giving rsync a list of sources.  The
--delete only operates within that list meaning that anything not
currently in that list is immune to deletion.

no, $SOURCE $TARGET are only placeholder here. In real there are normal
path's like: /foo/bar /new/foo/bar

The Rsync does some time correctly, like:

(sync) /foo/bar/bla_1
delete /foo/bar/mee/blub
...
(sync) /foo/bar/bla_2
...
delete /foo/bar/mee/borg
...

and then, it stops totally quiet.

I try'd to recall the command again an then ist wokrs again correctly
after the last output delete (/foo/bar/mee/borg...) and stopps again
after some correctly doing after deleting some files.

Same behavior by calling with --delete-before rsync breaks here too
after delete some files.

The very BIG Problem of this: NO ERRORS will be output
In the following the Backups are not complete synced, and the HDD-Store
goes to zero. :-(



Gruss
Franke



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BUG? rsync ends without message by delete files

[Roland via rsync]

and then, it stops totally quiet.


you mean it simply exits without any message?

what's the return code ( echo $? )

roland

Am 06.02.24 um 22:18 schrieb Franke via rsync:

Hi Kevin,

Am 06.02.24 um 20:55 schrieb Kevin Korb:

The other likely cause is your $SOURCE being something that contains a *
or other wildcard.  If there is a wildcard in the source parameter then
the shell expands that wildcard giving rsync a list of sources.  The
--delete only operates within that list meaning that anything not
currently in that list is immune to deletion.

no, $SOURCE $TARGET are only placeholder here. In real there are normal
path's like: /foo/bar /new/foo/bar

The Rsync does some time correctly, like:

(sync) /foo/bar/bla_1
delete /foo/bar/mee/blub
...
(sync) /foo/bar/bla_2
...
delete /foo/bar/mee/borg
...

and then, it stops totally quiet.

I try'd to recall the command again an then ist wokrs again correctly
after the last output delete (/foo/bar/mee/borg...) and stopps again
after some correctly doing after deleting some files.

Same behavior by calling with --delete-before rsync breaks here too
after delete some files.

The very BIG Problem of this: NO ERRORS will be output
In the following the Backups are not complete synced, and the HDD-Store
goes to zero. :-(



Gruss
Franke



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BUG? rsync ends without message by delete files

[Kevin Korb via rsync]

Normally, when rsync isn't deleting things the problem is that there is
some kind of error (possibly scrolled off screen unnoticed) but it
sounds like you are getting no output at all which would eliminate that
possibility.

The other likely cause is your $SOURCE being something that contains a *
or other wildcard.  If there is a wildcard in the source parameter then
the shell expands that wildcard giving rsync a list of sources.  The
--delete only operates within that list meaning that anything not
currently in that list is immune to deletion.

If that doesn't answer your question I would suggest adding -ii (doubled
--itemize-changes) as it will show you the files it is analyzing and
what it plans to do about them.

On Tue, 6 Feb 2024, Franke via rsync wrote:


Date: Tue, 6 Feb 2024 19:53:32 +0100
From: Franke via rsync 
To: just subscribed for rsync-qa from bugzilla via rsync

Subject: BUG? rsync ends without message by delete files

Hi!

Is this a bug, or did I miss a setting?

When synchronizing and deleting files, rsync (version 3.2.7  protocol
version 31 AND before) simply stops whitout comments.
No error, no message, nothing in the syslog.

Rsync works normally, but after delete some on, it ends.

I try'd --max-delete=1 to, but no effekt.

Command are:

rsync -vaxHSPAX --delete $SOURCE $TARGET

The dirs containing in sync containing approximately 40 million files to
delete.

And I can't delete it by hand as we need to drive some recoveries after
a server crash and relocation.


Any Ideas?



Gruss
Franke




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync replacing symlinks without warning (resend)

[Kevin Korb via rsync]

rsync's -v is fairly useless.  Learn to use -i instead or in addition to.

On 2/4/24 12:58, Andreas Gruenbacher via rsync wrote:

Hello,

when trying to rsync files between hosts, I ran into a surprising case
in which rsync replaces a symlink with a directory, with no indication
of any kind.

In the following reproducer, rsync is called as follows:

   rsync --verbose --recursive --relative --delete a/./lib/modules b/

Directory b contains a 'lib' symlink pointing to 'usr/lib', and rsync
removes that and replaces it with a directory.

In my real-world use case, this caused '/lib' -> '/usr/lib' symlinks
to be replaced with '/lib' directories, which left the receiving test
machines in a fairly sad state.

I have since figured out that I can get rsync to behave as expected by
adding the --keep-dirlinks option, but ...

it's very unfortunate that when rsync does that kind of thing, it
leaves no indication in the 'rsync --dry-run' and 'rsync -v' output.
Could that please be fixed?

Thanks,
Andreas


#! /bin/sh

tmp=$(mktemp -dt ${0##*/}.XX)
trap 'cd /; rm -rf $tmp' EXIT
cd "$tmp"

umask 022

mkdir -p a/lib/modules
echo foo > a/lib/modules/foo

mkdir -p b/usr/lib/modules
ln -s usr/lib b/lib

show() {
 find "$@" | xargs stat -c "%F %N" | sort -k2
}

echo "from:"
show a

echo
echo "to:"
show b

echo
echo "rsync:"
rsync \
 --verbose \
 --recursive \
 --relative \
 --delete \
 a/./lib/modules \
 b/

echo
echo "to:"
show b

# SCRIPT OUTPUT with rsync 3.2.7:
# ==
=
# from:
# directory 'a'
# directory 'a/lib'
# directory 'a/lib/modules'
# regular file 'a/lib/modules/foo'
#
# to:
# directory 'b'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# symbolic link 'b/lib' -> 'usr/lib'
#
# rsync:
# sending incremental file list
# lib/
# lib/modules/
# lib/modules/foo
#
# sent 160 bytes  received 47 bytes  414.00 bytes/sec
# total size is 4  speedup is 0.02
#
# to:
# directory 'b'
# directory 'b/lib'
# directory 'b/lib/modules'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# regular file 'b/lib/modules/foo'




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync replacing symlinks without warning (resend)

[Andreas Gruenbacher via rsync]

Hello,

when trying to rsync files between hosts, I ran into a surprising case
in which rsync replaces a symlink with a directory, with no indication
of any kind.

In the following reproducer, rsync is called as follows:

  rsync --verbose --recursive --relative --delete a/./lib/modules b/

Directory b contains a 'lib' symlink pointing to 'usr/lib', and rsync
removes that and replaces it with a directory.

In my real-world use case, this caused '/lib' -> '/usr/lib' symlinks
to be replaced with '/lib' directories, which left the receiving test
machines in a fairly sad state.

I have since figured out that I can get rsync to behave as expected by
adding the --keep-dirlinks option, but ...

it's very unfortunate that when rsync does that kind of thing, it
leaves no indication in the 'rsync --dry-run' and 'rsync -v' output.
Could that please be fixed?

Thanks,
Andreas


#! /bin/sh

tmp=$(mktemp -dt ${0##*/}.XX)
trap 'cd /; rm -rf $tmp' EXIT
cd "$tmp"

umask 022

mkdir -p a/lib/modules
echo foo > a/lib/modules/foo

mkdir -p b/usr/lib/modules
ln -s usr/lib b/lib

show() {
find "$@" | xargs stat -c "%F %N" | sort -k2
}

echo "from:"
show a

echo
echo "to:"
show b

echo
echo "rsync:"
rsync \
--verbose \
--recursive \
--relative \
--delete \
a/./lib/modules \
b/

echo
echo "to:"
show b

# SCRIPT OUTPUT with rsync 3.2.7:
# ==
=
# from:
# directory 'a'
# directory 'a/lib'
# directory 'a/lib/modules'
# regular file 'a/lib/modules/foo'
#
# to:
# directory 'b'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# symbolic link 'b/lib' -> 'usr/lib'
#
# rsync:
# sending incremental file list
# lib/
# lib/modules/
# lib/modules/foo
#
# sent 160 bytes  received 47 bytes  414.00 bytes/sec
# total size is 4  speedup is 0.02
#
# to:
# directory 'b'
# directory 'b/lib'
# directory 'b/lib/modules'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# regular file 'b/lib/modules/foo'


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Release the fix for the argv use-after-free issue with popt 1.19?

[Matt McCutchen via rsync]

I'm using Fedora 38's rsync-3.2.7-2.fc38.x86_64 package, and the other
day, I noticed that one of my backup scripts was creating directories
with garbage names.  Eventually I tracked the problem down to the argv
use-after-free issue with popt 1.19 that was fixed in commit
8990ad96de881f7332d16d32485f9d8b841a87d2.  That fix has not been
released.  Is an rsync release coming any time soon?  If not, I'll ask
Fedora to consider adding the individual patch to their package.  (And
in the meantime, I've switched to using my own rsync 3.2.7 package with
the patch.)

Thanks,
Matt

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Archiving to vfat

[Ian Z via rsync]

--modify-window was it, thanks very much. Works flawlessly now.

Thanks also for the s,a,rt, tip.

-- 
Ian

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Archiving to vfat

[Kevin Korb via rsync]

Also, instead of -a use -rt.  Those are the only parts of -a that FAT 
even pretends to support.


On 1/21/24 16:42, Roland via rsync wrote:

it's most likely because of vfat timestamp limitation

try

--modify-window
   When comparing two timestamps, rsync treats the
timestamps as being equal if they differ by no more than  the modify-window
   value.   This  is  normally  0 (for an exact match), but
you may find it useful to set this to a larger value in some situa-
   tions.  In particular, when transferring to or from an MS
Windows FAT filesystem (which represents  times  with  a 2-second
   resolution), --modify-window=1 is useful (allowing times
to differ by up to 1 second).

roland

Am 21.01.24 um 22:15 schrieb Ian Z via rsync:

I am trying to use rsync between two local directories on Linux.

The source directory is on a normal ext4 partition, under my home
directory. The destination is an SD card that I insert into the card
reader on the computer, formatted with a vfat filesystem.

The command line is like

   rsync -avC --delete /home/itz/foo/ /media/itz/DEAD-BEEF/foo/

This does not work as I expected: all files are always transferred,
even when I run this command 2 times with nothing in between. I
expected only new or changed files to be transferred. Is this

- a user error
- a rsync bug
- a limitation of the vfat filesystem?

In any case, what can I do, using rsync or possibly something else, to
do what I want? I note that the absolute timestamps on the files are
unimportant; I can change them to whatever. All that matters is that
new and changed files are copied every time I run this command, and
only those files.





--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Archiving to vfat

[Roland via rsync]

it's most likely because of vfat timestamp limitation

try

--modify-window
  When comparing two timestamps, rsync treats the
timestamps as being equal if they differ by no more than  the modify-window
  value.   This  is  normally  0 (for an exact match), but
you may find it useful to set this to a larger value in some situa-
  tions.  In particular, when transferring to or from an MS
Windows FAT filesystem (which represents  times  with  a 2-second
  resolution), --modify-window=1 is useful (allowing times
to differ by up to 1 second).

roland

Am 21.01.24 um 22:15 schrieb Ian Z via rsync:

I am trying to use rsync between two local directories on Linux.

The source directory is on a normal ext4 partition, under my home
directory. The destination is an SD card that I insert into the card
reader on the computer, formatted with a vfat filesystem.

The command line is like

   rsync -avC --delete /home/itz/foo/ /media/itz/DEAD-BEEF/foo/

This does not work as I expected: all files are always transferred,
even when I run this command 2 times with nothing in between. I
expected only new or changed files to be transferred. Is this

- a user error
- a rsync bug
- a limitation of the vfat filesystem?

In any case, what can I do, using rsync or possibly something else, to
do what I want? I note that the absolute timestamps on the files are
unimportant; I can change them to whatever. All that matters is that
new and changed files are copied every time I run this command, and
only those files.



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Archiving to vfat

[Ian Z via rsync]

I am trying to use rsync between two local directories on Linux.

The source directory is on a normal ext4 partition, under my home
directory. The destination is an SD card that I insert into the card
reader on the computer, formatted with a vfat filesystem.

The command line is like

  rsync -avC --delete /home/itz/foo/ /media/itz/DEAD-BEEF/foo/

This does not work as I expected: all files are always transferred,
even when I run this command 2 times with nothing in between. I
expected only new or changed files to be transferred. Is this

- a user error
- a rsync bug
- a limitation of the vfat filesystem?

In any case, what can I do, using rsync or possibly something else, to
do what I want? I note that the absolute timestamps on the files are
unimportant; I can change them to whatever. All that matters is that
new and changed files are copied every time I run this command, and
only those files.

-- 
Ian

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Everything working as expected, so shouldn't ERROR be WARNING

[Roger Price via rsync]

On Thu, 18 Jan 2024, Paul Slootman via rsync wrote:

On Thu 18 Jan 2024, Roger Price via rsync wrote:

I get the messages

 sending incremental file list
 ERROR: daemon refused to receive file "rprice/demo.dvi"

I understand that the remote daemon has refused file demo.dvi because I
specifically requested that dvi files not be transferred.  I choose that
myself in a regular configuration file.  So shouldn't it be a WARNING rather
than an ERROR?  I would expect to see


In this case you're in control of both ends of the transfer, so you know
that *.dvi files won't be transferred.

However, it could be that this rsync command is being run by someone who
expects rsync to do what they asked it to do, i.e. transfer the entire
contents of .../rprice to the remote server, and the client rsync can't
fulfil that request; hence the error.

If you don't want *.dvi files to be transferred, then you should add
   --exclude '*.dvi'
to the invocation.


I accept that if the message has to cover all the cases, including hardware 
misconfigurations and errors, then ERROR is necessary.


But it would have been nice to have a specific message for exclude = ... 
effect in remote rsyncd.conf even if the user has no direct access.


Roger

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Everything working as expected, so shouldn't ERROR be WARNING

[Paul Slootman via rsync]

On Thu 18 Jan 2024, Roger Price via rsync wrote:

> I am backing up a user's directories from local machine titan to remote
> machine maria. On the remote machine maria file /etc/rsyncd.conf contains
> 
>  [rprice-home]
>  ...
>  exclude = *.dvi
> 
> I start the backup by using this command on the local machine titan:
> 
>  rprice@titan ~ rsync -av --dry-run /mnt/home/rprice 
> rsync://rprice@maria/rprice-home
> 
> I get the messages
> 
>  sending incremental file list
>  ERROR: daemon refused to receive file "rprice/demo.dvi"
>  ...
> 
> I understand that the remote daemon has refused file demo.dvi because I
> specifically requested that dvi files not be transferred.  I choose that
> myself in a regular configuration file.  So shouldn't it be a WARNING rather
> than an ERROR?  I would expect to see

In this case you're in control of both ends of the transfer, so you know
that *.dvi files won't be transferred.

However, it could be that this rsync command is being run by someone who
expects rsync to do what they asked it to do, i.e. transfer the entire
contents of .../rprice to the remote server, and the client rsync can't
fulfil that request; hence the error.

If you don't want *.dvi files to be transferred, then you should add
--exclude '*.dvi'
to the invocation.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Everything working as expected, so shouldn't ERROR be WARNING

[Roger Price via rsync]

I am backing up a user's directories from local machine titan to remote machine 
maria. On the remote machine maria file /etc/rsyncd.conf contains


 [rprice-home]
 ...
 exclude = *.dvi

I start the backup by using this command on the local machine titan:

 rprice@titan ~ rsync -av --dry-run /mnt/home/rprice 
rsync://rprice@maria/rprice-home

I get the messages

 sending incremental file list
 ERROR: daemon refused to receive file "rprice/demo.dvi"
 ...

I understand that the remote daemon has refused file demo.dvi because I 
specifically requested that dvi files not be transferred.  I choose that myself 
in a regular configuration file.  So shouldn't it be a WARNING rather than an 
ERROR?  I would expect to see


 WARNING: daemon refused to receive file "rprice/demo.dvi"

Roger

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[PATCH] Fix INET6 detection on recent clang

[Chris Webb via rsync]

The implicit int return for main() in the configure test for INET6 is
now a hard error on recent clang, breaking the detection of IPv6 support.
Update this to int main(void) like the other configure tests.

The problem this causes in practice is quite subtle and easy to miss.
ssh is always run with -4 which works fine except when there is only v6
connectivity between a pair of hosts, whereupon ssh between them works
but rsync unexpectedly fails with 'Name has no usable address'.

Signed-off-by: Chris Webb 
---
 configure.ac | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index ccad7f13..9a766c94 100644
--- a/configure.ac
+++ b/configure.ac
@@ -392,7 +392,7 @@ AS_HELP_STRING([--disable-ipv6],[disable to omit ipv6 
support]),
 #include 
 #include 
 #include 
-main()
+int main(void)
 {
if (socket(AF_INET6, SOCK_STREAM, 0) < 0)
  exit(1);


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: process --files-from filelist as given

[Francis.Montagnac--- via rsync]

Hi.

On Sun, 31 Dec 2023 20:28:21 +0100 Roland via rsync wrote:

> apparently, rsync sorts the list of files  provided to "--files-from".

> how can i avoid sorting of that list ?

According to the man, this is not possible. See: SORTED TRANSFER ORDER
that suggest also the --delay‐updates option.

> I want to copy a list of files in specific order

Why ?

-- 
francis

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 15546] New: disable of sorting when files to transfer is fed via --files-from

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=15546

Bug ID: 15546
   Summary: disable of sorting when files to transfer is fed via
--files-from
   Product: rsync
   Version: 3.2.0
  Hardware: All
OS: All
Status: NEW
  Severity: enhancement
  Priority: P5
 Component: core
  Assignee: wa...@opencoder.net
  Reporter: devz...@web.de
QA Contact: rsync...@samba.org
  Target Milestone: ---

i need to transfer files in correct order so they are serialized on disk in a
special way to optimize access time.

apparently, files fed via --files-from automatically getting ordered.

it would be nice if this could be discabled. 

i could help myself with tar trough a pipe, but i would use my favourite and
most trusted file transfer tool to handle such a job, too.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: process --files-from filelist as given

[Roland via rsync]

I want to copy a list of files in specific order

Why ?


because i want to serialize files on disk so they are stored on disk in
the order being accessed regularly

i built that list for --files-from   via output from fatrace tool.

i now did use tar to transfer the files

i will add an RFE to bugzilla

thanks

roland

Am 31.12.23 um 20:56 schrieb francis.montag...@inria.fr:

Hi.

On Sun, 31 Dec 2023 20:28:21 +0100 Roland via rsync wrote:


apparently, rsync sorts the list of files  provided to "--files-from".
how can i avoid sorting of that list ?

According to the man, this is not possible. See: SORTED TRANSFER ORDER
that suggest also the --delay‐updates option.


I want to copy a list of files in specific order

Why ?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

process --files-from filelist as given

[Roland via rsync]

hello,

apparently,  rsync sorts the list of files  provided to "--files-from".

how can i avoid sorting of that list ?

I want to copy a list of files in specific order

regards

Roland


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync over ssh fails with --files-from

[Alex via rsync]

> The errors column is 0.  The drop column is 18.  The second bit number
> is the number of packets which should grow.  At least that is how I read
> it.  Column makes it more readable in a terminal but not so much in an
> email.
>

Yes, my apologies. I even debated inserting a screenshot. errs was
definitely the one I referenced, and it was increasing.

As it turns out, there was some kind of network configuration issue with my
network provider that caused this fscking problem.

I knew at some point it couldn't have been an rsync problem, but I also
wasn't sure if it was a local network problem (routing, problems with the
interface, number of open files, buffers, etc).

Now I can get on with more productive projects, ugh.

Thanks for everyone's help.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync over ssh fails with --files-from

[Kevin Korb via rsync]

The errors column is 0.  The drop column is 18.  The second bit number 
is the number of packets which should grow.  At least that is how I read 
it.  Column makes it more readable in a terminal but not so much in an 
email.


On 12/21/23 14:18, Alex wrote:
Can someone help me determine if these errors are normal or if this 
could somehow be the cause? I've removed the last three columns for 
readability - they were all zeros.


# column -t /proc/net/dev
Inter-|      Receive    |        Transmit
face         |bytes     packets  errs      drop  fifo  frame  compressed 
  multicast|bytes  packets    errs    drop  fifo
lo:          133093161  146045   0         0     0     0      0 
   0                133093161  146045  0     0     0
ens18:       166999724  256655   0         16    0     0      0 
   0                167638513  218267  0     0     0


The "errs" for ens18 are steadily increasing, but the "drop" column is 
steady. It's also curious the errors on loopback would also be increasing?


Other ideas on how to troubleshoot this would be greatly appreciated.

Thanks,
Alex

On Wed, Dec 20, 2023 at 9:27 PM Alex > wrote:


Hi,

On Wed, Dec 20, 2023 at 11:03 AM Kevin Korb via rsync
mailto:rsync@lists.samba.org>> wrote:

What is the error?  I assume you know that with that syntax the
filelist.txt is local rather than remote.


Yes, I do know it refers to the list of local files.

There is no error - it just hangs indefinitely until some timeout
period. This is what it looks like on the remote side:

$ ps ax|grep rsync
  324075 ?        Ss     0:00 rsync --server -logDtpRe.LsfxCIvu
. /var/tmp/one/
  324094 ?        S      0:00 rsync --server -logDtpRe.LsfxCIvu
. /var/tmp/one/

On the local side, with a few - added to rsync, I see this:

recv_file_list done
get_local_name count=9 /var/tmp/one/
generator starting pid=324075
delta-transmission enabled
recv_generator(config1.cf ,0)
config1.cf  is uptodate
send_files(0, config1.cf )

then it just stalls until it eventually times out. However, if I
remove any one of the nine files from the filelist, it completes
normally.

It actually also exhibits the same problem without a filelist at all
- as long as I'm transferring multiple files, it fails. I suppose if
I were syncing a directory with less than nine files, it might
succeed, but local directory to any remote directory on this one
server fails.

This also isn't a disk space or inode problem or corrupt filesystem
problem - the same command works from a different host to this one
problematic host without a problem.

I've also confirmed the open file limit is large enough on both sides:

# ulimit -n
5

I've now spent hours and hours trying to isolate and troubleshoot
this problem. It really feels like it's somehow related to the
number of files being transferred, not the size of the files. If I
break up a directory into multiple attempts, I can eventually
transfer all the files (maybe 40 in total), but trying to send all
40 files at once and none get transferred.

It almost certainly has something to do with building the initial
list of files. If that list is too large, the transfer fails, even
if the majority of the files in the source are already in the
destination.

It also happens when I'm pushing the files to the destination or
pulling them from the remote to local.

Thanks,
Alex






On 12/20/23 09:50, Alex via rsync wrote:
 > Hi, I've been using rsync on fedora over ssh to sync
directories for
 > decades, but suddenly having a problem with transferring
multiple files
 > at a time to one specific host using --files-from. I can't
think of what
 > might have changed to have caused this. Using rsync to
transfer a single
 > file to this problematic host works successfully. It appears
to be
 > related to the number of files in the --files-from filelist.
More than
 > nine and it stalls; less than nine and it finishes
successfully. I'm
 > using the same version of rsync on both sides.
 >
 > (Server) Protocol versions: remote=31, negotiated=31
 > (Client) Protocol versions: remote=31, negotiated=31
 >
 > When running the following command, it appears to collect the
list of
 > files to be transferred, successfully makes the connection
with the
 > remote server, but then just stalls. Using strace on the
remote side
 > shows rsync appears to be waiting for data.
 >
 > rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i
 > /root/keys/sync-key-v4' 

Re: rsync over ssh fails with --files-from

[Alex via rsync]

Can someone help me determine if these errors are normal or if this could
somehow be the cause? I've removed the last three columns for readability -
they were all zeros.

# column -t /proc/net/dev
Inter-|  Receive|Transmit
face |bytes packets  errs  drop  fifo  frame  compressed
 multicast|bytes  packetserrsdrop  fifo
lo:  133093161  146045   0 0 0 0  0   0
   133093161  146045  0 0 0
ens18:   166999724  256655   0 160 0  0   0
   167638513  218267  0 0 0

The "errs" for ens18 are steadily increasing, but the "drop" column is
steady. It's also curious the errors on loopback would also be increasing?

Other ideas on how to troubleshoot this would be greatly appreciated.

Thanks,
Alex

On Wed, Dec 20, 2023 at 9:27 PM Alex  wrote:

> Hi,
>
> On Wed, Dec 20, 2023 at 11:03 AM Kevin Korb via rsync <
> rsync@lists.samba.org> wrote:
>
>> What is the error?  I assume you know that with that syntax the
>> filelist.txt is local rather than remote.
>>
>
> Yes, I do know it refers to the list of local files.
>
> There is no error - it just hangs indefinitely until some timeout period.
> This is what it looks like on the remote side:
>
> $ ps ax|grep rsync
>  324075 ?Ss 0:00 rsync --server -logDtpRe.LsfxCIvu .
> /var/tmp/one/
>  324094 ?S  0:00 rsync --server -logDtpRe.LsfxCIvu .
> /var/tmp/one/
>
> On the local side, with a few - added to rsync, I see this:
>
> recv_file_list done
> get_local_name count=9 /var/tmp/one/
> generator starting pid=324075
> delta-transmission enabled
> recv_generator(config1.cf,0)
> config1.cf is uptodate
> send_files(0, config1.cf)
>
> then it just stalls until it eventually times out. However, if I remove
> any one of the nine files from the filelist, it completes normally.
>
> It actually also exhibits the same problem without a filelist at all - as
> long as I'm transferring multiple files, it fails. I suppose if I were
> syncing a directory with less than nine files, it might succeed, but local
> directory to any remote directory on this one server fails.
>
> This also isn't a disk space or inode problem or corrupt filesystem
> problem - the same command works from a different host to this one
> problematic host without a problem.
>
> I've also confirmed the open file limit is large enough on both sides:
>
> # ulimit -n
> 5
>
> I've now spent hours and hours trying to isolate and troubleshoot this
> problem. It really feels like it's somehow related to the number of files
> being transferred, not the size of the files. If I break up a directory
> into multiple attempts, I can eventually transfer all the files (maybe 40
> in total), but trying to send all 40 files at once and none get transferred.
>
> It almost certainly has something to do with building the initial list of
> files. If that list is too large, the transfer fails, even if the majority
> of the files in the source are already in the destination.
>
> It also happens when I'm pushing the files to the destination or pulling
> them from the remote to local.
>
> Thanks,
> Alex
>
>
>
>
>
>
>
>>
>> On 12/20/23 09:50, Alex via rsync wrote:
>> > Hi, I've been using rsync on fedora over ssh to sync directories for
>> > decades, but suddenly having a problem with transferring multiple files
>> > at a time to one specific host using --files-from. I can't think of
>> what
>> > might have changed to have caused this. Using rsync to transfer a
>> single
>> > file to this problematic host works successfully. It appears to be
>> > related to the number of files in the --files-from filelist. More than
>> > nine and it stalls; less than nine and it finishes successfully. I'm
>> > using the same version of rsync on both sides.
>> >
>> > (Server) Protocol versions: remote=31, negotiated=31
>> > (Client) Protocol versions: remote=31, negotiated=31
>> >
>> > When running the following command, it appears to collect the list of
>> > files to be transferred, successfully makes the connection with the
>> > remote server, but then just stalls. Using strace on the remote side
>> > shows rsync appears to be waiting for data.
>> >
>> > rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i
>> > /root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/
>> >
>> > sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of
>> > other ed25519 keys.
>> >
>> > Could it be related to packet size or some kind of network disparity?
>> > It's not related to the size of the files, as I've tried large and
>> small
>> > and it doesn't matter - if the number of files exceeds 9, it fails.
>> >
>> >
>> >
>> >
>> >
>> >
>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> Kevin Korb  Phone:(407) 252-6853
>> Systems Administrator   Internet:
>> FutureQuest, Inc.   

Re: rsync over ssh fails with --files-from

[Alex via rsync]

Hi,

On Wed, Dec 20, 2023 at 11:03 AM Kevin Korb via rsync 
wrote:

> What is the error?  I assume you know that with that syntax the
> filelist.txt is local rather than remote.
>

Yes, I do know it refers to the list of local files.

There is no error - it just hangs indefinitely until some timeout period.
This is what it looks like on the remote side:

$ ps ax|grep rsync
 324075 ?Ss 0:00 rsync --server -logDtpRe.LsfxCIvu .
/var/tmp/one/
 324094 ?S  0:00 rsync --server -logDtpRe.LsfxCIvu .
/var/tmp/one/

On the local side, with a few - added to rsync, I see this:

recv_file_list done
get_local_name count=9 /var/tmp/one/
generator starting pid=324075
delta-transmission enabled
recv_generator(config1.cf,0)
config1.cf is uptodate
send_files(0, config1.cf)

then it just stalls until it eventually times out. However, if I remove any
one of the nine files from the filelist, it completes normally.

It actually also exhibits the same problem without a filelist at all - as
long as I'm transferring multiple files, it fails. I suppose if I were
syncing a directory with less than nine files, it might succeed, but local
directory to any remote directory on this one server fails.

This also isn't a disk space or inode problem or corrupt filesystem problem
- the same command works from a different host to this one problematic host
without a problem.

I've also confirmed the open file limit is large enough on both sides:

# ulimit -n
5

I've now spent hours and hours trying to isolate and troubleshoot this
problem. It really feels like it's somehow related to the number of files
being transferred, not the size of the files. If I break up a directory
into multiple attempts, I can eventually transfer all the files (maybe 40
in total), but trying to send all 40 files at once and none get transferred.

It almost certainly has something to do with building the initial list of
files. If that list is too large, the transfer fails, even if the majority
of the files in the source are already in the destination.

It also happens when I'm pushing the files to the destination or pulling
them from the remote to local.

Thanks,
Alex







>
> On 12/20/23 09:50, Alex via rsync wrote:
> > Hi, I've been using rsync on fedora over ssh to sync directories for
> > decades, but suddenly having a problem with transferring multiple files
> > at a time to one specific host using --files-from. I can't think of what
> > might have changed to have caused this. Using rsync to transfer a single
> > file to this problematic host works successfully. It appears to be
> > related to the number of files in the --files-from filelist. More than
> > nine and it stalls; less than nine and it finishes successfully. I'm
> > using the same version of rsync on both sides.
> >
> > (Server) Protocol versions: remote=31, negotiated=31
> > (Client) Protocol versions: remote=31, negotiated=31
> >
> > When running the following command, it appears to collect the list of
> > files to be transferred, successfully makes the connection with the
> > remote server, but then just stalls. Using strace on the remote side
> > shows rsync appears to be waiting for data.
> >
> > rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i
> > /root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/
> >
> > sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of
> > other ed25519 keys.
> >
> > Could it be related to packet size or some kind of network disparity?
> > It's not related to the size of the files, as I've tried large and small
> > and it doesn't matter - if the number of files exceeds 9, it fails.
> >
> >
> >
> >
> >
> >
>
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb  Phone:(407) 252-6853
> Systems Administrator   Internet:
> FutureQuest, Inc.   ke...@futurequest.net  (work)
> Orlando, Floridak...@sanitarium.net (personal)
> Web page:   https://sanitarium.net/
> PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync over ssh fails with --files-from

[Kevin Korb via rsync]

What is the error?  I assume you know that with that syntax the 
filelist.txt is local rather than remote.


On 12/20/23 09:50, Alex via rsync wrote:
Hi, I've been using rsync on fedora over ssh to sync directories for 
decades, but suddenly having a problem with transferring multiple files 
at a time to one specific host using --files-from. I can't think of what 
might have changed to have caused this. Using rsync to transfer a single 
file to this problematic host works successfully. It appears to be 
related to the number of files in the --files-from filelist. More than 
nine and it stalls; less than nine and it finishes successfully. I'm 
using the same version of rsync on both sides.


(Server) Protocol versions: remote=31, negotiated=31
(Client) Protocol versions: remote=31, negotiated=31

When running the following command, it appears to collect the list of 
files to be transferred, successfully makes the connection with the 
remote server, but then just stalls. Using strace on the remote side 
shows rsync appears to be waiting for data.


rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i 
/root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/


sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of 
other ed25519 keys.


Could it be related to packet size or some kind of network disparity? 
It's not related to the size of the files, as I've tried large and small 
and it doesn't matter - if the number of files exceeds 9, it fails.









--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync over ssh fails with --files-from

[Alex via rsync]

Hi, I've been using rsync on fedora over ssh to sync directories for
decades, but suddenly having a problem with transferring multiple files at
a time to one specific host using --files-from. I can't think of what might
have changed to have caused this. Using rsync to transfer a single file to
this problematic host works successfully. It appears to be related to the
number of files in the --files-from filelist. More than nine and it stalls;
less than nine and it finishes successfully. I'm using the same version of
rsync on both sides.

(Server) Protocol versions: remote=31, negotiated=31
(Client) Protocol versions: remote=31, negotiated=31

When running the following command, it appears to collect the list of files
to be transferred, successfully makes the connection with the remote
server, but then just stalls. Using strace on the remote side shows rsync
appears to be waiting for data.

rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i
/root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/

sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of
other ed25519 keys.

Could it be related to packet size or some kind of network disparity? It's
not related to the size of the files, as I've tried large and small and it
doesn't matter - if the number of files exceeds 9, it fails.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync error messages always in English (= never translated)?

[rsync--- via rsync]

I want to recognize and handle some rsync error messages
in my log files (containing also the --itemize-changes output)
on different computers with different language/locale settings.

Can I rely on rsync to create only English error messages
to have a stable pattern to recognize?

PS: In the source code at https://github.com/WayneD/rsync I could not see
any indication of message translations...



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?

[Kevin Korb via rsync]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Unfortunately, exit 23 litterally just means something else went wrong
and might have scrolled off of the screen if you have rsync listing
files (--verbose or --itemize_changes).  Essentially, it is anything
that doesn't have its own exit code.  I just ignore it.  The most common
reaosn is file vanished.

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

On Thu, 14 Dec 2023, rsync--- via rsync wrote:


Date: Thu, 14 Dec 2023 19:20:15 +0100
From: rsync--- via rsync 
To: rsync@lists.samba.org
Subject: rsync exit code 23 (partial transfer due to errors): List of possible
 reasons and how to ignore some?

I am trying to find a solution for the open source Linux software

  "Back In Time" (https://github.com/bit-team/backintime)

where we evaluate the rsync exit code when taking a backup via rsync
and inform the user that an error has occured.

Questions:

1. Is there full list of possible reasons available that make rsync
  exit with the return value 23?

  'rsync error: some files/attrs were not transferred (see previous errors) 
(code 23) at main.c(1338) [sender=3.2.7]'

  I want to decide for each reason if we treat it as an error or warning.

2. I want to ignore some reasons for exit code 23 by only logging it, mainly:

  symlink has no referent: "/home/user/Documents/dead-link"

  Is there a way to
  - either prevent that this error leads to exit code 23 (if no other reason 
occurs)
  - or to prevent this specific check at all (but eg. copy the symlink "as is")?

THX a lot!



PS 1: The typcially used rsync command line looks like this:

rsync --recursive --times --devices --specials --hard-links --human-readable -s 
--copy-links --acls --perms --executability --group --owner --
info=progress2 --no-inc-recursive -l --delete --delete-excluded -v -i 
--out-format=BACKINTIME: %i %n%L --link-dest=../../20231214-143359-584/backup --
chmod=Du+wx --exclude=/home/username/temp/testBAK_profil1 
--exclude=/home/username/.local/share/backintime 
--exclude=.local/share/backintime/mnt --
include=/home/username/Documents/ --include=/home/username/ --include=/home/ 
--include=/home/username/temp/deleted_folder/ --
include=/home/username/temp/ --include=/home/username/.mozilla/ --exclude=.gvfs 
--exclude=.cache/* --exclude=.thumbnails* --
exclude=.local/share/[Tt]rash* --exclude=*.backup* --exclude=*~ 
--exclude=.dropbox* --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* --
exclude=/run/* --exclude=/etc/mtab --exclude=/var/cache/apt/archives/*.deb 
--exclude=lost+found/* --exclude=/tmp/* --exclude=/var/tmp/* --
exclude=/var/backups/* --exclude=.Private --exclude=lock 
--exclude=root_only_file.txt --include=/home/username/Documents/** --
include=/home/username/temp/deleted_folder/** 
--include=/home/username/.mozilla/** --exclude=* /
/home/username/temp/testBAK_profil1/backintime/computer1/username/1/new_snapshot/backup




PS 2: You can find more details or even answer in our Github issue too:

https://github.com/bit-team/backintime/issues/1587




-BEGIN PGP SIGNATURE-

iF0EARECAB0WIQSHERqysePm7S8yuR9UoLWOVtABBwUCZXtTzQAKCRBUoLWOVtAB
B2CnAJ9YGQ/gXTPP2Ntg3arHHDC11cRnfgCeJVpDOnGhTH0OreZfj8pIbnsL3SI=
=Eqei
-END PGP SIGNATURE-


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?

[rsync--- via rsync]

On Thu, 2023-12-14 at 14:09 -0500, Kevin Korb wrote:
> Unfortunately, exit 23 litterally just means something else went wrong
> and might have scrolled off of the screen if you have rsync listing
> files (--verbose or --itemize_changes).  Essentially, it is anything
> that doesn't have its own exit code.  I just ignore it.  The most common
> reaosn is file vanished.

THX for sharing your experiences and knowledge :-)

I have just tried to "reverse engineer" the possible reasons from the source 
code
and have found 21 reasons that I hope will never happen ;-)

So ignoring (or treating as a warning only) sound as best option so far.

-

Looking into the rsync source code I can see only one location where exit code 
23 is set:

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/errcode.h#L42
#define RERR_PARTIAL23  /* partial transfer */

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/log.c#L97
{ RERR_PARTIAL, "some files/attrs were not transferred (see previous 
errors)" },


https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/cleanup.c#L217-L218
if (io_error & IOERR_GENERAL || got_xfer_error)
exit_code = RERR_PARTIAL;



So the question is which reasons cause
- IOERR_GENERAL
- got_xfer_error
to be true?



IOERR_GENERAL is set for different reasons (first line is the log output format 
string):

1. receive_sums failed [what is that at all?]:
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L345-L348

2. send_files failed to open %s
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L358-L362

3. fstat failed
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L373C33-L373C45

4. read errors mapping %s
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L433-L436

5. change_dir %s failed
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L366-L369
 
6. skipping overly long name: %s
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1245-L1247

7. symlink has no referent: %s
   See the source code comments there when symlinks are checked:
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1271C1-L1282C28
   
8. readlink_stat(%s) failed
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1293-L1295

9. skipping file with bogus (zero) st_mode: %s
   
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1298-L1302

10. skipping symlink with 0-length value: %s

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1565-L1566

11. [%s] cannot convert filename: %s (%s)

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1595-L1601

12. [%s] cannot convert symlink data for: %s (%s)

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1609-L1614
   

13. get_acl(fname, ) < 0   // with no explicit error message!

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1628-L1632
   
14. get_xattr(fname, ) < 0  // with no explicit error message!

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1637-L1642

15. link_stat %s failed

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1809-L1811

16. opendir %s failed

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1841-L1843

17. filename overflows max-path len by %u: %s/%s

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1863-L1871

18. cannot send file with empty name in %s

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1873-L1877
   
19. readdir(%s)

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1886-L1889
   
20. link_stat %s failed

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L2396-L2399

21. cannot add local filter rules in long-named directory: %s

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/exclude.c#L815-L818



-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?

[rsync--- via rsync]

I am trying to find a solution for the open source Linux software

   "Back In Time" (https://github.com/bit-team/backintime)

where we evaluate the rsync exit code when taking a backup via rsync
and inform the user that an error has occured.

Questions:

1. Is there full list of possible reasons available that make rsync
   exit with the return value 23?

   'rsync error: some files/attrs were not transferred (see previous 
errors) (code 23) at main.c(1338) [sender=3.2.7]'

   I want to decide for each reason if we treat it as an error or warning.

2. I want to ignore some reasons for exit code 23 by only logging it, mainly:
  
   symlink has no referent: "/home/user/Documents/dead-link"

   Is there a way to 
   - either prevent that this error leads to exit code 23 (if no other reason 
occurs)
   - or to prevent this specific check at all (but eg. copy the symlink "as 
is")?

THX a lot!



PS 1: The typcially used rsync command line looks like this:

rsync --recursive --times --devices --specials --hard-links --human-readable -s 
--copy-links --acls --perms --executability --group --owner --
info=progress2 --no-inc-recursive -l --delete --delete-excluded -v -i 
--out-format=BACKINTIME: %i %n%L --link-dest=../../20231214-143359-584/backup --
chmod=Du+wx --exclude=/home/username/temp/testBAK_profil1 
--exclude=/home/username/.local/share/backintime 
--exclude=.local/share/backintime/mnt --
include=/home/username/Documents/ --include=/home/username/ --include=/home/ 
--include=/home/username/temp/deleted_folder/ --
include=/home/username/temp/ --include=/home/username/.mozilla/ --exclude=.gvfs 
--exclude=.cache/* --exclude=.thumbnails* --
exclude=.local/share/[Tt]rash* --exclude=*.backup* --exclude=*~ 
--exclude=.dropbox* --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* --
exclude=/run/* --exclude=/etc/mtab --exclude=/var/cache/apt/archives/*.deb 
--exclude=lost+found/* --exclude=/tmp/* --exclude=/var/tmp/* --
exclude=/var/backups/* --exclude=.Private --exclude=lock 
--exclude=root_only_file.txt --include=/home/username/Documents/** --
include=/home/username/temp/deleted_folder/** 
--include=/home/username/.mozilla/** --exclude=* /
/home/username/temp/testBAK_profil1/backintime/computer1/username/1/new_snapshot/backup




PS 2: You can find more details or even answer in our Github issue too:

https://github.com/bit-team/backintime/issues/1587


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync checksum inquiry

[brent kimberley via rsync]

For example,  is there any reason why rsync doesn't support blake2b( on 64b
engines ) and blake2s ( on "tiny" engines )?

On Sun, Oct 29, 2023, 5:49 p.m. brent kimberley 
wrote:

> Hi.
> What is the process for deciding what types of checksums can be included
> with rsync?
>
> Best Regards,
> Brent
>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync checksum inquiry

[brent kimberley via rsync]

What is the process for deciding what types of checksums can be included
with rsync?

>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[rsync--- via rsync]

Here is the missing attachment ;-)



On Fri, 2023-09-22 at 21:01 +0200, rsync--- via rsync wrote:
> On Fri, 2023-09-22 at 07:37 -0400, Kevin Korb wrote:
> > So I decided to do a quick test using the Linux kernel source tree since 
> > it has lots of files.
> 
> Excellent idea using kernel sources! A lot of different files...
> I will use this to create indicative benchmarks for different scenarios...
> 
> >   I duplicated a tree, used 'find . -type f -exec 
> > chmod 444 {} +' to make read only files for rsync to want to chmod, then 
> > used cp -al to make several duplicate trees using hard linked files.
> > [...]
> > But also, I did not experience the problem you are describing.  My 
> > surviving 
> > hard links in the duplicate trees were still 444.
> 
> If attached a script that does the same (with one file instead of the kernel 
> source tree)
> but in my case 444 became 644 again.
> 
> - Q: Which rsync version, distro and file system do you use?
> 
> - Q: Does my attached script reproduce the permission change?
> 
> - Q: Did my script attached to the initial post here reproduce the permission 
> change?
> 
> 
> 
> PS: In my case the attached script shows (excerpt):
> 
> ./setup_cp_al.sh
> 
> Tested with
> - rsync  version 3.2.7  protocol version 31
> - ext4 file system
> - Ubuntu 22.04
> 
> File stats BEFORE rsync --delete:
> 
>   File: snapshot2/read_only.txt
>   Size: 0   Blocks: 0  IO Block: 4096   regular empty file
> Device: 10305h/66309d   Inode: 17571021    Links: 3
> Access: (0444/-r--r--r--)  Uid: ( 1000/ user1)   Gid: ( 1000/ user1)
> Access: 2023-09-22 20:51:16.690961150 +0200
> Modify: 2023-09-22 20:51:16.690961150 +0200
> Change: 2023-09-22 20:51:16.694961109 +0200
>  Birth: 2023-09-22 20:51:16.690961150 +0200
> 
> File stats AFTER rsync --delete
> 
>   File: snapshot2/read_only.txt
>   Size: 0   Blocks: 0  IO Block: 4096   regular empty file
> Device: 10305h/66309d   Inode: 17571021    Links: 2
> Access: (0644/-rw-r--r--)  Uid: ( 1000/ user1)   Gid: ( 1000/ user1)
> Access: 2023-09-22 20:51:16.690961150 +0200
> Modify: 2023-09-22 20:51:16.690961150 +0200
> Change: 2023-09-22 20:51:16.694961109 +0200
>  Birth: 2023-09-22 20:51:16.690961150 +0200
> 
> Results (after deleting snapshot1 via rsync --delete):
> 1. The 'Change' time of the hardlinked file was updated
> 2. The hardlinked file has now different access rights (644 instead of 444 so 
> it is writable again!)
> This does NOT happen if 'rm -f snapshot1/' is used!
> 
> 



setup_cp_al.sh
Description: application/shellscript
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[rsync--- via rsync]

On Fri, 2023-09-22 at 07:37 -0400, Kevin Korb wrote:
> So I decided to do a quick test using the Linux kernel source tree since 
> it has lots of files.

Excellent idea using kernel sources! A lot of different files...
I will use this to create indicative benchmarks for different scenarios...

>   I duplicated a tree, used 'find . -type f -exec 
> chmod 444 {} +' to make read only files for rsync to want to chmod, then 
> used cp -al to make several duplicate trees using hard linked files.
> [...]
> But also, I did not experience the problem you are describing.  My surviving 
> hard links in the duplicate trees were still 444.

If attached a script that does the same (with one file instead of the kernel 
source tree)
but in my case 444 became 644 again.

- Q: Which rsync version, distro and file system do you use?

- Q: Does my attached script reproduce the permission change?

- Q: Did my script attached to the initial post here reproduce the permission 
change?



PS: In my case the attached script shows (excerpt):

./setup_cp_al.sh

Tested with
- rsync  version 3.2.7  protocol version 31
- ext4 file system
- Ubuntu 22.04

File stats BEFORE rsync --delete:

  File: snapshot2/read_only.txt
  Size: 0   Blocks: 0  IO Block: 4096   regular empty file
Device: 10305h/66309d   Inode: 17571021Links: 3
Access: (0444/-r--r--r--)  Uid: ( 1000/ user1)   Gid: ( 1000/ user1)
Access: 2023-09-22 20:51:16.690961150 +0200
Modify: 2023-09-22 20:51:16.690961150 +0200
Change: 2023-09-22 20:51:16.694961109 +0200
 Birth: 2023-09-22 20:51:16.690961150 +0200

File stats AFTER rsync --delete

  File: snapshot2/read_only.txt
  Size: 0   Blocks: 0  IO Block: 4096   regular empty file
Device: 10305h/66309d   Inode: 17571021Links: 2
Access: (0644/-rw-r--r--)  Uid: ( 1000/ user1)   Gid: ( 1000/ user1)
Access: 2023-09-22 20:51:16.690961150 +0200
Modify: 2023-09-22 20:51:16.690961150 +0200
Change: 2023-09-22 20:51:16.694961109 +0200
 Birth: 2023-09-22 20:51:16.690961150 +0200

Results (after deleting snapshot1 via rsync --delete):
1. The 'Change' time of the hardlinked file was updated
2. The hardlinked file has now different access rights (644 instead of 444 so 
it is writable again!)
This does NOT happen if 'rm -f snapshot1/' is used!


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[Paul Slootman via rsync]

On Fri 22 Sep 2023, Kevin Korb via rsync wrote:

> 444 {} +' to make read only files for rsync to want to chmod, then used cp
> -al to make several duplicate trees using hard linked files.  An rm -rf on
> one such tree took .97 seconds while an rsync deletion took 1.25 seconds.

Be sure to drop the caches before such tests every time:
echo 3 > /proc/sys/vm/drop_caches


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[Kevin Korb via rsync]

So I decided to do a quick test using the Linux kernel source tree since 
it has lots of files.  I duplicated a tree, used 'find . -type f -exec 
chmod 444 {} +' to make read only files for rsync to want to chmod, then 
used cp -al to make several duplicate trees using hard linked files.  An 
rm -rf on one such tree took .97 seconds while an rsync deletion took 
1.25 seconds.  Clearly I need a bigger test to play with as that margin 
could easily change just by different outputs if either had a -v.  But 
also, I did not experience the problem you are describing.  My surviving 
hard links in the duplicate trees were still 444.


BTW, it seems that rsync uses unlink() while rm uses unlinkat().  No 
idea if/why there is a performance diff there but I didn't look into it 
as it is now time for me to go to work.


On 9/22/23 04:14, rs...@altfeld-im.de wrote:

On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote:


I have heard in the past that rsyncing an empty dir over a tree to
delete the tree is faster than an rm -rf but I can't say I have ever
benchmarked it to get any actual numbers.


This **may** indeed be a myth (for a long time now) re-cited again and again and
- could no longer be valid today
- could apply only when deleting explicitly named files but not deleting the 
complete folder
   (as we need to do in "Back in Time")

At least I could not find a holistic benchmark with many files and different 
scenarios
(file systems, rsync'ing locally vs. over network, snapshot sizes, number of 
files, file sizes, rsync and rm versions...)

Q: Does `rsync` provide a test case that I could use as basis to prepare such a 
holistic benchmark?


But now that I am hearing
that rsync actually adds a bunch of pointless chmods to the process.  Is
it still faster given this problem?  If so maybe we should be trying to
investigate why rm is so slow.


Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not.




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[rsync--- via rsync]

On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote:

> I have heard in the past that rsyncing an empty dir over a tree to 
> delete the tree is faster than an rm -rf but I can't say I have ever 
> benchmarked it to get any actual numbers.

This **may** indeed be a myth (for a long time now) re-cited again and again and
- could no longer be valid today
- could apply only when deleting explicitly named files but not deleting the 
complete folder
  (as we need to do in "Back in Time")

At least I could not find a holistic benchmark with many files and different 
scenarios
(file systems, rsync'ing locally vs. over network, snapshot sizes, number of 
files, file sizes, rsync and rm versions...)

Q: Does `rsync` provide a test case that I could use as basis to prepare such a 
holistic benchmark?

> But now that I am hearing 
> that rsync actually adds a bunch of pointless chmods to the process.  Is 
> it still faster given this problem?  If so maybe we should be trying to 
> investigate why rm is so slow.

Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not.




-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[Kevin Korb via rsync]

I had intended to come back to this but because I didn't really think I 
had anything to add to the discussion I put it at a low enough priority 
that I forgot about it.  But I saw your bug report and was surprised to 
see that I was already unhelpful on this topic but because that original 
poster didn't have access to do an rm while you are trying to get a 
performance boost.


I have heard in the past that rsyncing an empty dir over a tree to 
delete the tree is faster than an rm -rf but I can't say I have ever 
benchmarked it to get any actual numbers.  But now that I am hearing 
that rsync actually adds a bunch of pointless chmods to the process.  Is 
it still faster given this problem?  If so maybe we should be trying to 
investigate why rm is so slow.  Otherwise, I would agree that an 
optimization to not do a bunch of pointless chmods is a good idea.  I 
would suspect that they are only there to prevent permission denied 
errors perhaps on some obscure filesystem (I tested on vfat since it is 
so non-compatible but read only files unlink fine there).


On 9/18/23 19:42, rsync--- via rsync wrote:

Context
---

I am one of the active developers of the open source application "Back in Time"
which uses "rsync" as backend and I want to fix an open issue:

 "Back in Time"-Bug:
 https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507

"Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking
unchanged files in the backups/snapshots and tries to keep the permissions
by using "--perms --group --owner" by default.

Older snapshots are then deleted according to a schedule.



Issue
-

When deleting a complete snapshot folder with rsync using an empty source folder
(which is a best practice for faster deletion than "rm -f") the permissions of 
hardlinked
files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted 
files
with existing hardlinks:

 mkdir empty
 rsync -a --delete -s empty/ snapshot1/

This distorts the backup history.

There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the 
issue is still unfixed:

 https://bugzilla.samba.org/show_bug.cgi?id=12806



Questions
-

1. Is there a reliable workaround ("--super" is proposed in the issue but may 
probably not always work)?
2. If a rsync developer is reading this:  Is there any chance to fix this?



Steps to reproduce
--

I have attached a bash script "setup.sh" as txt file which makes the issue 
fully reproducible (on Ubuntu 22.04).




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12806

--- Comment #7 from Aryo Da  ---
I think this a severe bug for all backup use cases of rsync that take a full
snapshot with permissions (--perms) by creating hardlinks to unchanged files +
copies of changed files (--link-dest):

-> Whenever an old snapshot is deleted with `rsync --delete` the permissions of
the hardlinked files change and trigger another copy of the files in a new
snapshot (= waste of storage space and backup run-time).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 12806] Deleting in a row of hardlinked snapshots resets file permissions.

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=12806

--- Comment #6 from Aryo Da  ---
Created attachment 18117
  --> https://bugzilla.samba.org/attachment.cgi?id=18117=edit
MRE (minimal reproducible example) as bash script to reproduce the bug

Rename to "setup.sh" and make it executable...

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[rsync--- via rsync]

Context
---

I am one of the active developers of the open source application "Back in Time"
which uses "rsync" as backend and I want to fix an open issue:

"Back in Time"-Bug:
https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507

"Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking
unchanged files in the backups/snapshots and tries to keep the permissions
by using "--perms --group --owner" by default.

Older snapshots are then deleted according to a schedule.



Issue
-

When deleting a complete snapshot folder with rsync using an empty source folder
(which is a best practice for faster deletion than "rm -f") the permissions of 
hardlinked
files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted 
files
with existing hardlinks:

mkdir empty
rsync -a --delete -s empty/ snapshot1/

This distorts the backup history.

There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the 
issue is still unfixed:

https://bugzilla.samba.org/show_bug.cgi?id=12806



Questions
-

1. Is there a reliable workaround ("--super" is proposed in the issue but may 
probably not always work)?
2. If a rsync developer is reading this:  Is there any chance to fix this?



Steps to reproduce
--

I have attached a bash script "setup.sh" as txt file which makes the issue 
fully reproducible (on Ubuntu 22.04).
#!/bin/bash
clear
echo
echo "Minimal reproducible example (MRE)"
echo "--"
echo
echo "Tested with"
echo "- rsync  version 3.2.7  protocol version 31"
echo "- ext4 file system"
echo
rm -rf MRE
mkdir MRE
cd MRE
mkdir source
touch source/read_only.txt
chmod 444 source/read_only.txt

mkdir snapshot1
# rsync --recursive --times --devices --specials --hard-links --human-readable 
-s --links --perms --executability --group --owner source/* snapshot1/
rsync --perms --executability --group --owner source/* snapshot1/

mkdir snapshot2
# rsync --recursive --times --devices --specials --hard-links --human-readable 
-s --links --perms --executability --group --owner --link-dest=../snapshot1 
source/* snapshot2/
rsync --perms --executability --group --owner --link-dest=../snapshot1 source/* 
snapshot2/
# stat snapshot1/read_only.txt 
echo "File stats BEFORE rsync --delete":
echo
stat snapshot2/read_only.txt

mkdir empty
# delete snapshot1 using an empty source directory (shall be the fastest way to 
delete a subdir)
# --debug=ALL
rsync -a --delete -s empty/ snapshot1/
# rm -f snapshot1/  # works as expected (unlink does neither change 'Changed' 
time nor file permissions)
# strace -c rm -rf snapshot1
echo
stat snapshot2/read_only.txt

echo
echo "Results (after deleting snapshot1 via rsync --delete):"
echo "1. The 'Change' time of the hardlinked file was updated"
echo "2. The hardlinked file has now different access rights (644 instead of 
444 so it is writable again!)"
echo "This does NOT happen if 'rm -f snapshot1/' is used!"
echo

echo "Files in snapshot2:"
ls -l snapshot2

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Feature Concept: enable iCloud Drive with rsync

[Kevin Korb via rsync]

Is this being accessed via a fuse mount?  If so it seems like that is 
where this kind of feature should be implemented (like a mount option to 
decide how to handle such files).  Rsync shouldn't need special features 
to deal with every kind of file storage.


On 9/12/23 05:22, Brian "bex" Exelbierd via rsync wrote:

Hi,

I have also posted this on GitHub but it isn’t clear that was the right 
place: https://github.com/WayneD/rsync/issues/522


iCloud Drive will evict files that are unused or when additional space 
is needed on the local drive. The evicted files are replace by 
"bookmark" files that allow MacOS to continue to report the files in the 
file system as though they were actually present. The files are 
downloaded again either on request or when needed.


rsync, like all similar tools I can find, doesn't have any way of 
handling these evicted files. I have been thinking about this and I 
think I know how to make it work.


The short explanation is that iCloud Drive preserves access to the 
required metadata. Here is an abbreviated output from `mdls` of an 
evicted file:


```
% mdls .1998-tax-return.pdf.icloud
kMDItemContentCreationDate = 2018-08-28 18:32:24 +
kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +
kMDItemContentModificationDate = 2018-08-28 18:32:24 +
kMDItemDateAdded = 2021-11-05 15:42:15 +
kMDItemDisplayName = "1998-tax-return.pdf"
kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +
kMDItemFSCreationDate = 2018-08-28 18:32:24 +
kMDItemFSSize = 1929932
kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +
kMDItemLogicalSize = 1929932
kMDItemPhysicalSize = 1929932
```

This metadata, I think, is enough to pass the rsync quick check as 
described in the man page. Therefore, I suspect what needs to happen, 
from a code perspective, is that rsync needs to be modified to do the 
following when it finds an evicted file. All evicted files are named 
consistently, `..iCloud` so they are easy to spot.


1. Perform the system call equivalent of the mdls above to obtain the 
appropriate dates and sizes.
2. If the file cannot be skipped because it either has changed or a 
checksum is required, perform the system call equivalent of `brctl 
download ` to get the file downloaded.

3. Rsync the file
4. If the file was downloaded by rsync, perform the system call 
equivalent of `brctl evict ` to remove the file to leave the 
system in the same state.


This simplistic algorithm would leave some open issues/caveats:
- It is likely that the rsync can only be run one-way from iCloud Drive 
to non-iCloud Drive data storage. While it is possible it could be run 
two-way research is needed on whether you’d have to download the old 
file before you replace it.
- Timeouts may happen. iCloud Drive still gets stuck sometimes and there 
will be non-zero time during downloads that don’t get stuck.
- There is no guarantee there is ever enough room on disk to hold a 
specific file from iCloud Drive. This is likely resolved by MacOS 
directly, however, this may take excessive time.
- If you thought running with checksums was slow before, strap in, 
because those downloads are going to add up.


All that said, I think this would accomplish a high percentage of what 
users are looking for.


Is this kind of a patch welcome? Is the plan sound? I haven't written C 
since college (in the 90s!) and would be open to advice/mentoring.


Thank you.

regards,

bex



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Feature Concept: enable iCloud Drive with rsync

[Brian "bex" Exelbierd via rsync]

Hi,

I have also posted this on GitHub but it isn’t clear that was the right place: 
https://github.com/WayneD/rsync/issues/522

iCloud Drive will evict files that are unused or when additional space is 
needed on the local drive. The evicted files are replace by "bookmark" files 
that allow MacOS to continue to report the files in the file system as though 
they were actually present. The files are downloaded again either on request or 
when needed.

rsync, like all similar tools I can find, doesn't have any way of handling 
these evicted files. I have been thinking about this and I think I know how to 
make it work.

The short explanation is that iCloud Drive preserves access to the required 
metadata. Here is an abbreviated output from `mdls` of an evicted file:

```
% mdls .1998-tax-return.pdf.icloud
kMDItemContentCreationDate = 2018-08-28 18:32:24 +
kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +
kMDItemContentModificationDate = 2018-08-28 18:32:24 +
kMDItemDateAdded = 2021-11-05 15:42:15 +
kMDItemDisplayName = "1998-tax-return.pdf"
kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +
kMDItemFSCreationDate = 2018-08-28 18:32:24 +
kMDItemFSSize = 1929932
kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +
kMDItemLogicalSize = 1929932
kMDItemPhysicalSize = 1929932
```

This metadata, I think, is enough to pass the rsync quick check as described in 
the man page. Therefore, I suspect what needs to happen, from a code 
perspective, is that rsync needs to be modified to do the following when it 
finds an evicted file. All evicted files are named consistently, 
`..iCloud` so they are easy to spot.

1. Perform the system call equivalent of the mdls above to obtain the 
appropriate dates and sizes.
2. If the file cannot be skipped because it either has changed or a checksum is 
required, perform the system call equivalent of `brctl download ` to 
get the file downloaded.
3. Rsync the file
4. If the file was downloaded by rsync, perform the system call equivalent of 
`brctl evict ` to remove the file to leave the system in the same 
state.

This simplistic algorithm would leave some open issues/caveats:
- It is likely that the rsync can only be run one-way from iCloud Drive to 
non-iCloud Drive data storage. While it is possible it could be run two-way 
research is needed on whether you’d have to download the old file before you 
replace it.
- Timeouts may happen. iCloud Drive still gets stuck sometimes and there will 
be non-zero time during downloads that don’t get stuck.
- There is no guarantee there is ever enough room on disk to hold a specific 
file from iCloud Drive. This is likely resolved by MacOS directly, however, 
this may take excessive time.
- If you thought running with checksums was slow before, strap in, because 
those downloads are going to add up.

All that said, I think this would accomplish a high percentage of what users 
are looking for.

Is this kind of a patch welcome? Is the plan sound? I haven't written C since 
college (in the 90s!) and would be open to advice/mentoring.

Thank you.

regards,

bex
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Why try to update (some) permissions which are the same?

[Hardy via rsync]

Am 06.09.23 um 08:49 schrieb Paul Slootman via rsync:


The current version is 3.2.7, especially 2.6.8 is quite ancient.
You may want to upgrade before going bug hunting, chances are your
problem has already been fixed.


Oh yes, exactly the step to 3.* did a lot both to option and PERFORMANCE!


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Why try to update (some) permissions which are the same?

[Paul Slootman via rsync]

On Sun 03 Sep 2023, Perry Hutchison via rsync wrote:

> On the source system:
> 
> $ rsync --version
> rsync  version 2.6.8  protocol version 29

> On the destination system:

> $ rsync --version
> rsync  version 3.0.7  protocol version 30

The current version is 3.2.7, especially 2.6.8 is quite ancient.
You may want to upgrade before going bug hunting, chances are your
problem has already been fixed.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Why try to update (some) permissions which are the same?

[Perry Hutchison via rsync]

Kevin Korb  wrote:
> On Sun, 3 Sep 2023, Perry Hutchison via rsync wrote:
> > On the source system:
> > ...
> > $ ll -d fcst-200[89] fcst-201[01]
> > dr-xr-xr-x  2 perryh  perryh  7168 Nov 27  2009 fcst-2008
> > dr-xr-xr-x  2 perryh  perryh  9216 Jul 21  2010 fcst-2009
> > drwxr-xr-x  2 perryh  perryh  9216 Jul  7  2011 fcst-2010
> > drwxr-xr-x  2 perryh  perryh  4608 Jul  7  2011 fcst-2011
> >
> > $ rsync -uHSxz4rlptOgv --itemize-changes -e rsh fcst-200[89] fcst-201[01] 
> > fbsd81:/home/perryh
> > building file list ... done
> > rsync: failed to modify permissions on "/home/perryh/fcst-2008": Operation 
> > not permitted (1)
> > rsync: failed to modify permissions on "/home/perryh/fcst-2009": Operation 
> > not permitted (1)
> > ...
> >
> > On the destination system:
> >
> > $ ll -d fcst-200[89] fcst-201[01]
> > dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 7168 Nov 27  2009 fcst-2008
> > dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul 21  2010 fcst-2009
> > drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul  7  2011 fcst-2010
> > drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 4608 Jul  7  2011 fcst-2011
> > ...
> >
> > The question is:  why does rsync attempt (and fail) to change the
> > permissions of two destination directories, and not the other two,
> > when the permissions of all four destination directories already
> > match the corresponding source directories?  (The change attempt
> > fails because of the destinations' uchg,uunlnk flags, but as far
> > as I can see the change should never have been attempted in the
> > first place.)
>
> You have --itemize-changes but either it didn't or you filtered it out.

I had noticed that adding --itemize-changes to the command line did
not result in any additional output.  Dunno why, nor whether the two
issues might be related.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Why try to update (some) permissions which are the same?

[Kevin Korb via rsync]

You have --itemize-changes but either it didn't or you filtered it out.

--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

On Sun, 3 Sep 2023, Perry Hutchison via rsync wrote:


Date: Sun, 03 Sep 2023 00:35:29 -0700
From: Perry Hutchison via rsync 
To: rsync@lists.samba.org
Subject: Why try to update (some) permissions which are the same?

On the source system:

$ rsync --version
rsync  version 2.6.8  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.

Capabilities: 64-bit files, socketpairs, hard links, ACLs, symlinks, batchfiles,
 inplace, IPv6, file flags, 32-bit system inums, 64-bit internal 
inums
...

$ ll -d fcst-200[89] fcst-201[01]
dr-xr-xr-x  2 perryh  perryh  7168 Nov 27  2009 fcst-2008
dr-xr-xr-x  2 perryh  perryh  9216 Jul 21  2010 fcst-2009
drwxr-xr-x  2 perryh  perryh  9216 Jul  7  2011 fcst-2010
drwxr-xr-x  2 perryh  perryh  4608 Jul  7  2011 fcst-2011

$ rsync -uHSxz4rlptOgv --itemize-changes -e rsh fcst-200[89] fcst-201[01] 
fbsd81:/home/perryh
building file list ... done
rsync: failed to modify permissions on "/home/perryh/fcst-2008": Operation not 
permitted (1)
rsync: failed to modify permissions on "/home/perryh/fcst-2009": Operation not 
permitted (1)

sent 15794 bytes  received 20 bytes  31628.00 bytes/sec
total size is 278224  speedup is 17.59
rsync error: some files could not be transferred (code 23) at main.c(892) 
[sender=2.6.8]


On the destination system:

$ ll -d fcst-200[89] fcst-201[01]
dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 7168 Nov 27  2009 fcst-2008
dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul 21  2010 fcst-2009
drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul  7  2011 fcst-2010
drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 4608 Jul  7  2011 fcst-2011

$ rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
   64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
   socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
   append, ACLs, xattrs, no iconv, symtimes
...


The question is:  why does rsync attempt (and fail) to change the
permissions of two destination directories, and not the other two,
when the permissions of all four destination directories already
match the corresponding source directories?  (The change attempt
fails because of the destinations' uchg,uunlnk flags, but as far
as I can see the change should never have been attempted in the
first place.)




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Why try to update (some) permissions which are the same?

[Perry Hutchison via rsync]

On the source system:

$ rsync --version
rsync  version 2.6.8  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.

Capabilities: 64-bit files, socketpairs, hard links, ACLs, symlinks, batchfiles,
  inplace, IPv6, file flags, 32-bit system inums, 64-bit internal 
inums
...

$ ll -d fcst-200[89] fcst-201[01]
dr-xr-xr-x  2 perryh  perryh  7168 Nov 27  2009 fcst-2008
dr-xr-xr-x  2 perryh  perryh  9216 Jul 21  2010 fcst-2009
drwxr-xr-x  2 perryh  perryh  9216 Jul  7  2011 fcst-2010
drwxr-xr-x  2 perryh  perryh  4608 Jul  7  2011 fcst-2011

$ rsync -uHSxz4rlptOgv --itemize-changes -e rsh fcst-200[89] fcst-201[01] 
fbsd81:/home/perryh
building file list ... done
rsync: failed to modify permissions on "/home/perryh/fcst-2008": Operation not 
permitted (1)
rsync: failed to modify permissions on "/home/perryh/fcst-2009": Operation not 
permitted (1)

sent 15794 bytes  received 20 bytes  31628.00 bytes/sec
total size is 278224  speedup is 17.59
rsync error: some files could not be transferred (code 23) at main.c(892) 
[sender=2.6.8]


On the destination system:

$ ll -d fcst-200[89] fcst-201[01]
dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 7168 Nov 27  2009 fcst-2008
dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul 21  2010 fcst-2009
drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul  7  2011 fcst-2010
drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 4608 Jul  7  2011 fcst-2011

$ rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, no iconv, symtimes
...


The question is:  why does rsync attempt (and fail) to change the
permissions of two destination directories, and not the other two,
when the permissions of all four destination directories already
match the corresponding source directories?  (The change attempt
fails because of the destinations' uchg,uunlnk flags, but as far
as I can see the change should never have been attempted in the
first place.)

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

sync based on filename

[Fourhundred Thecat via rsync]

Hello,

I need to sync SRC to DST, but:

 - not based on file modification times
 - and not based on filesize

 + but on filename. Ie if this is current state, then sync:

  /SRC/2023-08-21 /DST/2023-08-19

but not if /DST contains newer directory (by name, not by modification time)

so, imagine i have this:

if /SRC/1980-01-01 has newer modification time than /DST/2023-08-21,
then rsync -r --delete /SRC /DST will overwrirte  2023-08-21 with
1980-01-01. which is not what I want.

I only need to overwrite if source is "newer" in sense of "date filename"

is this possible with rsync ?

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: print only first level directory name when copying files

[Kevin Korb via rsync]

find /mnt/foo/* -maxdepth 0 -print -exec rsync -an `realpath {}` 
/mnt/bar/ \;

 (realpath eliminates the trailing slashes)

On 8/3/23 22:27, Fourhundred Thecat via rsync wrote:

Hello,

I am copying /mnt/foo to /mnt/bar/

   rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/

/mnt/foo contains deep directory structure, ie:

   /mnt/foo/aaa/
   /mnt/foo/aaa/somestuff/
   /mnt/foo/aaa/somestuff/file1

   /mnt/foo/bbb/
   /mnt/foo/bbb/someotherstuff/
   /mnt/foo/bbb/someotherstuff/file2

I am not interested in details which individual files were copied, just
the main directory. Is it somehow possible for rsync to only report the
first level directory?

ie, to have output like this when copying:

   /mnt/foo/aaa/
   /mnt/foo/bbb/


thanks,



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: print only first level directory name when copying files

[Fourhundred Thecat via rsync]

> On 2023-08-04 06:13, Perry Hutchison wrote:

Perry Hutchison via rsync  wrote:

On second thought, that grep will match any directory name having 3
*or more* levels.  This:

   rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/ | egrep 
'^/[^/]*/[^/]*/[^/]*/$'

should match only those with exactly 3 levels.


I think that should work!

thank you

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: print only first level directory name when copying files

[Perry Hutchison via rsync]

Perry Hutchison via rsync  wrote:

> Fourhundred Thecat via rsync <400the...@lists.samba.org> wrote:
>
> > I am copying /mnt/foo to /mnt/bar/
> >
> >rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/
> >
> > /mnt/foo contains deep directory structure, ie:
> >
> >/mnt/foo/aaa/
> >/mnt/foo/aaa/somestuff/
> >/mnt/foo/aaa/somestuff/file1
> >
> >/mnt/foo/bbb/
> >/mnt/foo/bbb/someotherstuff/
> >/mnt/foo/bbb/someotherstuff/file2
> >
> > I am not interested in details which individual files were copied, just
> > the main directory. Is it somehow possible for rsync to only report the
> > first level directory?
> >
> > ie, to have output like this when copying:
> >
> >/mnt/foo/aaa/
> >/mnt/foo/bbb/
>
> Do you need this reported in real time (e.g. to monitor progress),
> or just as a logfile?  If the latter, filtering the output with grep
> might work -- something like this (untested):
>
>   rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/ | egrep '^/.*/.*/.*/$'

On second thought, that grep will match any directory name having 3
*or more* levels.  This:

  rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/ | egrep '^/[^/]*/[^/]*/[^/]*/$'

should match only those with exactly 3 levels.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: print only first level directory name when copying files

[Perry Hutchison via rsync]

Fourhundred Thecat via rsync <400the...@lists.samba.org> wrote:

> I am copying /mnt/foo to /mnt/bar/
>
>rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/
>
> /mnt/foo contains deep directory structure, ie:
>
>/mnt/foo/aaa/
>/mnt/foo/aaa/somestuff/
>/mnt/foo/aaa/somestuff/file1
>
>/mnt/foo/bbb/
>/mnt/foo/bbb/someotherstuff/
>/mnt/foo/bbb/someotherstuff/file2
>
> I am not interested in details which individual files were copied, just
> the main directory. Is it somehow possible for rsync to only report the
> first level directory?
>
> ie, to have output like this when copying:
>
>/mnt/foo/aaa/
>/mnt/foo/bbb/

Do you need this reported in real time (e.g. to monitor progress),
or just as a logfile?  If the latter, filtering the output with grep
might work -- something like this (untested):

  rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/ | egrep '^/.*/.*/.*/$'

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

print only first level directory name when copying files

[Fourhundred Thecat via rsync]

Hello,

I am copying /mnt/foo to /mnt/bar/

  rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/

/mnt/foo contains deep directory structure, ie:

  /mnt/foo/aaa/
  /mnt/foo/aaa/somestuff/
  /mnt/foo/aaa/somestuff/file1

  /mnt/foo/bbb/
  /mnt/foo/bbb/someotherstuff/
  /mnt/foo/bbb/someotherstuff/file2

I am not interested in details which individual files were copied, just
the main directory. Is it somehow possible for rsync to only report the
first level directory?

ie, to have output like this when copying:

  /mnt/foo/aaa/
  /mnt/foo/bbb/


thanks,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Consider POSIX_FADV_NOREUSE?

[Ronan Pigott via rsync]

Hey rsync,

Seems like there was an effort a while back to make use of POSIX_FADV_DONTNEED
on linux [1]. Though the linux patches landed, apparently the rsync ones never
did.

Over a decade later we live in a different world. Linux is using the new-ish
MGLRU on many distros, and (~6.3+) even has support for POSIX_FADV_NOREUSE [2],
which was dismissed in prior discussions for lack of support.

Maybe it's time to re-evaluate the utility of posix_fadvise in rsync?

[1] https://lkml.org/lkml/2010/11/21/59
[2] https://lore.kernel.org/all/20221230215252.2628425-2-yuz...@google.com/T/#u

Cheers,
Ronan

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to tune rsync to speed up?

[Kevin Korb via rsync]

NFS is slowing things down even more than your bandwidth measurements as 
it is also forcing --whole-file.


On 8/2/23 05:03, Perry Hutchison via rsync wrote:

Sebastian G??decke via rsync  wrote:


We're facing some flapping traffic when rsyncing atm 70T from
one server to an DELL Isilon.
Both systems are connected with 10G Fiber (not Channel).
So we started with one simple "rsync -a /src /dest" to the DELL
by using NFS3.
...
I always thought (and had observed it so far with rsync) that it
makes full use of the network card ...


If you are using NFS, it is NFS -- not rsync -- which is managing the
network traffic.  In such a setup rsync will perform about as well as
cp(1).



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to tune rsync to speed up?

[Perry Hutchison via rsync]

Sebastian G??decke via rsync  wrote:

> We're facing some flapping traffic when rsyncing atm 70T from
> one server to an DELL Isilon.
> Both systems are connected with 10G Fiber (not Channel).
> So we started with one simple "rsync -a /src /dest" to the DELL
> by using NFS3.
> ...
> I always thought (and had observed it so far with rsync) that it
> makes full use of the network card ...

If you are using NFS, it is NFS -- not rsync -- which is managing the
network traffic.  In such a setup rsync will perform about as well as
cp(1).

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

How to tune rsync to speed up?

[Sebastian Gödecke via rsync]

Hi there,
we're facing some flapping traffic when rsyncing atm 70T from one server to
an DELL Isilon.
Both systems are connected with 10G Fiber (not Channel).
So we started with one simple "rsync -a /src /dest" to the DELL by using
NFS3.
So it runs with around 3Gbit for some seconds, than it were 30Kbit for also
some seconds and again "some" Gig were transferred and than again it
decreases to 50kbit.
This was all the time for about 24 hours.
Than (this is actually not my doing, it's one of our researchers) i send
him a little pythonscript to use more rsyncprocesses. He modified it and
now it's around 5Gbps and 10Gbps.
So this helps a lot, but now I'm wondering what's the reason, because I
always thought (and had observed it so far with rsync) that it makes full
use of the network card (if the data is provided accordingly)
Some thoughts about that?

Here some specs:
DELL Server :
7515 with 1T RAM, sfp+ nic,
Software: opensuse leap 15.2 , rsync 3.1.3

DELL Isilon:
sfp+ connection, OS: 9.5.0.3

Thanks Sebastian

-- 
Mit freundlichen Grüßen
Sebastian Gödecke
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Interaction between hosts allow and hosts deny

[Roger Price via rsync]

I am trying to master the hosts allow module parameters described by man 
rsyncd.conf .  Quoting from the man page:


 > hosts allow
 > This parameter allows you to specify a list of comma- and/or
 > whitespace-separated patterns that are matched against a connecting client's
 > hostname and IP address.  If none of the patterns match, then the connection
 > is rejected.

I understand from this that if I specify only IPv4 addresses, no IPv6 traffic is 
accepted.  Is this correct?


Quoting again from the man page:

 > Note IPv6 link-local addresses can have a scope in the address specification:
 >  fe80::1%link1
 >  fe80::%link1/64
 >  fe80::%link1/:::::

Should I understand that the %link1 in these examples could be replaced by say 
%wlan0 or %eth0 , and that I could write fe80::%wlan0/64 ?


Roger

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7809] I/O errors other than IOERR_GENERAL should not suppress deletion

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7809

--- Comment #6 from Jeffrey Simon  ---
3. The script does not work from launchd running as root.

I should have given the failure mode, which is the following:

rsync: opendir "/Volumes/Backup1/." failed: Operation not permitted (1)
rsync error: some files could not be transferred (code 23) at
/AppleInternal/Library/BuildRoots/c2cb9645-dafc-11ed-aa26-6ec1e3b3f7b3/Library/Caches/com.apple.xbs/Sources/rsync/rsync/main.c(996)
[sender=2.6.9]

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7809] I/O errors other than IOERR_GENERAL should not suppress deletion

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7809

--- Comment #5 from Jeffrey Simon  ---
"Excludes are relative to the source dir". Are you saying that the excludes
should be --exclude=.DocumentRevisions-V100 --exclude=.TemporaryItems
--exclude=.Trashes?

That is a rhetorical question, because I no longer get an issue with with
excludes. After I added sudo, those errors no longer occur.

However, I am still having a problem, as follows:

1. The plain rsync command (with sudo and excludes) works from the command
line.

2. The same rsync command (with sudo and excludes) works as part of a script
from the command line.

3. The script does not work from launchd running as root.

Because rsync appears to be working properly, my issues are most likely with
the environmental differences between command line and launchd. So except for
the oddities that these issues only showed up on macOS Ventura, but did not
happen on macOS Monterey, I believe the issue can be considered somewhat
resolved. I say "somewhat" because it is not clear to me why sudo should be
necessary from the command line, but it is.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7809] I/O errors other than IOERR_GENERAL should not suppress deletion

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7809

--- Comment #4 from Kevin Korb  ---
Your excludes aren't working because excludes are relative to the source dir
not /.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html