Re: BUG? rsync ends without message by delete files

[Roland via rsync]

here is another report of this behaviour.
https://unix.stackexchange.com/questions/754923/rsync-just-stops

nothing appropriate in bugzilla, besides
https://bugzilla.samba.org/show_bug.cgi?id=13317

do you use zfs or is there full-space/quota condition while running?

if you can't resolve, please file a bugreport in bugzilla

roland


Am 06.02.24 um 22:18 schrieb Franke via rsync:

Hi Kevin,

Am 06.02.24 um 20:55 schrieb Kevin Korb:

The other likely cause is your $SOURCE being something that contains a *
or other wildcard.  If there is a wildcard in the source parameter then
the shell expands that wildcard giving rsync a list of sources.  The
--delete only operates within that list meaning that anything not
currently in that list is immune to deletion.

no, $SOURCE $TARGET are only placeholder here. In real there are normal
path's like: /foo/bar /new/foo/bar

The Rsync does some time correctly, like:

(sync) /foo/bar/bla_1
delete /foo/bar/mee/blub
...
(sync) /foo/bar/bla_2
...
delete /foo/bar/mee/borg
...

and then, it stops totally quiet.

I try'd to recall the command again an then ist wokrs again correctly
after the last output delete (/foo/bar/mee/borg...) and stopps again
after some correctly doing after deleting some files.

Same behavior by calling with --delete-before rsync breaks here too
after delete some files.

The very BIG Problem of this: NO ERRORS will be output
In the following the Backups are not complete synced, and the HDD-Store
goes to zero. :-(



Gruss
Franke



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BUG? rsync ends without message by delete files

[Roland via rsync]

and then, it stops totally quiet.


you mean it simply exits without any message?

what's the return code ( echo $? )

roland

Am 06.02.24 um 22:18 schrieb Franke via rsync:

Hi Kevin,

Am 06.02.24 um 20:55 schrieb Kevin Korb:

The other likely cause is your $SOURCE being something that contains a *
or other wildcard.  If there is a wildcard in the source parameter then
the shell expands that wildcard giving rsync a list of sources.  The
--delete only operates within that list meaning that anything not
currently in that list is immune to deletion.

no, $SOURCE $TARGET are only placeholder here. In real there are normal
path's like: /foo/bar /new/foo/bar

The Rsync does some time correctly, like:

(sync) /foo/bar/bla_1
delete /foo/bar/mee/blub
...
(sync) /foo/bar/bla_2
...
delete /foo/bar/mee/borg
...

and then, it stops totally quiet.

I try'd to recall the command again an then ist wokrs again correctly
after the last output delete (/foo/bar/mee/borg...) and stopps again
after some correctly doing after deleting some files.

Same behavior by calling with --delete-before rsync breaks here too
after delete some files.

The very BIG Problem of this: NO ERRORS will be output
In the following the Backups are not complete synced, and the HDD-Store
goes to zero. :-(



Gruss
Franke



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Archiving to vfat

[Roland via rsync]

it's most likely because of vfat timestamp limitation

try

--modify-window
  When comparing two timestamps, rsync treats the
timestamps as being equal if they differ by no more than  the modify-window
  value.   This  is  normally  0 (for an exact match), but
you may find it useful to set this to a larger value in some situa-
  tions.  In particular, when transferring to or from an MS
Windows FAT filesystem (which represents  times  with  a 2-second
  resolution), --modify-window=1 is useful (allowing times
to differ by up to 1 second).

roland

Am 21.01.24 um 22:15 schrieb Ian Z via rsync:

I am trying to use rsync between two local directories on Linux.

The source directory is on a normal ext4 partition, under my home
directory. The destination is an SD card that I insert into the card
reader on the computer, formatted with a vfat filesystem.

The command line is like

   rsync -avC --delete /home/itz/foo/ /media/itz/DEAD-BEEF/foo/

This does not work as I expected: all files are always transferred,
even when I run this command 2 times with nothing in between. I
expected only new or changed files to be transferred. Is this

- a user error
- a rsync bug
- a limitation of the vfat filesystem?

In any case, what can I do, using rsync or possibly something else, to
do what I want? I note that the absolute timestamps on the files are
unimportant; I can change them to whatever. All that matters is that
new and changed files are copied every time I run this command, and
only those files.



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: process --files-from filelist as given

[Roland via rsync]

I want to copy a list of files in specific order

Why ?


because i want to serialize files on disk so they are stored on disk in
the order being accessed regularly

i built that list for --files-from   via output from fatrace tool.

i now did use tar to transfer the files

i will add an RFE to bugzilla

thanks

roland

Am 31.12.23 um 20:56 schrieb francis.montag...@inria.fr:

Hi.

On Sun, 31 Dec 2023 20:28:21 +0100 Roland via rsync wrote:


apparently, rsync sorts the list of files  provided to "--files-from".
how can i avoid sorting of that list ?

According to the man, this is not possible. See: SORTED TRANSFER ORDER
that suggest also the --delay‐updates option.


I want to copy a list of files in specific order

Why ?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

process --files-from filelist as given

[Roland via rsync]

hello,

apparently,  rsync sorts the list of files  provided to "--files-from".

how can i avoid sorting of that list ?

I want to copy a list of files in specific order

regards

Roland


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: syncing huge files/devices: bmapfs

[Roland via rsync]

nice,

do you have performance comparison vs. plain fail on the same hardware
setup ?

thank you

roland

Am 10.02.23 um 07:56 schrieb Heiko Schlittermann via rsync:

fuse driver that maps the huge file (or device) as a directory of files
representing blocks of the backing file (or device).

Using Rsync now on these smaller files I circumvented the above
mentioned issues (be they real or just a result of my stupidity).

Maybe you find bmapfs useful for other scenarios too. I'd be happy if


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files

[Roland via rsync]

why not avoid using "--remove-source-files" and delete files
manually/via extra step afterwards ?

Am 17.10.22 um 08:59 schrieb Sridhar Sarnobat via rsync:

90% of my data losses are caused by rsync'ing from dir A to dir A
(accidental incorrect copy and paste, or where dir B is a symlink to
dir A). The source gets erased unintentionally.

It may seem dumb but when you get really long commands and really long
file paths it will happen a 1-2 times a yearwhen I'm doing it
dozens/hundreds of times every week.

*Is there an existing mechanism to protect against this?* rsync should
logically never erase all copies of data.

I know I can backup files (which I do for my smaller drives), or
create wrapper scripts but I'm not asking about those.I don't think
adopting --dry-run is practical being such a heavy user (and moreover,
it's unlikely to warn what will happen).

I'm not a C developer so I probably can't do this myself but if
anyone's encouraging me to then I'd give it a shot.


Sridhar Sarnobat
San Jose, CA 95128 | USA
Phone: +1 (650) 260-3851 | ss...@cornell.edu
SMS me via email: 6073395...@txt.att.net
Google Chat: sarnobat.hotm...@gmail.com
http://www.facebook.com/sridhar.sarnobat
http://www.linkedin.com/in/sarnobat 
http://github.com/sarnobat


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

BUG?: rsync timeout non-effective

[Roland via rsync]

apparently, it must be a bug


if i do

/root/rsync/rsync-3.2.5/rsync -avi --timeout=10 --exclude='/proc'
--exclude='/dev/'  root@172.20.37.189:/ /zfspool/backup/172.20.37.189/

it hangs forever , as soon as rsync tries to traverse the
suspended/hanging /iscsipool on the remote machine


if i do
/root/rsync/rsync-3.2.5/rsync -avi --timeout=10 --exclude='/proc'
--exclude='/dev/'  root@172.20.37.189:/iscsipool
/zfspool/backup/172.20.37.189/

i.e. if i try to rsync the "hung/stalled" directory directly,  i'm
getting proper timeout behaviour:


receiving incremental file list
[Receiver] io timeout after 10 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(197)
[Receiver=3.2.5]



so, apparently there seem to be circumstances where the timeout
detection is not effective


if i run rsync locally on the machine containing the hung/stalled dir,
it behaves like this:


# rsync -av --timeout=1 --dry-run /iscsipool /tmp
sending incremental file list
[Receiver] io timeout after 1 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(197)
[Receiver=3.2.3]



regards
roland





root@debian:~# zpool status
  pool: iscsipool
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool
clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
config:

    NAME  STATE READ WRITE CKSUM
    iscsipool UNAVAIL  0 0 0 insufficient
replicas
      wwn-0x  UNAVAIL  4 1 0

errors: No known data errors


Am 01.09.22 um 17:45 schrieb Roland:

there is connection timeout and I/O timeout

i'm using the I/O timeout.

for "--timeout" manpage is "if no data is transferredthen rsync
will exit"

no data is being transferred for hours, but it doesn't exit.

so, what's the problem here ?



       --timeout=SECONDS
  This option allows you to set a maximum I/O timeout in
seconds.  If no data is transferred for the specified time then rsync
will exit.  The default is 0, which means no timeout.

   --contimeout=SECONDS
  This option allows you to set the amount of time that
rsync will wait for its connection to an rsync daemon to succeed.  If
the timeout is reached, rsync exits with an error.




Am 01.09.22 um 17:32 schrieb Kevin Korb via rsync:

You are using rsync over ssh.  The connection timeout (and port)
options don't matter if rsync isn't doing the networking.

On 9/1/22 08:49, Roland via rsync wrote:

hello,

i do some backup via rsync/ssh and pull data from remote machine to
local machine.

whenever on remote machine there is some network attached subdir
getting
"stuck" (i.e. read/write is blocking, eg. by suspended zfs pool because
of error), rsync backup also hangs forever.

i wonder what's the purpose of "--timeout" then.

local machine:  3.1.2
remote machine: 3.2.3

command
rsync -avi --stats --delete --delete-excluded --inplace --numeric-ids
--out-format=%-15i %-15b %-15l %-20M %n%L --timeout=900
--rsync-path=export PATH=/usr/local/bin:$PATH;rsync
--exclude-from=/backup/rsync.exclude.all
--exclude-from=/zfspool/backup/myhost/rsync.exclude root@myhost:/
/zfspool/backup/myhost/backup



what can i do to make this more robust ?

rsync should not hang, as it's getting started via script and the hang
blocks backup of other hosts.


roland







--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync timeout non-effective

[Roland via rsync]

there is connection timeout and I/O timeout

i'm using the I/O timeout.

for "--timeout" manpage is "if no data is transferredthen rsync will
exit"

no data is being transferred for hours, but it doesn't exit.

so, what's the problem here ?



       --timeout=SECONDS
  This option allows you to set a maximum I/O timeout in
seconds.  If no data is transferred for the specified time then rsync
will exit.  The default is 0, which means no timeout.

   --contimeout=SECONDS
  This option allows you to set the amount of time that
rsync will wait for its connection to an rsync daemon to succeed.  If
the timeout is reached, rsync exits with an error.




Am 01.09.22 um 17:32 schrieb Kevin Korb via rsync:

You are using rsync over ssh.  The connection timeout (and port)
options don't matter if rsync isn't doing the networking.

On 9/1/22 08:49, Roland via rsync wrote:

hello,

i do some backup via rsync/ssh and pull data from remote machine to
local machine.

whenever on remote machine there is some network attached subdir getting
"stuck" (i.e. read/write is blocking, eg. by suspended zfs pool because
of error), rsync backup also hangs forever.

i wonder what's the purpose of "--timeout" then.

local machine:  3.1.2
remote machine: 3.2.3

command
rsync -avi --stats --delete --delete-excluded --inplace --numeric-ids
--out-format=%-15i %-15b %-15l %-20M %n%L --timeout=900
--rsync-path=export PATH=/usr/local/bin:$PATH;rsync
--exclude-from=/backup/rsync.exclude.all
--exclude-from=/zfspool/backup/myhost/rsync.exclude root@myhost:/
/zfspool/backup/myhost/backup



what can i do to make this more robust ?

rsync should not hang, as it's getting started via script and the hang
blocks backup of other hosts.


roland







--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync timeout non-effective

[Roland via rsync]

hello,

i do some backup via rsync/ssh and pull data from remote machine to
local machine.

whenever on remote machine there is some network attached subdir getting
"stuck" (i.e. read/write is blocking, eg. by suspended zfs pool because
of error), rsync backup also hangs forever.

i wonder what's the purpose of "--timeout" then.

local machine:  3.1.2
remote machine: 3.2.3

command
rsync -avi --stats --delete --delete-excluded --inplace --numeric-ids
--out-format=%-15i %-15b %-15l %-20M %n%L --timeout=900
--rsync-path=export PATH=/usr/local/bin:$PATH;rsync
--exclude-from=/backup/rsync.exclude.all
--exclude-from=/zfspool/backup/myhost/rsync.exclude root@myhost:/
/zfspool/backup/myhost/backup



what can i do to make this more robust ?

rsync should not hang, as it's getting started via script and the hang
blocks backup of other hosts.


roland



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

bug#282 : timeout option does not work as expected when remote system is hung

[Roland via rsync]

hello,

I have opened https://github.com/WayneD/rsync/issues/282

looks similar to https://github.com/WayneD/rsync/issues/164 and
https://github.com/WayneD/rsync/issues/8

roland


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [PATCH] Reduce memory usage

[Roland via rsync]

hello,

it's fantastic to see that such optimizations still being found.

out of curiosity - what is the status of this?  will it get merged ?

roland

Am 28.09.21 um 12:05 schrieb Jindřich Makovička via rsync:

In 2004, an allocation optimization has been added to the file
list handling code, that preallocates 32k of file_struct pointers
in a file_list. This optimization predates the incremental
recursion feature, for which it is not appropriate anymore. When
copying a tree containing a large number of small directories,
using the incremental recursion, rsync allocates many short
file_lists. Suddenly, the unused file_struct pointers can easily
take 90-95% of the memory allocated by rsync.

This can be easily reproduced by using

valgrind --tool=massif ./rsync -anx /usr /tmp/

and checking the memory profile of the first (sender) process.

This patch changes the flist functions to start only with 32 entries
for the partial file lists, instead of 32 * 1024.

It also modifies the condition for the debug notification that the
allocated memory moved, so it does not depend on the initial allocation
size. Instead, it now simply checks if the file list has been already
allocated.
---
  flist.c | 9 +++--
  rsync.h | 5 +++--
  2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/flist.c b/flist.c
index 3442d868..530d336e 100644
--- a/flist.c
+++ b/flist.c
@@ -305,7 +305,7 @@ static void flist_expand(struct file_list *flist, int extra)

new_ptr = realloc_array(flist->files, struct file_struct *, 
flist->malloced);

-   if (DEBUG_GTE(FLIST, 1) && flist->malloced != FLIST_START) {
+   if (DEBUG_GTE(FLIST, 1) && flist->files) {
rprintf(FCLIENT, "[%s] expand file_list pointer array to %s bytes, 
did%s move\n",
who_am_i(),
big_num(sizeof flist->files[0] * flist->malloced),
@@ -2186,8 +2186,10 @@ struct file_list *send_file_list(int f, int argc, char 
*argv[])
  #endif

flist = cur_flist = flist_new(0, "send_file_list");
+   flist_expand(flist, FLIST_START_LARGE);
if (inc_recurse) {
dir_flist = flist_new(FLIST_TEMP, "send_file_list");
+   flist_expand(dir_flist, FLIST_START_LARGE);
flags |= FLAG_DIVERT_DIRS;
} else
dir_flist = cur_flist;
@@ -2541,10 +2543,13 @@ struct file_list *recv_file_list(int f, int dir_ndx)
  #endif

flist = flist_new(0, "recv_file_list");
+   flist_expand(flist, FLIST_START_LARGE);

if (inc_recurse) {
-   if (flist->ndx_start == 1)
+   if (flist->ndx_start == 1) {
dir_flist = flist_new(FLIST_TEMP, "recv_file_list");
+   flist_expand(dir_flist, FLIST_START_LARGE);
+   }
dstart = dir_flist->used;
} else {
dir_flist = flist;
diff --git a/rsync.h b/rsync.h
index 88319732..f8fcbffb 100644
--- a/rsync.h
+++ b/rsync.h
@@ -918,8 +918,9 @@ extern int xattrs_ndx;
   * Start the flist array at FLIST_START entries and grow it
   * by doubling until FLIST_LINEAR then grow by FLIST_LINEAR
   */
-#define FLIST_START(32 * 1024)
-#define FLIST_LINEAR   (FLIST_START * 512)
+#define FLIST_START(32)
+#define FLIST_START_LARGE  (32 * 1024)
+#define FLIST_LINEAR   (FLIST_START_LARGE * 512)

  /*
   * Extent size for allocation pools: A minimum size of 128KB


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Copying TBs -> error -> work around

[Roland via rsync]

>with rsync hanging - after breakout on /home for writing I then get:
>"Read-only file system"

if your filesystem switches to read-only, you have a serious problem
with your system/storage, not with rsync.

rsync (or the workload) is simply triggering the problem.

regards
roland


Am 10.09.20 um 07:30 schrieb Philip Rhoades via rsync:

People,

When I did:

  rsync -av /home/ /mntb5/  # about 4TB

I got errors like:

  'rsync [sender] expand file_list pointer array to xxx bytes, "did
move"'

with rsync hanging - after breakout on /home for writing I then get:

  "Read-only file system"

So after unmounting and remounting /home I did:

  cd /home
  find /home/ -type d | sort > ./home_dirs_sorted.txt

delete first line "/home/" of ./home_dirs_sorted.txt then:

  while read dir ; do echo $dir ; rsync -lptgod "$dir" /mntb5/"$dir" ;
done < ./home_dirs_sorted.txt

and:

  while read dir ; do echo $dir ; rsync -lptgoD "$dir"/\.[a-zA-Z0-9]*
/mntb5/"$dir"/ ; done < ./home_dirs_sorted.txt

and finally with no problems:

  rsync -av --exclude-from=/usr/local/bin/nfb_caches.txt /home/ /mntb5/

If there was a more sensible / efficient way of getting this done I
would like to know about it!

Thanks,

Phil.



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--exclude vs --filter

[Roland via rsync]

Hello,

i have a backup script where i first used "--exclude-from" only. users
of my script can set exclude statements in backup client specific
property files.

later on, i needed to extend the script because of special requirement ,
so i needed to introduce usage of "--filter='merge $filterfile'"

i want to harmonize things and simplify the script, as i needed to
duplicate statements because of that and it looks ugly and is difficult
to maintain.

can "--filter" statements completely replace "--exclude" statements ,
i.e. is "--filter" simply to be considered as an "--exclude on steriods"
, or are there chances that i need "--exclude" though ?  (for excample
because rule evaluation may differ ?)

regards
roland



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [PATCHv2] SSE2/SSSE3 optimized version of get_checksum1() for x86-64

[Roland via rsync]

would it perhaps make sense to have a "--disable-sse2/3" commandline
switch in rsync, too - at least for some timeframe until this is
considered "rock solid" ?

i dislike having automatic cpu feature switching code in a tool which
needs to be reliable for me, this new optimization may have issues - and
without such switch it can't be easily workarounded without replacing
the binary/package.

regards
roland


Am 19.05.20 um 16:28 schrieb Jorrit Jongma via rsync:

I've read up some more on the subject, and it seems the proper way to
do this with GCC is g++ and target attributes. I've refactored the
patch that way, and it indeed uses SSSE3 automatically on supporting
CPUs, regardless of the build host, so this should be ideal both for
home builders and distros.

Getting the code to build right in c++ mode (checksum_sse2.cpp only)
was a bit of an adventure, requiring modifications to mkproto.awk,
configure.ac, and Makefile.in.

I've done my best to prevent any c++ compilation happening in case the
optimizations are not enabled (g++ isn't used, build target isn't
x86-64, or --disable-sse2 was passed to configure) and prevent a
dependency on libstdc++. I've tested that part extensively but it
would be great if the maintainer (and others) could give this part a
close second look.

This patch _replaces_ my previous submit, it does not build on top of it.

GitHub:

https://github.com/Chainfire/rsync/commit/ef3c13390601752ef652b37c15610e12e2309fea

https://github.com/Chainfire/rsync/commit/ef3c13390601752ef652b37c15610e12e2309fea.patch

Raw:

 From ef3c13390601752ef652b37c15610e12e2309fea Mon Sep 17 00:00:00 2001
From: Jorrit Jongma 
Date: Tue, 19 May 2020 14:52:40 +0200
Subject: [PATCH] SSE2/SSSE3 optimized version of get_checksum1() for x86-64

Requires compilation using GCC C++ front end, build scripts have been
modified accordingly. C++ is only used when the optimization is enabled
(g++ as compiler, x86-64 build target, --disable-sse2 not passed to
configure).
---
  Makefile.in   |  12 +-
  checksum.c|   2 +
  checksum_sse2.cpp | 289 ++
  configure.ac  |  22 
  mkproto.awk   |   2 +
  5 files changed, 325 insertions(+), 2 deletions(-)
  create mode 100644 checksum_sse2.cpp

diff --git a/Makefile.in b/Makefile.in
index 59649562..0953e601 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -12,6 +12,9 @@ LIBS=@LIBS@
  CC=@CC@
  CFLAGS=@CFLAGS@
  CPPFLAGS=@CPPFLAGS@
+CXX=@CXX@
+CXXFLAGS=@CXXFLAGS@
+CXXUSED=@CXXUSED@
  EXEEXT=@EXEEXT@
  LDFLAGS=@LDFLAGS@
  LIBOBJDIR=lib/
@@ -27,7 +30,11 @@ SHELL=/bin/sh
  VERSION=@RSYNC_VERSION@

  .SUFFIXES:
-.SUFFIXES: .c .o
+ifeq ($(CXXUSED),yes)
+.SUFFIXES: .c .cpp .o
+else
+.SUFFIXES: .c .o
+endif

  GENFILES=configure.sh aclocal.m4 config.h.in proto.h proto.h-tstamp
rsync.1 rsync-ssl.1 rsyncd.conf.5
  HEADERS=byteorder.h config.h errcode.h proto.h rsync.h ifuncs.h
itypes.h inums.h \
@@ -41,10 +48,11 @@ OBJS1=flist.o rsync.o generator.o receiver.o
cleanup.o sender.o exclude.o \
  OBJS2=options.o io.o compat.o hlink.o token.o uidlist.o socket.o hashtable.o \
   fileio.o batch.o clientname.o chmod.o acls.o xattrs.o
  OBJS3=progress.o pipe.o
+CXXOBJ=@CXXOBJ@
  DAEMON_OBJ = params.o loadparm.o clientserver.o access.o connection.o
authenticate.o
  popt_OBJS=popt/findme.o  popt/popt.o  popt/poptconfig.o \
   popt/popthelp.o popt/poptparse.o
-OBJS=$(OBJS1) $(OBJS2) $(OBJS3) $(DAEMON_OBJ) $(LIBOBJ) @BUILD_ZLIB@
@BUILD_POPT@
+OBJS=$(OBJS1) $(OBJS2) $(OBJS3) $(CXXOBJ) $(DAEMON_OBJ) $(LIBOBJ)
@BUILD_ZLIB@ @BUILD_POPT@

  TLS_OBJ = tls.o syscall.o t_stub.o lib/compat.o lib/snprintf.o
lib/permstring.o lib/sysxattrs.o @BUILD_POPT@

diff --git a/checksum.c b/checksum.c
index cd234038..7c42742a 100644
--- a/checksum.c
+++ b/checksum.c
@@ -99,6 +99,7 @@ int canonical_checksum(int csum_type)
   return csum_type >= CSUM_MD4 ? 1 : 0;
  }

+#ifndef ENABLE_SSE2 // see checksum_sse2.cpp
  /*
a simple 32 bit checksum that can be updated from either end
(inspired by Mark Adler's Adler-32 checksum)
@@ -119,6 +120,7 @@ uint32 get_checksum1(char *buf1, int32 len)
   }
   return (s1 & 0x) + (s2 << 16);
  }
+#endif

  void get_checksum2(char *buf, int32 len, char *sum)
  {
diff --git a/checksum_sse2.cpp b/checksum_sse2.cpp
new file mode 100644
index ..9a2ae86b
--- /dev/null
+++ b/checksum_sse2.cpp
@@ -0,0 +1,289 @@
+/*
+ * SSE2/SSSE3-optimized routines to support checksumming of bytes.
+ *
+ * Copyright (C) 1996 Andrew Tridgell
+ * Copyright (C) 1996 Paul Mackerras
+ * Copyright (C) 2004-2020 Wayne Davison
+ * Copyright (C) 2020 Jorrit Jongma
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the 

Re: osx permission issue

[Roland via rsync]

what makes you believe that ssh is accessing the files ?

ok, rsync on remote mac is being spawned by sshd, so i gave full disk
access to the sshd binary , too (as it may inherit restrictions to sub
processes) and restarted ssh service ,

but unfortunately it makes no difference.

did you mean that or something different ?

roland

Am 28.03.20 um 16:20 schrieb Ben Bass:


Hi Roland.

You might have to give ssh full disk access on the remote Mac, as that
is what is accessing the files according to the Mac.

On Sat, Mar 28, 2020 at 9:12 AM Roland via rsync
mailto:rsync@lists.samba.org>> wrote:

hello,

does somebody know how to circumvent that "extra file access
restriction
feature" introduced in osx some time ago ?

i already tried adding rsync binary to programms with "full disk
access"
privilege ( system-prefs -> security & privacy -> privacy -> full disk
access)  , since running as root is not sufficient - but it does
not work.

i want to make sure that every file on osx is getting backup

i run rsync on linux to remotely backup osx system (via ssh).

regards
roland


rsync:

readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/0/com.apple.routined")
failed: Operation not permitted (1)
rsync:

readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/0/com.apple.Safari/SafariFamily")
failed: Operation not permitted (1)
rsync:

readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/C/com.apple.WebKit.WebContent.Sandbox")
failed: Operation not permitted (1)
rsync:

readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/C/com.apple.QuickLook.thumbnailcache")
failed: Operation not permitted (1)
rsync:

readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/C/com.apple.WebKit.Networking.Sandbox")
failed: Operation not permitted (1)
rsync: opendir
"/private/var/folders/zz/zyxvpxvq6csfxvn_n0sm6d/0" failed:
Operation not permitted (1)
rsync: opendir
"/private/var/folders/zz/zyxvpxvq6csfxvn_n0sm6d/C" failed:
Operation not permitted (1)

# ls -l@ /private/var/folders/zz/zyxvpxvq6csfxvn_n0sm6d/
total 0
drwxr-xr-x@  2 _locationd  _locationd   64  3 Nov  2017 0
 com.apple.rootless     -1
drwx--@ 17 _locationd  _locationd  544 19 Jan 23:03 C
 com.apple.rootless     -1
drwx--@  2 _locationd  _locationd   64  3 Nov  2017 T
 com.apple.rootless      7


--
Please use reply-all for most replies to avoid omitting the
mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html



--
-Ben
(917) 536-0998
b...@benbass.com
<mailto:b...@benbass.com>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

osx permission issue

[Roland via rsync]

hello,

does somebody know how to circumvent that "extra file access restriction
feature" introduced in osx some time ago ?

i already tried adding rsync binary to programms with "full disk access"
privilege ( system-prefs -> security & privacy -> privacy -> full disk
access)  , since running as root is not sufficient - but it does not work.

i want to make sure that every file on osx is getting backup

i run rsync on linux to remotely backup osx system (via ssh).

regards
roland


rsync:
readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/0/com.apple.routined")
failed: Operation not permitted (1)
rsync:
readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/0/com.apple.Safari/SafariFamily")
failed: Operation not permitted (1)
rsync:
readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/C/com.apple.WebKit.WebContent.Sandbox")
failed: Operation not permitted (1)
rsync:
readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/C/com.apple.QuickLook.thumbnailcache")
failed: Operation not permitted (1)
rsync:
readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/C/com.apple.WebKit.Networking.Sandbox")
failed: Operation not permitted (1)
rsync: opendir
"/private/var/folders/zz/zyxvpxvq6csfxvn_n0sm6d/0" failed:
Operation not permitted (1)
rsync: opendir
"/private/var/folders/zz/zyxvpxvq6csfxvn_n0sm6d/C" failed:
Operation not permitted (1)

# ls -l@ /private/var/folders/zz/zyxvpxvq6csfxvn_n0sm6d/
total 0
drwxr-xr-x@  2 _locationd  _locationd   64  3 Nov  2017 0
    com.apple.rootless     -1
drwx--@ 17 _locationd  _locationd  544 19 Jan 23:03 C
    com.apple.rootless     -1
drwx--@  2 _locationd  _locationd   64  3 Nov  2017 T
    com.apple.rootless      7


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync --inplace dead slow

[Roland --- via rsync]

hello,

i observe a weird performance problem while trying to rsync .vhd image
files from xenserver to linux backupserver.

i have broken the problem down to a single 500gb file which gets
pathological slow on subsequent runs.

rsync begins with decent speed (250MB/s), i can see that with --progress
and also verify with iotop.


After a while, rsync performance drops down to <3MB/s where i see cpu at
100% on the sender side.


I'm using rsync in daemon mode, receiver is 3.1.2 and sender is 3.1.1.
No encryption or compression in place.


What is causing this slowness?

I know about
https://www.anchor.com.au/blog/2013/08/out-tridging-tridge/, but
according to that article, the problem should have been adressed.

Is there still possibility for "pathological slow performance" which
lies in algorithm/design? (as it does not look like a server/system
performance for me)

regards
Roland


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync performance weirdness

[Roland --- via rsync]

hi,

i observe some weirdness in rsync file transfer i cannot explain.

i'm transferring data from 2 freenas storages to a linux vm with zfsonlinux.

"fnask" is older freenas with rsync 3.1.1 running in daemon mode,
freenas-bnkw is more recent with rsync 3.1.2, also daemon-mode.

on the linux vm where the data is being pulled from the 2 freenas boxes,
i look onto io bandwidth being used with "iotop -a".

transfer from freenas-bnkw storage is much much slower then from fnask
storage.

why can i see two rsync processes transferring data from "fnask" but i
see only one rsync process with "freenas-bnkw" where the parent process
is transferring nothing ?

is there a difference in how this is being handled between 3.1.1 and
3.1.2 on the remote side ?

is this the reason for the slowness ?

regards
roland


Total DISK READ : 278.72 M/s | Total DISK WRITE :   0.00 B/s
Actual DISK READ:  97.09 M/s | Actual DISK WRITE:   0.00 B/s
  TID  PRIO  USER    DISK READ>  DISK WRITE  SWAPIN  IO COMMAND
32497 be/4 root    261.41 G  0.00 B  0.00 % 12.87 % rsync -av
--inplace fnask::backupall_S2V2_xen-sr-rpkn01-nfs/ .
32498 be/4 root    200.56 G 12.88 G  0.00 % 21.41 % rsync -av
--inplace fnask::backupall_S2V2_xen-sr-rpkn01-nfs/ .
24005 be/4 root 81.52 G 12.34 G  0.00 % 46.80 % rsync -av
--inplace freenas-bnkw::backupall_zfspool_sas_seagate/sr_rpkn02/ .
 3599 be/0 root 10.93 G  0.00 B  0.00 %  0.00 % [z_rd_int]
 3604 be/0 root 10.84 G  0.00 B  0.00 %  0.00 % [z_rd_int]
 3605 be/0 root 10.81 G  0.00 B  0.00 %  0.00 % [z_rd_int]


  ├─screen,14590
  │   └─bash,14591
  │   └─rsync,32497 -av --inplace
fnask::backupall_S2V2_xen-sr-rpkn01-nfs/ .
  │   └─rsync,32498 -av --inplace
fnask::backupall_S2V2_xen-sr-rpkn01-nfs/ .
  ├─screen,32261
  │   └─bash,32262
  │   └─rsync,24004 -av --inplace
freenas-bnkw::backupall_zfspool_sas_seagate/sr_rpkn02/ .
  │   └─rsync,24005 -av --inplace
freenas-bnkw::backupall_zfspool_sas_seagate/sr_rpkn02/ .



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Do grsync/rsync work well on Mac?

[Roland --- via rsync]

have a look at this one:
http://osxdaily.com/2016/04/17/speed-up-time-machine-by-removing-low-process-priority-throttling/

i also recommend taking a look at borgbackup and
https://github.com/borgbase/vorta

roland


Am 21.04.19 um 05:32 schrieb Peng Yu via rsync:

Hi,

I experience a slow backup problem with Time Machine on Mac. The info
below does not help to reduce the backup time to a reasonable period.

https://support.apple.com/en-us/HT204412

So I'd like to use grsync/rsync to incrementally back my disk. I know
that Time Machine somehow can create hard links to manage incremental
links. Since grsync/rsync is primarily targeted at Linux, is it a good
idea to use it for Mac?

Also, by using grsync/rsync, I guess that I will loose the ability to
recover the bootable ability of the disk (as in TimeMachine). Is there
a walkaround for this? Thanks.



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html