Re: rsync as a de-duplication-only tool, using --link-dest

[Kevin Korb via rsync]

I don't believe that what you are asking for can be done with rsync.  At 
first thought you can't mix --ignore-existing with --ignore-non-existing 
as that would ignore everything.  Something would have to at least exist 
and not be ignored for rsync to link to it.


Anyway, for a laugh, I asked chatgpt to make something to do this. 
After I got my laugh I cleaned up some of the silly stuff it did and 
came up with this:


#!/bin/bash

# Define the directories to compare
dir1="$1"
dir2="$2"

# Recursively list all files in both directories
files1=$(find "$dir1" -type f)

# Loop through files in first directory
for file1 in $files1; do
# Get relative path of file1
rel_path="${file1#$dir1}"
file2="$dir2$rel_path"

# Check if file exists in the second directory
if [ -f "$file2" ]; then
# Get metadata of both files
metadata1=$(stat -c "%Y%s" "$file1")
metadata2=$(stat -c "%Y%s" "$file2")

# Compare metadata
if [ "$metadata1" -eq "$metadata2" ]; then
# Delete file1 and create a hard link to file2
# rm "$file1"
# ln "$file2" "$file1"
echo "Hard linked: $file2 to $File1"
# else
# echo "Different: $file1"
fi
fi
done

Note that I only tested it a little bit which is why anything actually 
destructive is commented.


On 5/1/24 19:34, B via rsync wrote:
Recently I was thinking about --link-dest= and if it was possible to use 
rsync to de-duplicate two nearly-identical directory structures.


Normally I would use a tool like hardlink, jdupes, or rdfind, but in 
this case the files are huge and numerous, so hashing them would take 
forever. I did a test run and these tools mostly choked to death after a 
few hours.


These directories were made using rsync in the first place, so I know 
the files are duplicate and I would be willing to use rsync's 
quick-check (path/filename, mtime, size) to assume uniqueness of the files.


My objective is to hard-link files with the same relative path/filename, 
mtime, and size. Nothing more. Files which are different should not be 
touched. Files which exist in the destination but not the source should 
not be deleted. Files which exist in the source but not the destination 
should not be transferred.


The problem is that I don't want to create any new files in the 
destination. That's the sticking point.


I thought maybe I could do something wacky like 'rsync -a 
--ignore-existing --ignore-non-existing --link-dest="../new/" old/ new', 
but that doesn't work. The existing files get ignored and nothing is 
linked.


Is there a way to do this with rsync?





--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync replacing symlinks without warning (resend)

[Kevin Korb via rsync]

I'm not really blaming the user.  If it were up to me, -v would include -i.

On 2/9/24 05:36, Andreas Gruenbacher wrote:

On Sun, Feb 4, 2024 at 7:20 PM Kevin Korb via rsync
 wrote:

rsync's -v is fairly useless.  Learn to use -i instead or in addition to.


Well, note that I didn't say anything about the lib/ directory in that
command; it's just that rsync decided to remove the symlink component
from the path lib/modules/.

Wouldn't it be more useful to make it obvious in the --verbose and
--dry-run output what rsync has actually decided to do instead of
blaming the user, which is what you're doing? Especially if the
behavior is all but self-explanatory?

Thanks,
Andreas


On 2/4/24 12:58, Andreas Gruenbacher via rsync wrote:

Hello,

when trying to rsync files between hosts, I ran into a surprising case
in which rsync replaces a symlink with a directory, with no indication
of any kind.

In the following reproducer, rsync is called as follows:

rsync --verbose --recursive --relative --delete a/./lib/modules b/

Directory b contains a 'lib' symlink pointing to 'usr/lib', and rsync
removes that and replaces it with a directory.

In my real-world use case, this caused '/lib' -> '/usr/lib' symlinks
to be replaced with '/lib' directories, which left the receiving test
machines in a fairly sad state.

I have since figured out that I can get rsync to behave as expected by
adding the --keep-dirlinks option, but ...

it's very unfortunate that when rsync does that kind of thing, it
leaves no indication in the 'rsync --dry-run' and 'rsync -v' output.
Could that please be fixed?

Thanks,
Andreas


#! /bin/sh

tmp=$(mktemp -dt ${0##*/}.XX)
trap 'cd /; rm -rf $tmp' EXIT
cd "$tmp"

umask 022

mkdir -p a/lib/modules
echo foo > a/lib/modules/foo

mkdir -p b/usr/lib/modules
ln -s usr/lib b/lib

show() {
  find "$@" | xargs stat -c "%F %N" | sort -k2
}

echo "from:"
show a

echo
echo "to:"
show b

echo
echo "rsync:"
rsync \
  --verbose \
  --recursive \
  --relative \
  --delete \
  a/./lib/modules \
  b/

echo
echo "to:"
show b

# SCRIPT OUTPUT with rsync 3.2.7:
# ==
=
# from:
# directory 'a'
# directory 'a/lib'
# directory 'a/lib/modules'
# regular file 'a/lib/modules/foo'
#
# to:
# directory 'b'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# symbolic link 'b/lib' -> 'usr/lib'
#
# rsync:
# sending incremental file list
# lib/
# lib/modules/
# lib/modules/foo
#
# sent 160 bytes  received 47 bytes  414.00 bytes/sec
# total size is 4  speedup is 0.02
#
# to:
# directory 'b'
# directory 'b/lib'
# directory 'b/lib/modules'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# regular file 'b/lib/modules/foo'




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BUG? rsync ends without message by delete files

[Kevin Korb via rsync]

now it sounds like you have too many hard links for rsync to handle.

On 2/7/24 08:05, Franke via rsync wrote:

Am 06.02.24 um 23:20 schrieb Roland:

and then, it stops totally quiet.

you mean it simply exits without any message?



Yes rsync ends totally quit.



what's the return code ( echo $? )


After some more tests, the ERR-Code are:

  "22 - Error allocating core memory buffers"
(with --delete-before)


But, now there are only ~100.000 Files to delete, in some Tranches.


IMHO:

rsync read the Fileslist without problems, begins with deleting (and on
thi can decrease the list too) and fail than with a Momory-Error? :-o

Source- and Target-system have 64GB RAM, could be enough...

This seem for me as a Bug.


And sorry, but i cant make Issues on Github.



Gruss
Franke



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BUG? rsync ends without message by delete files

[Kevin Korb via rsync]

Normally, when rsync isn't deleting things the problem is that there is
some kind of error (possibly scrolled off screen unnoticed) but it
sounds like you are getting no output at all which would eliminate that
possibility.

The other likely cause is your $SOURCE being something that contains a *
or other wildcard.  If there is a wildcard in the source parameter then
the shell expands that wildcard giving rsync a list of sources.  The
--delete only operates within that list meaning that anything not
currently in that list is immune to deletion.

If that doesn't answer your question I would suggest adding -ii (doubled
--itemize-changes) as it will show you the files it is analyzing and
what it plans to do about them.

On Tue, 6 Feb 2024, Franke via rsync wrote:


Date: Tue, 6 Feb 2024 19:53:32 +0100
From: Franke via rsync 
To: just subscribed for rsync-qa from bugzilla via rsync

Subject: BUG? rsync ends without message by delete files

Hi!

Is this a bug, or did I miss a setting?

When synchronizing and deleting files, rsync (version 3.2.7  protocol
version 31 AND before) simply stops whitout comments.
No error, no message, nothing in the syslog.

Rsync works normally, but after delete some on, it ends.

I try'd --max-delete=1 to, but no effekt.

Command are:

rsync -vaxHSPAX --delete $SOURCE $TARGET

The dirs containing in sync containing approximately 40 million files to
delete.

And I can't delete it by hand as we need to drive some recoveries after
a server crash and relocation.


Any Ideas?



Gruss
Franke




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync replacing symlinks without warning (resend)

[Kevin Korb via rsync]

rsync's -v is fairly useless.  Learn to use -i instead or in addition to.

On 2/4/24 12:58, Andreas Gruenbacher via rsync wrote:

Hello,

when trying to rsync files between hosts, I ran into a surprising case
in which rsync replaces a symlink with a directory, with no indication
of any kind.

In the following reproducer, rsync is called as follows:

   rsync --verbose --recursive --relative --delete a/./lib/modules b/

Directory b contains a 'lib' symlink pointing to 'usr/lib', and rsync
removes that and replaces it with a directory.

In my real-world use case, this caused '/lib' -> '/usr/lib' symlinks
to be replaced with '/lib' directories, which left the receiving test
machines in a fairly sad state.

I have since figured out that I can get rsync to behave as expected by
adding the --keep-dirlinks option, but ...

it's very unfortunate that when rsync does that kind of thing, it
leaves no indication in the 'rsync --dry-run' and 'rsync -v' output.
Could that please be fixed?

Thanks,
Andreas


#! /bin/sh

tmp=$(mktemp -dt ${0##*/}.XX)
trap 'cd /; rm -rf $tmp' EXIT
cd "$tmp"

umask 022

mkdir -p a/lib/modules
echo foo > a/lib/modules/foo

mkdir -p b/usr/lib/modules
ln -s usr/lib b/lib

show() {
 find "$@" | xargs stat -c "%F %N" | sort -k2
}

echo "from:"
show a

echo
echo "to:"
show b

echo
echo "rsync:"
rsync \
 --verbose \
 --recursive \
 --relative \
 --delete \
 a/./lib/modules \
 b/

echo
echo "to:"
show b

# SCRIPT OUTPUT with rsync 3.2.7:
# ==
=
# from:
# directory 'a'
# directory 'a/lib'
# directory 'a/lib/modules'
# regular file 'a/lib/modules/foo'
#
# to:
# directory 'b'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# symbolic link 'b/lib' -> 'usr/lib'
#
# rsync:
# sending incremental file list
# lib/
# lib/modules/
# lib/modules/foo
#
# sent 160 bytes  received 47 bytes  414.00 bytes/sec
# total size is 4  speedup is 0.02
#
# to:
# directory 'b'
# directory 'b/lib'
# directory 'b/lib/modules'
# directory 'b/usr'
# directory 'b/usr/lib'
# directory 'b/usr/lib/modules'
# regular file 'b/lib/modules/foo'




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Archiving to vfat

[Kevin Korb via rsync]

Also, instead of -a use -rt.  Those are the only parts of -a that FAT 
even pretends to support.


On 1/21/24 16:42, Roland via rsync wrote:

it's most likely because of vfat timestamp limitation

try

--modify-window
   When comparing two timestamps, rsync treats the
timestamps as being equal if they differ by no more than  the modify-window
   value.   This  is  normally  0 (for an exact match), but
you may find it useful to set this to a larger value in some situa-
   tions.  In particular, when transferring to or from an MS
Windows FAT filesystem (which represents  times  with  a 2-second
   resolution), --modify-window=1 is useful (allowing times
to differ by up to 1 second).

roland

Am 21.01.24 um 22:15 schrieb Ian Z via rsync:

I am trying to use rsync between two local directories on Linux.

The source directory is on a normal ext4 partition, under my home
directory. The destination is an SD card that I insert into the card
reader on the computer, formatted with a vfat filesystem.

The command line is like

   rsync -avC --delete /home/itz/foo/ /media/itz/DEAD-BEEF/foo/

This does not work as I expected: all files are always transferred,
even when I run this command 2 times with nothing in between. I
expected only new or changed files to be transferred. Is this

- a user error
- a rsync bug
- a limitation of the vfat filesystem?

In any case, what can I do, using rsync or possibly something else, to
do what I want? I note that the absolute timestamps on the files are
unimportant; I can change them to whatever. All that matters is that
new and changed files are copied every time I run this command, and
only those files.





--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync over ssh fails with --files-from

[Kevin Korb via rsync]

The errors column is 0.  The drop column is 18.  The second bit number 
is the number of packets which should grow.  At least that is how I read 
it.  Column makes it more readable in a terminal but not so much in an 
email.


On 12/21/23 14:18, Alex wrote:
Can someone help me determine if these errors are normal or if this 
could somehow be the cause? I've removed the last three columns for 
readability - they were all zeros.


# column -t /proc/net/dev
Inter-|      Receive    |        Transmit
face         |bytes     packets  errs      drop  fifo  frame  compressed 
  multicast|bytes  packets    errs    drop  fifo
lo:          133093161  146045   0         0     0     0      0 
   0                133093161  146045  0     0     0
ens18:       166999724  256655   0         16    0     0      0 
   0                167638513  218267  0     0     0


The "errs" for ens18 are steadily increasing, but the "drop" column is 
steady. It's also curious the errors on loopback would also be increasing?


Other ideas on how to troubleshoot this would be greatly appreciated.

Thanks,
Alex

On Wed, Dec 20, 2023 at 9:27 PM Alex <mailto:mysqlstud...@gmail.com>> wrote:


Hi,

On Wed, Dec 20, 2023 at 11:03 AM Kevin Korb via rsync
mailto:rsync@lists.samba.org>> wrote:

What is the error?  I assume you know that with that syntax the
filelist.txt is local rather than remote.


Yes, I do know it refers to the list of local files.

There is no error - it just hangs indefinitely until some timeout
period. This is what it looks like on the remote side:

$ ps ax|grep rsync
  324075 ?        Ss     0:00 rsync --server -logDtpRe.LsfxCIvu
. /var/tmp/one/
  324094 ?        S      0:00 rsync --server -logDtpRe.LsfxCIvu
. /var/tmp/one/

On the local side, with a few - added to rsync, I see this:

recv_file_list done
get_local_name count=9 /var/tmp/one/
generator starting pid=324075
delta-transmission enabled
recv_generator(config1.cf <http://config1.cf/>,0)
config1.cf <http://config1.cf/> is uptodate
send_files(0, config1.cf <http://config1.cf/>)

then it just stalls until it eventually times out. However, if I
remove any one of the nine files from the filelist, it completes
normally.

It actually also exhibits the same problem without a filelist at all
- as long as I'm transferring multiple files, it fails. I suppose if
I were syncing a directory with less than nine files, it might
succeed, but local directory to any remote directory on this one
server fails.

This also isn't a disk space or inode problem or corrupt filesystem
problem - the same command works from a different host to this one
problematic host without a problem.

I've also confirmed the open file limit is large enough on both sides:

# ulimit -n
5

I've now spent hours and hours trying to isolate and troubleshoot
this problem. It really feels like it's somehow related to the
number of files being transferred, not the size of the files. If I
break up a directory into multiple attempts, I can eventually
transfer all the files (maybe 40 in total), but trying to send all
40 files at once and none get transferred.

It almost certainly has something to do with building the initial
list of files. If that list is too large, the transfer fails, even
if the majority of the files in the source are already in the
destination.

It also happens when I'm pushing the files to the destination or
pulling them from the remote to local.

Thanks,
Alex






On 12/20/23 09:50, Alex via rsync wrote:
 > Hi, I've been using rsync on fedora over ssh to sync
directories for
 > decades, but suddenly having a problem with transferring
multiple files
 > at a time to one specific host using --files-from. I can't
think of what
 > might have changed to have caused this. Using rsync to
transfer a single
 > file to this problematic host works successfully. It appears
to be
 > related to the number of files in the --files-from filelist.
More than
 > nine and it stalls; less than nine and it finishes
successfully. I'm
 > using the same version of rsync on both sides.
 >
 > (Server) Protocol versions: remote=31, negotiated=31
 > (Client) Protocol versions: remote=31, negotiated=31
 >
 > When running the following command, it appears to collect the
list of
 > files to be transferred, successfully makes the connection
with the
 > remote server, but then just stalls. Using strace on the
remote side
 > shows rsync appears to be waiting for data.
 >
  

Re: rsync over ssh fails with --files-from

[Kevin Korb via rsync]

What is the error?  I assume you know that with that syntax the 
filelist.txt is local rather than remote.


On 12/20/23 09:50, Alex via rsync wrote:
Hi, I've been using rsync on fedora over ssh to sync directories for 
decades, but suddenly having a problem with transferring multiple files 
at a time to one specific host using --files-from. I can't think of what 
might have changed to have caused this. Using rsync to transfer a single 
file to this problematic host works successfully. It appears to be 
related to the number of files in the --files-from filelist. More than 
nine and it stalls; less than nine and it finishes successfully. I'm 
using the same version of rsync on both sides.


(Server) Protocol versions: remote=31, negotiated=31
(Client) Protocol versions: remote=31, negotiated=31

When running the following command, it appears to collect the list of 
files to be transferred, successfully makes the connection with the 
remote server, but then just stalls. Using strace on the remote side 
shows rsync appears to be waiting for data.


rsync -a --files-from=/etc/mail/filelist.txt -e 'ssh -i 
/root/keys/sync-key-v4' /etc/mail/tmp/ polaris:/var/tmp/one/


sync-key-v4 is a passwordless ed25519 key, but I've tried a handful of 
other ed25519 keys.


Could it be related to packet size or some kind of network disparity? 
It's not related to the size of the files, as I've tried large and small 
and it doesn't matter - if the number of files exceeds 9, it fails.









--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?

[Kevin Korb via rsync]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Unfortunately, exit 23 litterally just means something else went wrong
and might have scrolled off of the screen if you have rsync listing
files (--verbose or --itemize_changes).  Essentially, it is anything
that doesn't have its own exit code.  I just ignore it.  The most common
reaosn is file vanished.

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

On Thu, 14 Dec 2023, rsync--- via rsync wrote:


Date: Thu, 14 Dec 2023 19:20:15 +0100
From: rsync--- via rsync 
To: rsync@lists.samba.org
Subject: rsync exit code 23 (partial transfer due to errors): List of possible
 reasons and how to ignore some?

I am trying to find a solution for the open source Linux software

  "Back In Time" (https://github.com/bit-team/backintime)

where we evaluate the rsync exit code when taking a backup via rsync
and inform the user that an error has occured.

Questions:

1. Is there full list of possible reasons available that make rsync
  exit with the return value 23?

  'rsync error: some files/attrs were not transferred (see previous errors) 
(code 23) at main.c(1338) [sender=3.2.7]'

  I want to decide for each reason if we treat it as an error or warning.

2. I want to ignore some reasons for exit code 23 by only logging it, mainly:

  symlink has no referent: "/home/user/Documents/dead-link"

  Is there a way to
  - either prevent that this error leads to exit code 23 (if no other reason 
occurs)
  - or to prevent this specific check at all (but eg. copy the symlink "as is")?

THX a lot!



PS 1: The typcially used rsync command line looks like this:

rsync --recursive --times --devices --specials --hard-links --human-readable -s 
--copy-links --acls --perms --executability --group --owner --
info=progress2 --no-inc-recursive -l --delete --delete-excluded -v -i 
--out-format=BACKINTIME: %i %n%L --link-dest=../../20231214-143359-584/backup --
chmod=Du+wx --exclude=/home/username/temp/testBAK_profil1 
--exclude=/home/username/.local/share/backintime 
--exclude=.local/share/backintime/mnt --
include=/home/username/Documents/ --include=/home/username/ --include=/home/ 
--include=/home/username/temp/deleted_folder/ --
include=/home/username/temp/ --include=/home/username/.mozilla/ --exclude=.gvfs 
--exclude=.cache/* --exclude=.thumbnails* --
exclude=.local/share/[Tt]rash* --exclude=*.backup* --exclude=*~ 
--exclude=.dropbox* --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* --
exclude=/run/* --exclude=/etc/mtab --exclude=/var/cache/apt/archives/*.deb 
--exclude=lost+found/* --exclude=/tmp/* --exclude=/var/tmp/* --
exclude=/var/backups/* --exclude=.Private --exclude=lock 
--exclude=root_only_file.txt --include=/home/username/Documents/** --
include=/home/username/temp/deleted_folder/** 
--include=/home/username/.mozilla/** --exclude=* /
/home/username/temp/testBAK_profil1/backintime/computer1/username/1/new_snapshot/backup




PS 2: You can find more details or even answer in our Github issue too:

https://github.com/bit-team/backintime/issues/1587




-BEGIN PGP SIGNATURE-

iF0EARECAB0WIQSHERqysePm7S8yuR9UoLWOVtABBwUCZXtTzQAKCRBUoLWOVtAB
B2CnAJ9YGQ/gXTPP2Ntg3arHHDC11cRnfgCeJVpDOnGhTH0OreZfj8pIbnsL3SI=
=Eqei
-END PGP SIGNATURE-


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[Kevin Korb via rsync]

So I decided to do a quick test using the Linux kernel source tree since 
it has lots of files.  I duplicated a tree, used 'find . -type f -exec 
chmod 444 {} +' to make read only files for rsync to want to chmod, then 
used cp -al to make several duplicate trees using hard linked files.  An 
rm -rf on one such tree took .97 seconds while an rsync deletion took 
1.25 seconds.  Clearly I need a bigger test to play with as that margin 
could easily change just by different outputs if either had a -v.  But 
also, I did not experience the problem you are describing.  My surviving 
hard links in the duplicate trees were still 444.


BTW, it seems that rsync uses unlink() while rm uses unlinkat().  No 
idea if/why there is a performance diff there but I didn't look into it 
as it is now time for me to go to work.


On 9/22/23 04:14, rs...@altfeld-im.de wrote:

On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote:


I have heard in the past that rsyncing an empty dir over a tree to
delete the tree is faster than an rm -rf but I can't say I have ever
benchmarked it to get any actual numbers.


This **may** indeed be a myth (for a long time now) re-cited again and again and
- could no longer be valid today
- could apply only when deleting explicitly named files but not deleting the 
complete folder
   (as we need to do in "Back in Time")

At least I could not find a holistic benchmark with many files and different 
scenarios
(file systems, rsync'ing locally vs. over network, snapshot sizes, number of 
files, file sizes, rsync and rm versions...)

Q: Does `rsync` provide a test case that I could use as basis to prepare such a 
holistic benchmark?


But now that I am hearing
that rsync actually adds a bunch of pointless chmods to the process.  Is
it still faster given this problem?  If so maybe we should be trying to
investigate why rm is so slow.


Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not.




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?

[Kevin Korb via rsync]

I had intended to come back to this but because I didn't really think I 
had anything to add to the discussion I put it at a low enough priority 
that I forgot about it.  But I saw your bug report and was surprised to 
see that I was already unhelpful on this topic but because that original 
poster didn't have access to do an rm while you are trying to get a 
performance boost.


I have heard in the past that rsyncing an empty dir over a tree to 
delete the tree is faster than an rm -rf but I can't say I have ever 
benchmarked it to get any actual numbers.  But now that I am hearing 
that rsync actually adds a bunch of pointless chmods to the process.  Is 
it still faster given this problem?  If so maybe we should be trying to 
investigate why rm is so slow.  Otherwise, I would agree that an 
optimization to not do a bunch of pointless chmods is a good idea.  I 
would suspect that they are only there to prevent permission denied 
errors perhaps on some obscure filesystem (I tested on vfat since it is 
so non-compatible but read only files unlink fine there).


On 9/18/23 19:42, rsync--- via rsync wrote:

Context
---

I am one of the active developers of the open source application "Back in Time"
which uses "rsync" as backend and I want to fix an open issue:

 "Back in Time"-Bug:
 https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507

"Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking
unchanged files in the backups/snapshots and tries to keep the permissions
by using "--perms --group --owner" by default.

Older snapshots are then deleted according to a schedule.



Issue
-

When deleting a complete snapshot folder with rsync using an empty source folder
(which is a best practice for faster deletion than "rm -f") the permissions of 
hardlinked
files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted 
files
with existing hardlinks:

 mkdir empty
 rsync -a --delete -s empty/ snapshot1/

This distorts the backup history.

There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the 
issue is still unfixed:

 https://bugzilla.samba.org/show_bug.cgi?id=12806



Questions
-

1. Is there a reliable workaround ("--super" is proposed in the issue but may 
probably not always work)?
2. If a rsync developer is reading this:  Is there any chance to fix this?



Steps to reproduce
--

I have attached a bash script "setup.sh" as txt file which makes the issue 
fully reproducible (on Ubuntu 22.04).




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Feature Concept: enable iCloud Drive with rsync

[Kevin Korb via rsync]

Is this being accessed via a fuse mount?  If so it seems like that is 
where this kind of feature should be implemented (like a mount option to 
decide how to handle such files).  Rsync shouldn't need special features 
to deal with every kind of file storage.


On 9/12/23 05:22, Brian "bex" Exelbierd via rsync wrote:

Hi,

I have also posted this on GitHub but it isn’t clear that was the right 
place: https://github.com/WayneD/rsync/issues/522


iCloud Drive will evict files that are unused or when additional space 
is needed on the local drive. The evicted files are replace by 
"bookmark" files that allow MacOS to continue to report the files in the 
file system as though they were actually present. The files are 
downloaded again either on request or when needed.


rsync, like all similar tools I can find, doesn't have any way of 
handling these evicted files. I have been thinking about this and I 
think I know how to make it work.


The short explanation is that iCloud Drive preserves access to the 
required metadata. Here is an abbreviated output from `mdls` of an 
evicted file:


```
% mdls .1998-tax-return.pdf.icloud
kMDItemContentCreationDate = 2018-08-28 18:32:24 +
kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +
kMDItemContentModificationDate = 2018-08-28 18:32:24 +
kMDItemDateAdded = 2021-11-05 15:42:15 +
kMDItemDisplayName = "1998-tax-return.pdf"
kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +
kMDItemFSCreationDate = 2018-08-28 18:32:24 +
kMDItemFSSize = 1929932
kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +
kMDItemLogicalSize = 1929932
kMDItemPhysicalSize = 1929932
```

This metadata, I think, is enough to pass the rsync quick check as 
described in the man page. Therefore, I suspect what needs to happen, 
from a code perspective, is that rsync needs to be modified to do the 
following when it finds an evicted file. All evicted files are named 
consistently, `..iCloud` so they are easy to spot.


1. Perform the system call equivalent of the mdls above to obtain the 
appropriate dates and sizes.
2. If the file cannot be skipped because it either has changed or a 
checksum is required, perform the system call equivalent of `brctl 
download ` to get the file downloaded.

3. Rsync the file
4. If the file was downloaded by rsync, perform the system call 
equivalent of `brctl evict ` to remove the file to leave the 
system in the same state.


This simplistic algorithm would leave some open issues/caveats:
- It is likely that the rsync can only be run one-way from iCloud Drive 
to non-iCloud Drive data storage. While it is possible it could be run 
two-way research is needed on whether you’d have to download the old 
file before you replace it.
- Timeouts may happen. iCloud Drive still gets stuck sometimes and there 
will be non-zero time during downloads that don’t get stuck.
- There is no guarantee there is ever enough room on disk to hold a 
specific file from iCloud Drive. This is likely resolved by MacOS 
directly, however, this may take excessive time.
- If you thought running with checksums was slow before, strap in, 
because those downloads are going to add up.


All that said, I think this would accomplish a high percentage of what 
users are looking for.


Is this kind of a patch welcome? Is the plan sound? I haven't written C 
since college (in the 90s!) and would be open to advice/mentoring.


Thank you.

regards,

bex



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Why try to update (some) permissions which are the same?

[Kevin Korb via rsync]

You have --itemize-changes but either it didn't or you filtered it out.

--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

On Sun, 3 Sep 2023, Perry Hutchison via rsync wrote:


Date: Sun, 03 Sep 2023 00:35:29 -0700
From: Perry Hutchison via rsync 
To: rsync@lists.samba.org
Subject: Why try to update (some) permissions which are the same?

On the source system:

$ rsync --version
rsync  version 2.6.8  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.

Capabilities: 64-bit files, socketpairs, hard links, ACLs, symlinks, batchfiles,
 inplace, IPv6, file flags, 32-bit system inums, 64-bit internal 
inums
...

$ ll -d fcst-200[89] fcst-201[01]
dr-xr-xr-x  2 perryh  perryh  7168 Nov 27  2009 fcst-2008
dr-xr-xr-x  2 perryh  perryh  9216 Jul 21  2010 fcst-2009
drwxr-xr-x  2 perryh  perryh  9216 Jul  7  2011 fcst-2010
drwxr-xr-x  2 perryh  perryh  4608 Jul  7  2011 fcst-2011

$ rsync -uHSxz4rlptOgv --itemize-changes -e rsh fcst-200[89] fcst-201[01] 
fbsd81:/home/perryh
building file list ... done
rsync: failed to modify permissions on "/home/perryh/fcst-2008": Operation not 
permitted (1)
rsync: failed to modify permissions on "/home/perryh/fcst-2009": Operation not 
permitted (1)

sent 15794 bytes  received 20 bytes  31628.00 bytes/sec
total size is 278224  speedup is 17.59
rsync error: some files could not be transferred (code 23) at main.c(892) 
[sender=2.6.8]


On the destination system:

$ ll -d fcst-200[89] fcst-201[01]
dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 7168 Nov 27  2009 fcst-2008
dr-xr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul 21  2010 fcst-2009
drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 9216 Jul  7  2011 fcst-2010
drwxr-xr-x  2 perryh  perryh  uchg,uunlnk 4608 Jul  7  2011 fcst-2011

$ rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
   64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
   socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
   append, ACLs, xattrs, no iconv, symtimes
...


The question is:  why does rsync attempt (and fail) to change the
permissions of two destination directories, and not the other two,
when the permissions of all four destination directories already
match the corresponding source directories?  (The change attempt
fails because of the destinations' uchg,uunlnk flags, but as far
as I can see the change should never have been attempted in the
first place.)




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: print only first level directory name when copying files

[Kevin Korb via rsync]

find /mnt/foo/* -maxdepth 0 -print -exec rsync -an `realpath {}` 
/mnt/bar/ \;

 (realpath eliminates the trailing slashes)

On 8/3/23 22:27, Fourhundred Thecat via rsync wrote:

Hello,

I am copying /mnt/foo to /mnt/bar/

   rsync --info=name1,del2 -rl /mnt/foo /mnt/bar/

/mnt/foo contains deep directory structure, ie:

   /mnt/foo/aaa/
   /mnt/foo/aaa/somestuff/
   /mnt/foo/aaa/somestuff/file1

   /mnt/foo/bbb/
   /mnt/foo/bbb/someotherstuff/
   /mnt/foo/bbb/someotherstuff/file2

I am not interested in details which individual files were copied, just
the main directory. Is it somehow possible for rsync to only report the
first level directory?

ie, to have output like this when copying:

   /mnt/foo/aaa/
   /mnt/foo/bbb/


thanks,



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to tune rsync to speed up?

[Kevin Korb via rsync]

NFS is slowing things down even more than your bandwidth measurements as 
it is also forcing --whole-file.


On 8/2/23 05:03, Perry Hutchison via rsync wrote:

Sebastian G??decke via rsync  wrote:


We're facing some flapping traffic when rsyncing atm 70T from
one server to an DELL Isilon.
Both systems are connected with 10G Fiber (not Channel).
So we started with one simple "rsync -a /src /dest" to the DELL
by using NFS3.
...
I always thought (and had observed it so far with rsync) that it
makes full use of the network card ...


If you are using NFS, it is NFS -- not rsync -- which is managing the
network traffic.  In such a setup rsync will perform about as well as
cp(1).



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync sends again already existing files

[Kevin Korb via rsync]

You should also read about --inplace.  Without it --no-whole-file you 
are telling it to do all the extra data diffing only to write out an 
entire new file anyway (just using data from source and target to create 
it).


On 6/30/23 21:29, Selva Nair via rsync wrote:



So this disable a lot of interest in Rsync :-( Isn't there a way to
disable
"--whole-file"?


"--no-whole-file"  should do it though for local copies, forcing delta
transfer is not going to speed up anything in most cases.

Selva



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync sends again already existing files

[Kevin Korb via rsync]

On 6/29/23 16:31, Stephane Ascoet via rsync wrote:

Kevin Korb  le 29/06/2023 04:52:

--itemize-changes will cause rsync to tell you what it thinks is


Hi, thank you so much! Today I used a little different way of doing it, 
and another computer, and the behaviour is the same. It seems that the 
reason is a different timestamp. So the whole file is replaced just for 
this? I've always read that Rsync was using an advanced checksum 
mecanism instead of timestamps-only?


Your rsync isn't networking.  That means --whole-file is forced making 
rsync work pretty much like cp.  If rsync was networking it would 
determine what is different about the files then use the existing 
version + changes to make a new file (this is just extra effort when 
rsync isn't networking).



different.  Also, -z is counter-productive when rsync isn't networking.


Thanks again. It seems that "--progress" is redondant with "-P"(but it's 
because I use a list of rsync commands skeletons, and some doesn't have 
"-P" and I want to be sure to always have the progress indication).
But, since I will now always use "-i", do "-v" and "--progress" add 
something to "-i"?




-i, -v, and --progress all only affect the output.  Once -i is in play 
-v adds a header and footer and --progress of course  adds the per-file 
progress bar.  -P is just --partial and --progress.  I am not a fan of 
-P as those 2 options are very different and I don't really see why they 
should go together.


--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync sends again already existing files

[Kevin Korb via rsync]

--itemize-changes will cause rsync to tell you what it thinks is 
different.  Also, -z is counter-productive when rsync isn't networking.


On 6/28/23 22:28, Stephane Ascoet via rsync wrote:


Hi, /media/clec1enextsanstampon/gigamopourcdas4/ contains an old copy of 
files of /media/mo. I'm updating with "cd /media/clec1enextsanstampon/ 
&& rsync --backup 
--backup-dir=/media/clec1enextsanstampon/elementssupprimesdepourcdas4/ 
--del -hPrStvXz --progress /media/mo/ 
/media/clec1enextsanstampon/gigamopourcdas4/"
But unmodified and already in place files in 
/media/clec1enextsanstampon/gigamopourcdas4/ are re-sended from 
/media/mo instead of being skipped, why?




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [External] Re: ctrl -c while executing --progress --size-only --partial results in unhidden but incomplete file

[Kevin Korb via rsync]

Rsync has no memory of what other instances of rsync have done in the 
past.  An existing file with a different name is of no interest to it 
unless --delete in which case it would be deleted.  Maybe what you 
really want is --partial-dir?


On 6/7/23 16:17, Lacey, Nathan wrote:

I'm suggesting a partial file that isn't hidden is worse.
Because any other app looking at the file would think the transfer is 
complete and try to use it.


It doesn't need to be a orphaned file,  when called next time with 
--partial rsync could re-opened the incomplete/hidden file, than it 
could  continue putting more bytes to the file until it was really 
completed.


When running in --partial mode, the filename wouldn't have to be random, 
instead it could be based on the source ip address, so that rsync knows 
what file to continue processing on.


*From:* rsync  on behalf of Kevin Korb 
via rsync 

*Sent:* Wednesday, June 7, 2023 3:26 PM
*To:* rsync@lists.samba.org 
*Subject:* [External] Re: ctrl -c while executing --progress --size-only 
--partial results in unhidden but incomplete file
[You don't often get email from rsync@lists.samba.org. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification 
<https://aka.ms/LearnAboutSenderIdentification> ]


This message is from an EXTERNAL SENDER - be CAUTIOUS of links and 
attachments. THINK BEFORE YOU CLICK.




That is what --partial does.  It keeps the partial file.  If it kept the
temporary file with the random file name that would just be a useless
orphaned file.

On 6/7/23 15:02, Lacey, Nathan via rsync wrote:
This is a clone from https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D=0 

<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D=0>
<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D=0 

<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D=0>>


run rsync with --progress --size-only --partial (version 3.1.2) also
tested/built current master ( 3.3.0pre1-5-g6f3c5ecc )
everything works fine unless I interrupt a transfer ( ctrl c )

I would expect to see the incomplete, temporary, hidden file present on
the remote machine.
So that when I call rsync again, it could just pick up where it left off
and continue transferring data.

But instead there is no hidden file, and the partial/incomplete file
exists, as the actual filename.

This is bad, because any code that is expecting existing non-hidden
files to be complete will have issues.

I apologize if I've missed the command line options I'm looking for
Thanks for maintaining a critical piece of software!





--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
     Kevin Korb  Phone:    (407) 252-6853
     Systems Administrator   Internet:
     FutureQuest, Inc.   ke...@futurequest.net  (work)
     Orlando, Florida    k...@sanitarium.net (personal)
     Web page: 
https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fsanitarium.net%2F=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=j42u1PbBcts8QAs7rqJTVuLXLW2mKy6WaifX756cEFI%3D=0 
<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fsanitarium.net%2F=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C6

Re: ctrl -c while executing --progress --size-only --partial results in unhidden but incomplete file

[Kevin Korb via rsync]

That is what --partial does.  It keeps the partial file.  If it kept the 
temporary file with the random file name that would just be a useless 
orphaned file.


On 6/7/23 15:02, Lacey, Nathan via rsync wrote:
This is a clone from https://github.com/WayneD/rsync/issues/484 



run rsync with --progress --size-only --partial (version 3.1.2) also 
tested/built current master ( 3.3.0pre1-5-g6f3c5ecc )

everything works fine unless I interrupt a transfer ( ctrl c )

I would expect to see the incomplete, temporary, hidden file present on 
the remote machine.
So that when I call rsync again, it could just pick up where it left off 
and continue transferring data.


But instead there is no hidden file, and the partial/incomplete file 
exists, as the actual filename.


This is bad, because any code that is expecting existing non-hidden 
files to be complete will have issues.


I apologize if I've missed the command line options I'm looking for
Thanks for maintaining a critical piece of software!





--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Per file Log output to understand hard-link decision

[Kevin Korb via rsync]

The only way I know of to determine this behavior is to use the 
--link-dest option in rsync OR use the much older cp -al then rsync over 
top of it method.  With --link-dest a change in the file's metadata 
causes rsync to duplicate the file in order to store both versions of 
the metadata.  With the old cp -al method rsync just makes the metadata 
change on the hard link already in the target which causes all instances 
of the file to be changed rather than the file being duplicated.


On 5/14/23 08:38, c.buhtz--- via rsync wrote:

Hello,

I know it is a often discussed topic how rsync decide about using
hardlinks or copy a file. Even if content is unchanged problems are
often file permissions and owner ships. I know that.

Is it possible to configure rsync that way that it logs for each file
its decision about using a hardlink and if not why exactly it doesn't?

The background of my question:
I'm part of maintainer team of "Back In Time" a desktop backup software
using rsync in the back. For years we have users reporting about the
hardlink problem. And we are sure that it isn't their fault but ours.
We can't reproduce the problems for sure but we observe the behavior
also on our own machines sometimes.

It would help our investigation if we could better understand the
hardlink-decision for each file and folder.

Kind
Christian



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: prevent filter excluded files from being deleted

[Kevin Korb via rsync]

I am not 100% sure I am interpreting this correctly but I think you are 
complaining that the file was being deleted in the first command?  If 
so, instead of -F try --include='*/' --exclude='*'.  Otherwise, maybe 
you want a second -F?


On 3/6/23 16:04, Heiko Schlittermann via rsync wrote:

Hello,

given are 2 directories:

 a
 ├── a-file
 └── .rsync-filter

 b
 └── a-file

I'd like to sync a/ -> b/, but I'd like to *exclude* all files. But I do
not want to delete the excluded files. (The real scenario is a way more
complex, the above is my reproducer.)

and the following rsync command: `rsync -F --del a/ b/`
Note, there is *no* `--delete-excluded`:

 $ rsync -F --del -inav a/ b/
 sending incremental file list
 *deleting   a-file
 .d..t.. ./

 sent 62 bytes  received 29 bytes  182.00 bytes/sec
 total size is 0  speedup is 0.00 (DRY RUN)

Why? Because it is excluded? Using the `--exclude` option does what I
expect:

 $ rsync --exclude 'a-file' --del -inav a/ b/
 sending incremental file list
 .d..t.. ./
 >f+ .rsync-filter

 sent 92 bytes  received 22 bytes  228.00 bytes/sec
 total size is 4  speedup is 0.04 (DRY RUN)

even does what I expect:

 $ rsync --filter '- *' --del -inav a/ b/
 sending incremental file list
 .d..t.. ./

 sent 58 bytes  received 19 bytes  154.00 bytes/sec
 total size is 0  speedup is 0.00 (DRY RUN)

Version:
rsync  version 3.2.3  protocol version 31

 Best regards from Dresden/Germany
 Viele Grüße aus Dresden
 Heiko Schlittermann
--
  SCHLITTERMANN.de  internet & unix support -
  Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
  gnupg encrypted messages are welcome --- key ID: F69376CE -




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Apparent synchronization bug in rsync

[Kevin Korb via rsync]

The included github test case failed for me on vanilla rsync-3.2.7...

% ./test.sh
new.bin test.bin differ: byte 28345, line 1
% md5sum -b *.bin
7e11df4130968b5e4859f58b9bd6b453 *initial.bin
ff14ab080514d5681c99e4e1de01e29f *new.bin
151dd8b1c7bd628afcf38ac6c44155be *test.bin
% rsync --version
rsync  version 3.2.7  protocol version 31
Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
hardlink-symlinks, no IPv6, atimes, batchfiles, inplace, append, ACLs,
xattrs, optional secluded-args, iconv, prealloc, stop-at, no crtimes
Optimizations:
no SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none
Compress list:
zstd lz4 zlibx zlib none
Daemon auth list:
sha512 sha256 sha1 md5 md4

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.


On 12/9/22 00:02, Wayne Davison wrote:

On Thu, Dec 8, 2022 at 12:49 PM Kevin Korb wrote:

The first "Bug Fix" listed in
https://rsync.samba.org/ftp/rsync/NEWS#3.2.4
 is almost certainly
related.  Apparently it was fixed via remote option but since your
test doesn't include networking that fix wouldn't help.


No, the bug was fixed in 3.2.4 and it included an extra feature that 
allows rsync to bypass the bug in an older remote rsync that still has 
the bug.


On 12/8/22 15:03, Mark Raymond wrote:

This looks like a bug to me; am I missing something? If it is a bug,
I'm  happy to create an issue in GitHub.


I have managed to get a test case that fails.  If you want to also 
supply one, feel free.


..wayne..


--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Apparent synchronization bug in rsync

[Kevin Korb via rsync]

The first "Bug Fix" listed in 
https://rsync.samba.org/ftp/rsync/NEWS#3.2.4 is almost certainly 
related.  Apparently it was fixed via remote option but since your test 
doesn't include networking that fix wouldn't help.


On 12/8/22 15:03, Mark Raymond via rsync wrote:
I believe I have found a synchronization bug in rsync. I have a pair of 
files, which when synced with||

||
|rsync --no-whole-file --inplace -S|||
||
|are not| identical afterwards. All three options are required, if I 
remove any one of the the files sync correctly. I have put the two 
files, along with a minimal test script, in this GitHub repo: 
https://github.com/mark-raymond/rsync-test


This looks like a bug to me; am I missing something? If it is a bug, I'm 
happy to create an issue in GitHub.

||



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files

[Kevin Korb via rsync]

Rsync does not have a --no-clobber.  -n is --dry-run.  It shows what 
rsync would do without actually doing it.  But I didn't know you were 
using --backup.  Anyway, I tried it again with --backup and 
--remove-source-files and it did in fact delete all the files (but not 
the dirs).  It didn't have any output indicating such so it seems like a 
problem to me.  BTW, it isn't the --backup either.  Just the 
--remove-source-files is enough I just didn't know it because it isn't 
in the output so --dry-run gives no clue.


On 10/17/22 23:51, Sridhar Sarnobat wrote:

I think I must be missing something.  If source and dest are the same
place rsync shouldn't do anything unless it it responding to changes
happening at the same time.  For example, when I do 'rsync -vain
--remove-source-files /tmp/ /tmp/' rsync does nothing.


Thanks for the response. You are using "-n" / "--no-clobber".

In my case I'm using "-b" / "--backup" (or overwrite). My use case is to 
get everything out of the source location, and "--no-clobber" will leave 
things behind. So I guess I should refine my issue to "when you're not 
using --no-clobber."


Do I have a case here? Or is this erasing both the source and 
destination a valid behaviour in my case?


--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: [3.1.3] --remove-source-files $SRC/ $SRC/ - erases all files

[Kevin Korb via rsync]

I think I must be missing something.  If source and dest are the same 
place rsync shouldn't do anything unless it it responding to changes 
happening at the same time.  For example, when I do 'rsync -vain 
--remove-source-files /tmp/ /tmp/' rsync does nothing.


On 10/17/22 23:12, Sridhar Sarnobat via rsync wrote:
 >> why not avoid using "--remove-source-files" and delete files 
manually/via extra step afterwards


Multiple reasons. Among them:
1) my jobs run a long time and there are a lot of them. Trying to 
remember what I did over ssh (where history doesn't get saved) is an 
extra cognitive load and the system is in a transient state (I've ended 
up with duplicate files unless I do the extra work later - potentially 
much later depending on the size of the job). I may also have multiple 
rsync commands running in separate terminals.
2) I am managing a lot of large files, and my collection of disks 
doesn't have 2x free space.

3) 2 operations instead of 1 gives more room to make a mistake.

This is a workaround that I can occasionally use, and you could even say 
"why not just use ext4magic?" to recover accidentally deleted files etc.


Regardless, the semantics of deleting source files without leaving one 
copy of your files intact should never happen right?





--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to use --rsh with multiple sub-arguments?

[Kevin Korb via rsync]

You can put it all in quotes.  Also, if you are scripting you might find 
setting $RSYNC_RSH easier.


On 10/7/22 04:46, c.buhtz--- via rsync wrote:

Hello,

rsync does offer the --rsh argument. I would like to use it that way

rsync --rsh=ssh -o ServerAliveInterval=240 -o LogLevel=Error -o 
IdentityFile=/home/user/.ssh/id_rsa -p 22 --delete --delete-excluded -v -i / 
user@localhost:"/tmp/tmp6x75lz10"

You see there are several sub-arguments that need to be put to "ssh" via the "--rsh". How 
can I be sure that e.g. "-o ServerAliveInterval" is interpreted by ssh and not by rsync?



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: ignore mtime - or any metadata

[Kevin Korb via rsync]

I can't say I have tried it but -pog would all also result in 
duplicating files (assuming they are different).


On 9/25/22 14:49, Hardy via rsync wrote:

Thanks Kevin,

-a implies rlptgoD and I obviously used it by comfort and habit. 
Omitting -t (and perhaps also gop, not really needed) might solve my 
problem? Will give it a try.


Am 25.09.22 um 20:30 schrob Kevin Korb via rsync:
-a is telling rsync to copy the metadata.  --size-only is telling 
rsync to skip the contents of files that are the same time.  Without 
--link-dest rsync would just update the metadata on the target. 
However, with --link-dest rsync is being told to make the same file 
with 2 different metadatas.  It can only do so by duplicating the file 
(effectively the same as --copy-dest).


On 9/25/22 14:26, Hardy via rsync wrote:

Hi,

SHORT:
rsync shall ignore differences in meta (inode) data, but does not 
even with --size-only. -- If in combination with --link-dest.

Is there any flag I missed?

EXPLANATION:
I use
rsync -avbuH --size-only --stats --delete-excluded 
--link-dest=/path/to/last "user@system:Shared/Data" /path/to/now


to sync backup-style and use the --link-dest option to minimize 
traffic (and space) and still have a full backup each time. So far 
nothing special.
Now it happens that at the source side data changes are moderate, but 
meta information (like atime, mtime) do change a lot. I didn't find a 
solution to make rsync ignore these differences to use a link. 
--size-only and -t (implied by -a) work ok with a simple syncs, but 
obviously (for me) it does not in combination with --link-dest. I can 
understand the motivation behind this: rsync is not allowed to change 
meta data of the existing inode, nor can it neglect the change - or 
can it?


Hardy







--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: ignore mtime - or any metadata

[Kevin Korb via rsync]

-a is telling rsync to copy the metadata.  --size-only is telling rsync 
to skip the contents of files that are the same time.  Without 
--link-dest rsync would just update the metadata on the target. 
However, with --link-dest rsync is being told to make the same file with 
2 different metadatas.  It can only do so by duplicating the file 
(effectively the same as --copy-dest).


On 9/25/22 14:26, Hardy via rsync wrote:

Hi,

SHORT:
rsync shall ignore differences in meta (inode) data, but does not even 
with --size-only. -- If in combination with --link-dest.

Is there any flag I missed?

EXPLANATION:
I use
rsync -avbuH --size-only --stats --delete-excluded 
--link-dest=/path/to/last "user@system:Shared/Data" /path/to/now


to sync backup-style and use the --link-dest option to minimize traffic 
(and space) and still have a full backup each time. So far nothing special.
Now it happens that at the source side data changes are moderate, but 
meta information (like atime, mtime) do change a lot. I didn't find a 
solution to make rsync ignore these differences to use a link. 
--size-only and -t (implied by -a) work ok with a simple syncs, but 
obviously (for me) it does not in combination with --link-dest. I can 
understand the motivation behind this: rsync is not allowed to change 
meta data of the existing inode, nor can it neglect the change - or can it?


Hardy



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: question abount pre-xfer exec

[Kevin Korb via rsync]

You aren't logging any stderr.  That is where any error messages would 
go.  Add some 2>&1.

Also, mount has a -v

On 9/24/22 09:15, dotdeb--- via rsync wrote:


I've been using rsync for years to backup my machines both at work and 
at home.
These days I faced a new "challenge": at work I connect my laptop to a 
docking station with an external usb disk. I'd like to use this disk as 
a backup volume.


I put my disk in /etc/fstab to be mounted at boot (with 'nofail' option 
to avoid errors when I'm at home). I have no problems if the laptop is 
booted after the connection to the docking station but, if I work at 
home, suspend the laptop and then go to work and connect it to the 
docking station and resume it, the /backup volume will not be mounted 
automatically.


I found that rsyncd.conf can execute scripts before and I tried to 
create a script to be executed (as early stage or pre transfer? a bit 
confuser about it) to check if /backup is mounted and mount it if not.


I verified that the script is executed (I put there some debugging 
"echo" sent to the log file) but the mount command within it does not 
mount anything.


Here it is the rsyncd.conf

##
read only = false
write only = false
usechroot = true
uid = 0
gid = 0

early exec = /tmp/test-pre-exec

[rsync-backup-xxx]
         comment Local rsync-backup of xxx
         path = /backup/xxx
         log file = /var/log/rsyncd.log
##

and the script /tmp/test-pre-exec

##
#!/bin/sh

echo -n "executing pre-xfer script ..." >> /var/log/rsyncd.log

if ! grep -qs '/backup ' /proc/mounts
then
     echo -n "mounting /backup ..." >> /var/log/rsyncd.log
     /usr/bin/mount >> /var/log/rsyncd.log
fi

echo " done" >> /var/log/rsyncd.log

exit 0
##




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync timeout non-effective

[Kevin Korb via rsync]

You are using rsync over ssh.  The connection timeout (and port) options 
don't matter if rsync isn't doing the networking.


On 9/1/22 08:49, Roland via rsync wrote:

hello,

i do some backup via rsync/ssh and pull data from remote machine to
local machine.

whenever on remote machine there is some network attached subdir getting
"stuck" (i.e. read/write is blocking, eg. by suspended zfs pool because
of error), rsync backup also hangs forever.

i wonder what's the purpose of "--timeout" then.

local machine:  3.1.2
remote machine: 3.2.3

command
rsync -avi --stats --delete --delete-excluded --inplace --numeric-ids
--out-format=%-15i %-15b %-15l %-20M %n%L --timeout=900
--rsync-path=export PATH=/usr/local/bin:$PATH;rsync
--exclude-from=/backup/rsync.exclude.all
--exclude-from=/zfspool/backup/myhost/rsync.exclude root@myhost:/
/zfspool/backup/myhost/backup



what can i do to make this more robust ?

rsync should not hang, as it's getting started via script and the hang
blocks backup of other hosts.


roland





--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: do I always have to use either --checksum or --times ?

[Kevin Korb via rsync]

Yeah, unfortunately rsync's --checksum doesn't even employ obvious 
optimizations that the man page says it does.  There is no reason for it 
to checksum files that only exist on one end or are different in sizes 
yet it does.  Just try doing an rsync of a huge tree to an empty dir. 
You won't run out of space.  You will get tired of waiting for it to do 
something and abort it.


On 7/15/22 02:57, Robin Lee Powell wrote:

--checksum is only slower than re-copying if your network connection
between the hosts is similar in speed (or faster than) each host's
local disk access.

If local disk access is 10x your network link, it is definitely not
slower than re-copying.

Having said that, it really is *very* slow, and before you use it
you should have a clear situation in mind that makes it plausible that
two files could have the same size and last mod time and still not
have the same data.

Or, alternately, be in a situation where a bit flip would be
catastrophic.  Such situations are not common.

On Thu, Jul 14, 2022 at 04:26:48AM -0400, Kevin Korb via rsync wrote:

You should almost never use --checksum.  It is slower than just re-copying
everything.  You should almost always use --times (or --archive which
includes --times).  Without this rsync is almost as dumb as cp.  Also, ssh
has been the default --ssh for a long time.

On 7/14/22 04:22, Fourhundred Thecat via rsync wrote:

Hello,

I want to sync local folder to remote server. When I run follwing
command repeatedly, it always transfers everything each time again and
again:

    rsync --rsh='ssh' foo/ server:/foo/

does it mean I have to always use either --checksum or --times, to
prevent repeated transfer of files that have not changed ?

thank you,



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: do I always have to use either --checksum or --times ?

[Kevin Korb via rsync]

You should almost never use --checksum.  It is slower than just 
re-copying everything.  You should almost always use --times (or 
--archive which includes --times).  Without this rsync is almost as dumb 
as cp.  Also, ssh has been the default --ssh for a long time.


On 7/14/22 04:22, Fourhundred Thecat via rsync wrote:

Hello,

I want to sync local folder to remote server. When I run follwing
command repeatedly, it always transfers everything each time again and
again:

   rsync --rsh='ssh' foo/ server:/foo/

does it mean I have to always use either --checksum or --times, to
prevent repeated transfer of files that have not changed ?

thank you,



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Does rsync verify its writes?

[Kevin Korb via rsync]

Note that it won't actually tell you that the file was written wrongly 
just that it is different.  In fact, you can only even know that it 
might be detecting a broken file with --itemize-changes and look for 
files with a checksum difference but not a timestamp difference.  Though 
there are legitimate causes for that.


On 7/12/22 17:26, Paul Slootman via rsync wrote:

On Tue 12 Jul 2022, Kevin Korb via rsync wrote:


Rsync does not verify writes.  --checksum doesn't verify anything. Sounds
like you want a file verification tool.  The simplest would be md5sum.


Running rsync --checksum directly after transferring your files will
verify that the files are written correctly (if the source hasn't
changed in the meantime). That might help to give some peace of mind.


Paul


On 7/12/22 02:31, Mark Filipak via rsync wrote:

Hello. Does rsync verify its writes?

Re, 'info rsync'.

Maybe I just being stupid, but there's no mention of verification in the
'DESCRIPTION' section, so despite the words in the 'OPTIONS' section,
'-c, --checksum' topic (which I may be misinterpreting), I assume rsync
does not verify except for the checksum directive.

I admit that I'm paranoid. ;-) ...Please clarify.

Regards, and Thanks,
Mark Filipak.



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Does rsync verify its writes?

[Kevin Korb via rsync]

Rsync does not verify writes.  --checksum doesn't verify anything. 
Sounds like you want a file verification tool.  The simplest would be 
md5sum.


On 7/12/22 02:31, Mark Filipak via rsync wrote:

Hello. Does rsync verify its writes?

Re, 'info rsync'.

Maybe I just being stupid, but there's no mention of verification in the 
'DESCRIPTION' section, so despite the words in the 'OPTIONS' section, 
'-c, --checksum' topic (which I may be misinterpreting), I assume rsync 
does not verify except for the checksum directive.


I admit that I'm paranoid. ;-) ...Please clarify.

Regards, and Thanks,
Mark Filipak.



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Customizing compression per file?

[Kevin Korb via rsync]

It actually does that by default.  Though you might need to add to the 
list of file types with --skip-compress.


On 6/24/22 19:35, Joseph Garvin via rsync wrote:
I have an rsync cron job to backup folders that contain many different 
types of files. Most of these are uncompressed files, so the -z option 
would speed up their transfer. However there are a few files that are 
already compressed where -z will just needlessly burn CPU and slow down 
the transfer. Is there any way to express something like, "compress all 
files except those with extension XXX?" The files are in a proprietary 
format so it doesn't make sense to add them to a whitelist in rsync itself.


Thanks,

Joe G.




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync Users and Groups

[Kevin Korb via rsync]

Nope.  Rsync groups are not groups of users they are just @users with 
their own password.  I believe the @ just designates that you intend 
multiple people to have that password and use that username.


On 6/24/22 12:34, Alexander Gribanov via rsync wrote:

Hello, Kevin! Thank You very much for the reply.

пт, 24 июн. 2022 г. в 19:00, Kevin Korb via rsync <mailto:rsync@lists.samba.org>>:


I think you are thinking too much of rsync here.  Rsync groups are the
same as users they just have an @ in front of the name.  If you want
UNIX style users and groups then use rsync over ssh and get the
bonus of
ssh's authentication as well as not needing an rsyncd.conf file at all.


Well, actually, I wouldn't like to do anything with system users and groups,
but it would be nice to have virtual users and groups, which are 
available for rsyncd only.

Is it possible?

Or maybe there is an opportunity to use external authentication 
mechanisms or some kind of modules, etc?


On 6/24/22 11:13, Alexander Gribanov via rsync wrote:
 > Hello everybody!
 >
 > I'm trying to configure rsync-server, but still I can't figure
out how
 > to manage users and groups.
 >
 > I read in the documentation that I could use both, but there is
nothing
 > about how to add a user into a particular group...
 >
 > Why am I doing this?
 > 1. I have about 10 modules, which are different and must be
available
 > for some users and not available for other users.
 > 2. There are many users, new users are arriving, some old users are
 > going out, so it would be easier to assign group permissions to the
 > modules and then manage just user-group relations, but I couldn't
find
 > that in documentation, nor examples on the internet...
 >
 > Could anybody please give me a hint or a clue about how to assign
a user
 > to a particular group? I'm sure that this should work, but I
can't find
 > it out myself :(
 >
 > Thank you very much.
 >

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

         Kevin Korb                      Phone:    (407) 252-6853
         Systems Administrator           Internet:
         FutureQuest, Inc.               ke...@futurequest.net  (work)
         Orlando, Florida k...@sanitarium.net
<mailto:k...@sanitarium.net> (personal)
         Web page: https://sanitarium.net/ <https://sanitarium.net/>
         PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync Users and Groups

[Kevin Korb via rsync]

I think you are thinking too much of rsync here.  Rsync groups are the 
same as users they just have an @ in front of the name.  If you want 
UNIX style users and groups then use rsync over ssh and get the bonus of 
ssh's authentication as well as not needing an rsyncd.conf file at all.


On 6/24/22 11:13, Alexander Gribanov via rsync wrote:

Hello everybody!

I'm trying to configure rsync-server, but still I can't figure out how 
to manage users and groups.


I read in the documentation that I could use both, but there is nothing 
about how to add a user into a particular group...


Why am I doing this?
1. I have about 10 modules, which are different and must be available 
for some users and not available for other users.
2. There are many users, new users are arriving, some old users are 
going out, so it would be easier to assign group permissions to the 
modules and then manage just user-group relations, but I couldn't find 
that in documentation, nor examples on the internet...


Could anybody please give me a hint or a clue about how to assign a user 
to a particular group? I'm sure that this should work, but I can't find 
it out myself :(


Thank you very much.



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: Problem with file exclusion pattern

[Kevin Korb via rsync]

Sure, that is the way it normally works.  By default everything is 
included.  Excludes exclude things.  Includes are exceptions to includes 
that follow.


Note that this can get very complicated.  --exclude-from=file.txt makes 
things much easier as you can use the +/- syntax to put both includes 
and excludes into the same file.


On 4/30/22 20:48, H via rsync wrote:

On 04/30/2022 08:22 PM, Kevin Korb via rsync wrote:

Includes override excludes that follow.  So, your include of * meant that 
nothing was being excluded.  An exclude before any includes isn't affected by 
the includes.

On 4/30/22 20:04, H via rsync wrote:

On 04/30/2022 07:56 PM, H via rsync wrote:


Ah, I was under the impression that all inclusion patterns need to preceed the 
exclusion patterns?



 Forwarded Message 
Subject: Re: Problem with file exclusion pattern
Date: Sat, 30 Apr 2022 18:49:22 -0400
From: Kevin Korb 
To: H 



Drop the include of * or move the exclude of *.~lock* before it.  Also,
-c is almost always a bad idea.

On 4/30/22 18:18, H via rsync wrote:

I am running rsync on Linux and have yet to find the appropriate pattern to 
exclude files containing '.~lock' as part of the name of files existing in any 
directory, ie ~/test and below.

rsync -vHrltDium -c --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --stats --include='*' 
--include='*/' --exclude='*.~lock*' --exclude='*' -e "ssh -y -p 22" ~/test/ 
someone@1.2.3.4:~/test
Can anyone point out what I have missed in the above?
Thank you.


--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
 Kevin Korb    Phone:    (407) 252-6853
 Systems Administrator    Internet:
 FutureQuest, Inc.    ke...@futurequest.net   (work)
 Orlando, Florida    k...@sanitarium.net  (personal)
 Web page:    https://sanitarium.net/
 PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



When I dropped the '*' include it did not sync files in the topmost directory? 
Perhaps it always needs to be included among the options?





Thank you. Would one ever put an exclude /after/ the includes?




--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Fwd: Problem with file exclusion pattern

[Kevin Korb via rsync]

Includes override excludes that follow.  So, your include of * meant 
that nothing was being excluded.  An exclude before any includes isn't 
affected by the includes.


On 4/30/22 20:04, H via rsync wrote:

On 04/30/2022 07:56 PM, H via rsync wrote:


Ah, I was under the impression that all inclusion patterns need to 
preceed the exclusion patterns?




 Forwarded Message 
Subject:Re: Problem with file exclusion pattern
Date:   Sat, 30 Apr 2022 18:49:22 -0400
From:   Kevin Korb 
To: H 



Drop the include of * or move the exclude of *.~lock* before it.  Also,
-c is almost always a bad idea.

On 4/30/22 18:18, H via rsync wrote:
> I am running rsync on Linux and have yet to find the appropriate pattern to 
exclude files containing '.~lock' as part of the name of files existing in any 
directory, ie ~/test and below.
> 
> rsync -vHrltDium -c --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --stats --include='*' --include='*/' --exclude='*.~lock*' --exclude='*' -e "ssh -y -p 22" ~/test/ someone@1.2.3.4:~/test
> 
> Can anyone point out what I have missed in the above?
> 
> Thank you.
> 
> 


--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net   (work)
Orlando, Floridak...@sanitarium.net  (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


When I dropped the '*' include it did not sync files in the topmost 
directory? Perhaps it always needs to be included among the options?





--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Trying to elevate rsync privileges when connecting over ssh without using NOPASSWD in sudoers

[Kevin Korb via rsync]

Rsync includes a script named rrsync that handles this perfectly.

On 3/12/22 01:08, Richard Hector via rsync wrote:

On 12/03/22 18:38, Richard Hector via rsync wrote:
And I do my backups (using dirvish) as root, using a key with a forced 
command.


FWIW, that forced command is here:

https://github.com/rwhector/dirvish-forced-command

It's rather unpolished and undocumented, but comments very welcome :-)

I've also had an issue due to some server-side-only arguments to rsync 
being undocumented, which means I can't validate them, and basically 
have to accept anything ... I'd love to know why this is or has to be 
the case :-) I didn't get any particularly useful answers back in 
January 2019 ...


Cheers,
Richard



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Confused as to why rsync thinks time, owner and group of many files differ

[Kevin Korb via rsync]

Are you using the same source and target each time?  I ask because the 
only discrepancy I see is the link count which shows that there are 11 
more instances of that inode on the source than the target.  Maybe 
instances in other snapshots are being updated/re-linked?


The only other thing to mention is that when you abort rsync (with -P or 
--inplace) incomplete files are left.  Rsync doesn't fix the owner+group 
until it is done with a directory and it doesn't fix the timestamp until 
it is done with a file.  This would be why you shouldn't mix those 
options with --update since the truncated file will be newer than the 
source file.


On 2/3/22 17:04, Andy Smith via rsync wrote:

Hi,

I am at the moment using rsync to move quite a big set of backups
from one machine to another. The source filesystem is xfs; the
target filesystem is btrfs.

For various reasons I have been stopping the rsync part way through
and re-starting. I have noticed that a large number of files are
transferred over and over and I can't work out why.

Example:

sudo rsync -iPva \
 --inplace \
 --numeric-ids \
 --delete \
 /data/backup/rsnapshot/daily.0/cacti/ \
 root@koff:/data/backup/rsnapshot/daily.0/cacti/

...
http://rsync.samba.org/
Capabilities:
 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
 socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
 append, ACLs, xattrs, iconv, symtimes, prealloc

Destination:

$ rsync --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
 socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
 batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
 symtimes, prealloc, stop-at, no crtimes
Optimizations:
 SIMD, asm, openssl-crypto
Checksum list:
 xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
 zstd lz4 zlibx zlib none

What am I missing?

Thanks,
Andy



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Copying a large file from an USB 3 disk (NTFS or EXT4) to another one (UFS/FreeBSD) is a very slow process....

[Kevin Korb via rsync]

Rsync is designed to reduce the amount of data transmitted over the 
network.  If rsync isn't networking it can't do that.  However, it still 
uses the same code so it is still using a sender and a receiver rather 
than simply reading and writing as cp does.  Also, rsync forces 
--whole-file because using rsync's algorithm to delta-copy is slower 
than just re-copying a file (especially if one of the local paths is 
really a network mount and double especially if that is the writing end).


On 11/28/21 16:44, Harry Mangalam wrote:

Can you elaborate on why this is?

I wrote a parallel rsync wrapper that works very well over networks but 
is similarly very slow over local disks. I thought it was a bug in my 
code but didn't get around to tracking it down since my use cases were 
all network/ parallel file systems.

Harry

On Sun, Nov 28, 2021, 11:43 AM Kevin Korb via rsync 
mailto:rsync@lists.samba.org>> wrote:


rsync is terribly slow at local copies.  Also, it doesn't do its normal
optimizing (see --whole-file).  Just use cp.

On 11/28/21 13:38, Mario Marietto via rsync wrote:
 > Hello to everyone.
 >
 > I'm copying a large file from a NTFS formatted disk to another
 > one,UFS/FreeBSD disk,both are removable disks attached to the USB 3
 > port. The file is 200 GB large and it is copied very slowly. Why
it is
 > so slow ? I don't know where to store my virtual machines. I
tried to
 > save them on the ext4 disk because I wanted to share them easily
between
 > Linux and FreeBSD but I've realized that when I mount the disk in
 > FreeBSD after some time it corrupts. I tried to store it on the NTFS
 > disk but it happens the same. So,now I'm on FreeBSD and I'm
copying them
 > to a dedicated UFS/FreeBSD style disk,but as I said,the speed is
very
 > slow. How can I increase the speed ?. Actually I'm using this
 > command,because I want to resume the uploading if it breaks at
some point :
 >
 >
 > root@marietto:/mnt/da3p2/bhyve/Ubuntu # rsync -avAXEWSlHh
 > /mnt/da0p1/Backups/OS/bhyve/Ubuntu/im* . --no-compress
--info=progress2
 >
 > sending incremental file list
 > impish-cuda-11-4-nvidia-470.img
 >
 > 2.13M 0% 9.49kB/s 6284:55:38
 >
 >
 > and : where do you save large files ? what's the procedure that
you use
 > to copy large files with a decent speed ? Unfortunately under
Linux is
 > not safe to use a RW ufs disk access. So,I'm out of solutions.
 >
 > -
 > Mario.
 >

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

         Kevin Korb                      Phone:    (407) 252-6853
         Systems Administrator           Internet:
         FutureQuest, Inc.               ke...@futurequest.net  (work)
         Orlando, Florida k...@sanitarium.net
<mailto:k...@sanitarium.net> (personal)
         Web page: https://sanitarium.net/ <https://sanitarium.net/>
         PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing

list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
<https://lists.samba.org/mailman/listinfo/rsync>
Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
<http://www.catb.org/~esr/faqs/smart-questions.html>



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Copying a large file from an USB 3 disk (NTFS or EXT4) to another one (UFS/FreeBSD) is a very slow process....

[Kevin Korb via rsync]

rsync is terribly slow at local copies.  Also, it doesn't do its normal 
optimizing (see --whole-file).  Just use cp.


On 11/28/21 13:38, Mario Marietto via rsync wrote:

Hello to everyone.

I'm copying a large file from a NTFS formatted disk to another 
one,UFS/FreeBSD disk,both are removable disks attached to the USB 3 
port. The file is 200 GB large and it is copied very slowly. Why it is 
so slow ? I don't know where to store my virtual machines. I tried to 
save them on the ext4 disk because I wanted to share them easily between 
Linux and FreeBSD but I've realized that when I mount the disk in 
FreeBSD after some time it corrupts. I tried to store it on the NTFS 
disk but it happens the same. So,now I'm on FreeBSD and I'm copying them 
to a dedicated UFS/FreeBSD style disk,but as I said,the speed is very 
slow. How can I increase the speed ?. Actually I'm using this 
command,because I want to resume the uploading if it breaks at some point :



root@marietto:/mnt/da3p2/bhyve/Ubuntu # rsync -avAXEWSlHh 
/mnt/da0p1/Backups/OS/bhyve/Ubuntu/im* . --no-compress --info=progress2


sending incremental file list
impish-cuda-11-4-nvidia-470.img

2.13M 0% 9.49kB/s 6284:55:38


and : where do you save large files ? what's the procedure that you use 
to copy large files with a decent speed ? Unfortunately under Linux is 
not safe to use a RW ufs disk access. So,I'm out of solutions.


-
Mario.



--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Workaround for rsyncing when fileflag schg is set?

[Kevin Korb via rsync]

No, rsync normally handles such problems well.  Unfortunately,
--fileflags is an OS vendor added feature rather than an official rsync
feature so it is less well thought out.

On 10/31/21 3:51 AM, Perry Hutchison wrote:
> That sort of snafu is why find(1) has the -depth directive.
> Does rsync have anything similar?
> 
> Kevin Korb via rsync  wrote:
> 
>> There maybe a proper solution but an obvious workaround would be to run
>> rsync twice.  The first time without the --fileflags option.
>>
>> --no-perms wouldn't help.  That is only the standard unix permissions.
>>
>> On 10/30/21 8:04 PM, Fred Fugate via rsync wrote:
>>> Hi,
>>>
>>> I have some subdirectories within a home directory that are chflagged
>>> schg, aka system immutable.
>>>
>>> When I rsync the home directory to another machine, I get lots of errors
>>> for these subdirectories and they fail to copy to the target. NOTE: The
>>> target is empty; rsync is copying to an empty remote machine, not trying
>>> to overwrite anything.
>>>
>>> I understand why this is happening; it's the schg flag. But is there a
>>> workaround using some combination of rsync options or multiple passes?
>>>
>>> Here is an example.
>>>
>>> The source directory is /Users/redacted.
>>>
>>> Within /Users/redacted is a subdirectory foo that has flag schg set:
>>> /Users/redacted/Documents/artwork/foo
>>>
>>> # ls -laO /Users/redacted/Documents/artwork
>>> total 80
>>> drwxr-xr-x ?? 20 redacted ??staff ??- ?? ?? ?? ??680 Sep 21 21:06 .
>>> drwx--@ 609 redacted ??staff ??- ?? ?? ??20706 Oct 29 16:07 ..
>>> -rw-r--r--@ ?? 1 redacted ??staff ??- ?? ?? ??18436 Sep 21 20:55 somefile
>>> drwxrwxrwx@ ??18 redacted ??staff ??schg ?? ?? 612 Apr 12 ??2006 foo
>>>
>>>
>>> My rsync args are:
>>>
>>> --verbose --archive --one-file-system --acls --hard-links --xattrs
>>> --protect-args --delete-after --numeric-ids --itemize-changes --crtimes
>>> --fileflags --force-change
>>> --rsync-path=/opt/rsync323/bin/rsync
>>>
>>> My rsync version is 3.2.3
>>>
>>> My execution and output looks like this, run as root:
>>>
>>> # /opt/rsync323/bin/rsync [ARGS ABOVE] /Users/redacted
>>> remotemachine.domain.com:/Users
>>> .
>>> .
>>> rsync: [receiver] mkstemp
>>> "/Users/redacted/Documents/artwork/foo/bar/.background.tiff.vYOAS2"
>>> failed: Operation not permitted (1)
>>> rsync: [receiver] mkstemp
>>> "/Users/redacted/Documents/artwork/foo/bar/.dc4.orange.CMYK.tiff.rYCmAP"
>>> failed: Operation not permitted (1)
>>> rsync: [receiver] mkstemp
>>> "/Users/redacted/Documents/artwork/foo/bar/.barcode.tiff.2E4mec" failed:
>>> Operation not permitted (1)
>>> .
>>> .
>>>
>>> In a nutshell:
>>>
>>> Subdirectory foo gets created on the receiver, but it gets created with
>>> the schg flag set.
>>>
>>> No further copying into foo can happen after that, because the schg flag
>>> prevents that.
>>>
>>> Subdirectory bar cannot be created under foo, and
>>> .background.tiff.vYOAS2 and other files cannot be created under bar, etc.
>>>
>>> How can I force the schg flags to be set on the receiver AFTER
>>> everything has been copied from the source?
>>>
>>> I really don't want to remove all of my schg settings on the source
>>> before rsyncing to the target.
>>>
>>> Would --no-perms allow this? But what would it break?

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Workaround for rsyncing when fileflag schg is set?

[Kevin Korb via rsync]

There maybe a proper solution but an obvious workaround would be to run
rsync twice.  The first time without the --fileflags option.

--no-perms wouldn't help.  That is only the standard unix permissions.

On 10/30/21 8:04 PM, Fred Fugate via rsync wrote:
> Hi,
> 
> I have some subdirectories within a home directory that are chflagged
> schg, aka system immutable.
> 
> When I rsync the home directory to another machine, I get lots of errors
> for these subdirectories and they fail to copy to the target. NOTE: The
> target is empty; rsync is copying to an empty remote machine, not trying
> to overwrite anything.
> 
> I understand why this is happening; it's the schg flag. But is there a
> workaround using some combination of rsync options or multiple passes?
> 
> 
> 
> Here is an example.
> 
> The source directory is /Users/redacted.
> 
> Within /Users/redacted is a subdirectory foo that has flag schg set:
> /Users/redacted/Documents/artwork/foo
> 
> # ls -laO /Users/redacted/Documents/artwork
> total 80
> drwxr-xr-x   20 redacted  staff  -        680 Sep 21 21:06 .
> drwx--@ 609 redacted  staff  -      20706 Oct 29 16:07 ..
> -rw-r--r--@   1 redacted  staff  -      18436 Sep 21 20:55 somefile
> drwxrwxrwx@  18 redacted  staff  schg     612 Apr 12  2006 foo
> 
> 
> My rsync args are:
> 
> --verbose --archive --one-file-system --acls --hard-links --xattrs
> --protect-args --delete-after --numeric-ids --itemize-changes --crtimes
> --fileflags --force-change
> --rsync-path=/opt/rsync323/bin/rsync
> 
> My rsync version is 3.2.3
> 
> My execution and output looks like this, run as root:
> 
> 
> # /opt/rsync323/bin/rsync [ARGS ABOVE] /Users/redacted
> remotemachine.domain.com:/Users
> .
> .
> rsync: [receiver] mkstemp
> "/Users/redacted/Documents/artwork/foo/bar/.background.tiff.vYOAS2"
> failed: Operation not permitted (1)
> rsync: [receiver] mkstemp
> "/Users/redacted/Documents/artwork/foo/bar/.dc4.orange.CMYK.tiff.rYCmAP"
> failed: Operation not permitted (1)
> rsync: [receiver] mkstemp
> "/Users/redacted/Documents/artwork/foo/bar/.barcode.tiff.2E4mec" failed:
> Operation not permitted (1)
> .
> .
> 
> 
> 
> 
> 
> In a nutshell:
> 
> Subdirectory foo gets created on the receiver, but it gets created with
> the schg flag set.
> 
> No further copying into foo can happen after that, because the schg flag
> prevents that.
> 
> Subdirectory bar cannot be created under foo, and
> .background.tiff.vYOAS2 and other files cannot be created under bar, etc.
> 
> How can I force the schg flags to be set on the receiver AFTER
> everything has been copied from the source?
> 
> I really don't want to remove all of my schg settings on the source
> before rsyncing to the target.
> 
> Would --no-perms allow this? But what would it break?
> 
> 
> 
> Thanks,
> Fred
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: BackUp with "inverse" increments

[Kevin Korb via rsync]

See --link-dest.  That is what makes rsync shine for backups.

On 10/6/21 10:09 AM, Helmut Jarausch via rsync wrote:
> Hi,
> 
> I'd like to mirror my root file system (e.g.) to a different disk. The
> mirror should always be most recent.
> In addition, I'd like to be able to restore my file system to the state
> of one or more backups before
> without storing full size snapshots.
> 
> The --backup-dir option does most of the work, but what happens if the
> old backup doesn't contain a file
> which is part of the current source. How can I restore the old version
> including deletion of files
> which didn't exist in the old version?
> 
> Many thanks for some hints or pointers,
> Helmut
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Passing result of shuf to rsync

[Kevin Korb via rsync]

Quoting from the man page

SORTED TRANSFER ORDER
   Rsync always sorts the specified filenames into its internal
transfer list.  This handles the merging together of the contents of identi‐
   cally named directories, makes it easy to remove duplicate
filenames, and may confuse someone when the files are transferred in a
differ‐
   ent order than what was given on the command-line.

   If you need a particular file to be transferred prior to another,
either separate the files into different rsync calls, or consider using
   --delay-updates (which doesn't affect the sorted transfer order,
but does make the final file-updating phase happen much more rapidly).

and...

  NOTE: sorting the list of files in the --files-from input
helps rsync to be more efficient, as it will avoid re-visiting the  path
  elements  that  are shared between adjacent entries.  If
the input is not sorted, some path elements (implied directories) may end
  up being scanned multiple times, and rsync will eventually
unduplicate them after they get turned into file-list elements.


On 9/17/21 8:41 PM, hancooper wrote:
> ‐‐‐ Original Message ‐‐‐
> On Saturday, September 18, 2021 12:29 AM, Kevin Korb  
> wrote:
> 
>> Well, what you have should function. But rsync is going to sort the list.
> 
> I want rsync to use the list as given to it by shuf.
> 
>> On 9/17/21 8:26 PM, hancooper wrote:
>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Saturday, September 18, 2021 12:06 AM, Kevin Korb via rsync 
>>> rsync@lists.samba.org wrote:
>>>
>>>> Not really sure what you are trying to accomplish here. Seems like it
>>>> should work the way you have it. Note that many wonky rsync kludges are
>>>> due to people not realizing that rsync can have multiple source
>>>> arguments. Instead of the source simply being . it can be a list of
>>>> stuff. Also, note that -a includes --recursive except when --files-from
>>>> is in play.
>>>
>>> I am trying send via rsync a shuffled list of files.
>>>
>>>> On 9/17/21 7:11 PM, hancooper via rsync wrote:
>>>>
>>>>> I am trying to pass filename results that have been shuffled, then pass
>>>>> the file to rsync.
>>>>> Not quite sure whether to use --files-from=- or some other way.
>>>>> shuf -n "$nf" -e ${dpath}/${incl} |
>>>>>     rsync -av --update --files-from=- . "$dst"
>>>>
>>>> --
>>>>
>>>> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,
>>>>
>>>> Kevin Korb Phone: (407) 252-6853
>>>>
>>>> Systems Administrator Internet:
>>>>
>>>> FutureQuest, Inc. ke...@futurequest.net (work)
>>>>
>>>> Orlando, Florida k...@sanitarium.net (personal)
>>>>
>>>> Web page: https://sanitarium.net/
>>>>
>>>> PGP public key available on web site.
>>>>
>>>> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,
>>>> ---
>>>>
>>>> Please use reply-all for most replies to avoid omitting the mailing list.
>>>> To unsubscribe or change options: 
>>>> https://lists.samba.org/mailman/listinfo/rsync
>>>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>> --
>>
>> ~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,
>> Kevin Korb Phone: (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. ke...@futurequest.net (work)
>> Orlando, Florida k...@sanitarium.net (personal)
>> Web page: https://sanitarium.net/
>> PGP public key available on web site.
>> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,._.,
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Passing result of shuf to rsync

[Kevin Korb via rsync]

Well, what you have should function.  But rsync is going to sort the list.

On 9/17/21 8:26 PM, hancooper wrote:
> ‐‐‐ Original Message ‐‐‐
> On Saturday, September 18, 2021 12:06 AM, Kevin Korb via rsync 
>  wrote:
> 
>> Not really sure what you are trying to accomplish here. Seems like it
>> should work the way you have it. Note that many wonky rsync kludges are
>> due to people not realizing that rsync can have multiple source
>> arguments. Instead of the source simply being . it can be a list of
>> stuff. Also, note that -a includes --recursive except when --files-from
>> is in play.
> 
> I am trying send via rsync a shuffled list of files.
> 
> 
>> On 9/17/21 7:11 PM, hancooper via rsync wrote:
>>
>>> I am trying to pass filename results that have been shuffled, then pass
>>> the file to rsync.
>>> Not quite sure whether to use --files-from=- or some other way.
>>> shuf -n "$nf" -e ${dpath}/${incl} |
>>>     rsync -av --update --files-from=- . "$dst"
>>
>> --
>>
>> ~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,
>> Kevin Korb Phone: (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. ke...@futurequest.net (work)
>> Orlando, Florida k...@sanitarium.net (personal)
>> Web page: https://sanitarium.net/
>> PGP public key available on web site.
>> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,._.,
>> --
>>
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options: 
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Passing result of shuf to rsync

[Kevin Korb via rsync]

Not really sure what you are trying to accomplish here.  Seems like it
should work the way you have it.  Note that many wonky rsync kludges are
due to people not realizing that rsync can have multiple source
arguments.  Instead of the source simply being . it can be a list of
stuff.  Also, note that -a includes --recursive except when --files-from
is in play.

On 9/17/21 7:11 PM, hancooper via rsync wrote:
> I am trying to pass filename results that have been shuffled, then pass
> the file to rsync.
> Not quite sure whether to use --files-from=- or some other way.
> 
> 
> shuf -n "$nf" -e ${dpath}/${incl} |
>     rsync -av --update --files-from=- . "$dst"
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync options –update and –append-verify

[Kevin Korb via rsync]

--inplace can prevent rsync from running you out of disk space when
updating a large file.  But you don't want to mix it with --update.
Judging by your other message you think that --update means modiify
files that are already on both ends.  It doesn't.  It only means that
rsync is forbidden  to change files on the target that have a newer
timestamp than is on the source.  If you don't need that then you don't
want --update.

Also, when in doubt, --dry-run.

On 9/11/21 11:51 PM, hancooper wrote:
> ‐‐‐ Original Message ‐‐‐
> On Sunday, September 12, 2021 2:34 AM, hancooper via rsync 
>  wrote:
> 
>> ‐‐‐ Original Message ‐‐‐
>> On Sunday, September 12, 2021 2:03 AM, Kevin Korb k...@sanitarium.net wrote:
>>
>>> I thought I did elaborate. If it is a problem for you then maybe you
>>> shouldn't be using --update. Or you should let rsync delete incomplete
>>> files upon abort as it does by default.
>>
>> I am using the following
>>
>> rsync -av --inplace --update --log-file=/media/hagbard/hc/snapshot/test1.log 
>> /media/hagbard/hc1/snapshot/test1 /media/hagbard/hc/snapshot/
>>
>> I want that if I run the rsync command again, the files already transferred 
>> are not resent, thus I used --update.
> 
> Can I re-run without the --update option?  Am I right to say that the files 
> already successfully transferred will not be re-sent again using -av 
> --inplace ?
> 
> What can I do for the files which have been sent but got rsync aborted during 
> transfer?
> 
> 
>>> On 9/11/21 9:29 PM, hancooper wrote:
>>>
>>>> ‐‐‐ Original Message ‐‐‐
>>>> On Saturday, September 11, 2021 11:20 PM, Kevin Korb via rsync 
>>>> rsync@lists.samba.org wrote:
>>>>
>>>>> --archive is all you really need. I actually wish --archive was the
>>>>> default because it is all most people need and with the exception of
>>>>> writing to a FAT filesystem it is almost always needed.
>>>>> --append is for very special cases and should only be used if you really
>>>>> know you need it and why. --append-verify only exists because people
>>>>> seem to think they need --append and get annoyed that it corrupts their
>>>>> files.
>>>>> --update is sometimes helpful however it interacts badly with --partial
>>>>> and --inplace.
>>>>
>>>> That's right. Can you elaborate how --update interacts badly with --partial
>>>> and --inplace?
>>>>
>>>>> Since rsync doesn't set the timestamp of a file until it
>>>>> is finished then any file left incomplete by an aborted rsync will have
>>>>> the timestamp of the abort not the source file. Therefore it will be
>>>>> newer than the source file and unless the source is updated rsync
>>>>> --update will never complete the file.
>>>>
>>>> Is there a solution to this problem and should we consider it a bug?
>>>> I am encountering this, and it's a real problem for me if files are
>>>> not completed.
>>>>
>>>>> On 9/11/21 6:50 PM, hancooper via rsync wrote:
>>>>>
>>>>>> I am struggling to understand exactly what the rsync options --update 
>>>>>> and --append-verify do.
>>>>>> Doing info rsync gives
>>>>>> -u, --update
>>>>>> This forces rsync to skip any files which exist on the destina‐
>>>>>> tion and have a modified time that is newer than the source
>>>>>> file. (If an existing destination file has a modification time
>>>>>> equal to the source file’s, it will be updated if the sizes are
>>>>>> different.)
>>>>>> --append-verify
>>>>>> This works just like the --append option, but the existing data
>>>>>> on the receiving side is included in the full-file checksum
>>>>>> verification step, which will cause a file to be resent if the
>>>>>> final verification step fails (rsync uses a normal, non-append‐
>>>>>> ing --inplace transfer for the resend).
>>>>>> I am using rsync to transfer directories recursively. There are times 
>>>>>> where I have to stop the rsync transfer, and resume the transfer a few 
>>>>>> hours or days later, without affecting the already transferred files at 
>>>>>> destination.
>>>>>> I also got some files that return errors by rsync, such as
>>>>>> rsyn

Re: Rsync options –update and –append-verify

[Kevin Korb via rsync]

I thought I did elaborate.  If it is a problem for you then maybe you
shouldn't be using --update.  Or you should let rsync delete incomplete
files upon abort as it does by default.

On 9/11/21 9:29 PM, hancooper wrote:
> ‐‐‐ Original Message ‐‐‐
> On Saturday, September 11, 2021 11:20 PM, Kevin Korb via rsync 
>  wrote:
> 
>> --archive is all you really need. I actually wish --archive was the
>> default because it is all most people need and with the exception of
>> writing to a FAT filesystem it is almost always needed.
>>
>> --append is for very special cases and should only be used if you really
>> know you need it and why. --append-verify only exists because people
>> seem to think they need --append and get annoyed that it corrupts their
>> files.
>>
>> --update is sometimes helpful however it interacts badly with --partial
>> and --inplace.
> 
> That's right.  Can you elaborate how --update interacts badly with --partial
> and --inplace?
> 
>> Since rsync doesn't set the timestamp of a file until it
>> is finished then any file left incomplete by an aborted rsync will have
>> the timestamp of the abort not the source file. Therefore it will be
>> newer than the source file and unless the source is updated rsync
>> --update will never complete the file.
> 
> Is there a solution to this problem and should we consider it a bug?
> I am encountering this, and it's a real problem for me if files are
> not completed.
> 
>> On 9/11/21 6:50 PM, hancooper via rsync wrote:
>>
>>> I am struggling to understand exactly what the rsync options --update and 
>>> --append-verify do.
>>> Doing info rsync gives
>>> -u, --update
>>> This forces rsync to skip any files which exist on the destina‐
>>> tion and have a modified time that is newer than the source
>>> file. (If an existing destination file has a modification time
>>> equal to the source file’s, it will be updated if the sizes are
>>> different.)
>>> --append-verify
>>> This works just like the --append option, but the existing data
>>> on the receiving side is included in the full-file checksum
>>> verification step, which will cause a file to be resent if the
>>> final verification step fails (rsync uses a normal, non-append‐
>>> ing --inplace transfer for the resend).
>>> I am using rsync to transfer directories recursively. There are times where 
>>> I have to stop the rsync transfer, and resume the transfer a few hours or 
>>> days later, without affecting the already transferred files at destination.
>>> I also got some files that return errors by rsync, such as
>>> rsync: read errors mapping 
>>> "/media/hc1/a1-chaos/amvib/IACL-2017-07-19T00:00:00-2017-07-19T23:59:59.mseed":
>>>  Input/output error (5)
>>> I would like to retry the transfer of these files, at a later time too, 
>>> without affecting the files that had been transferred successfully.
>>

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync options –update and –append-verify

[Kevin Korb via rsync]

--archive is all you really need.  I actually wish --archive was the
default because it is all most people need and with the exception of
writing to a FAT filesystem it is almost always needed.

--append is for very special cases and should only be used if you really
know you need it and why.  --append-verify only exists because people
seem to think they need --append and get annoyed that it corrupts their
files.

--update is sometimes helpful however it interacts badly with --partial
and --inplace.  Since rsync doesn't set the timestamp of a file until it
is finished then any file left incomplete by an aborted rsync will have
the timestamp of the abort not the source file.  Therefore it will be
newer than the source file and unless the source is updated rsync
--update will never complete the file.

On 9/11/21 6:50 PM, hancooper via rsync wrote:
> I am struggling to understand exactly what the rsync options --update and 
> --append-verify do.
> 
> Doing info rsync gives
> 
> -u, --update
>  This forces rsync to skip any files which exist on the destina‐
>  tion and have a modified time that is  newer  than  the  source
>  file.  (If an existing destination file has a modification time
>  equal to the source file’s, it will be updated if the sizes are
>  different.)
> 
> --append-verify
> This works just like the --append option, but the existing data
> on  the  receiving  side  is included in the full-file checksum
> verification step, which will cause a file to be resent if  the
> final verification step fails (rsync uses a normal, non-append‐
> ing --inplace transfer for the resend).
> 
> I am using rsync to transfer directories recursively. There are times where I 
> have to stop the rsync transfer, and resume the transfer a few hours or days 
> later, without affecting the already transferred files at destination.
> 
> I also got some files that return errors by rsync, such as
> 
> rsync: read errors mapping 
> "/media/hc1/a1-chaos/amvib/IACL-2017-07-19T00:00:00-2017-07-19T23:59:59.mseed":
>  Input/output error (5)
> 
> I would like to retry the transfer of these files, at a later time too, 
> without affecting the files that had been transferred successfully.
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Question about rsync -uav dir1/. dir2/.: possib to link?

[Kevin Korb via rsync]

Yes, cpio -l can be useful since cpio can easily operate on the output
from the very capable find command.

On 9/4/21 8:34 PM, Dan Stromberg wrote:
> 
> I was thinking --link-dest too.
> 
> Sometimes this can be done with cpio too; check out the -pdlv options.
> 
> On Sat, Sep 4, 2021 at 4:57 PM Kevin Korb via rsync
> mailto:rsync@lists.samba.org>> wrote:
> 
> Rsync does almost everything cp does but since it is designed to network
> it never got that feature.  I was thinking maybe --link-dest could be
> tortured into doing it but if it can I can't figure out how.  BTW, you
> have some pointless dots in there.
> 
> On 9/4/21 6:41 PM, L A Walsh via rsync wrote:
> > I noticed in looking at download dirs for a project, that
> > another mirror had "crept-in" for usage (where different mirrors
> > are stored under mirror-URL names). To copy over the diffs,
> > normally I'd do:
> >   rsync -uav dir1/. dir2/.
> > (where dir1="the new mirror that I'd switched
> > to by accident, and dir2=the original dir).
> >
> > The files were "smallish" so I just copied them, BUT I wass
> > wondering if there was an option similar to using 'cp' for
> > a dircopy, but instead of
> >   cp -a dr1 dr2
> > using:
> >   cp -al dr1 dr2
> >
> > to just hard-link over files from "dir1" to "dir2" (both
> > are on the same file system).
> >
> > I looked at (and tried) --link-dest=DIR
> > (hardlink to files in DIR when unchanged), but either I had the syntax
> > wrong, or didn't understand it as it didn't seem to do what I
> > wanted: cp'ing the new files in dir1 into the orig dir).
> >
> > Does rsync have an option to just "copy" over the new
> > files via a hardlink?
> >
> > Tnx!
> >
> >
> >
> >
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>         Kevin Korb                      Phone:    (407) 252-6853
>         Systems Administrator           Internet:
>         FutureQuest, Inc.               ke...@futurequest.net  (work)
>         Orlando, Florida                k...@sanitarium.net
> <mailto:k...@sanitarium.net> (personal)
>         Web page:                       https://sanitarium.net/
> <https://sanitarium.net/>
>         PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing
> list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> <https://lists.samba.org/mailman/listinfo/rsync>
> Before posting, read:
> http://www.catb.org/~esr/faqs/smart-questions.html
> <http://www.catb.org/~esr/faqs/smart-questions.html>
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Question about rsync -uav dir1/. dir2/.: possib to link?

[Kevin Korb via rsync]

Rsync does almost everything cp does but since it is designed to network
it never got that feature.  I was thinking maybe --link-dest could be
tortured into doing it but if it can I can't figure out how.  BTW, you
have some pointless dots in there.

On 9/4/21 6:41 PM, L A Walsh via rsync wrote:
> I noticed in looking at download dirs for a project, that
> another mirror had "crept-in" for usage (where different mirrors
> are stored under mirror-URL names). To copy over the diffs,
> normally I'd do:
>   rsync -uav dir1/. dir2/.
> (where dir1="the new mirror that I'd switched
> to by accident, and dir2=the original dir).
> 
> The files were "smallish" so I just copied them, BUT I wass
> wondering if there was an option similar to using 'cp' for
> a dircopy, but instead of
>   cp -a dr1 dr2
> using:
>   cp -al dr1 dr2
> 
> to just hard-link over files from "dir1" to "dir2" (both
> are on the same file system).
> 
> I looked at (and tried) --link-dest=DIR
> (hardlink to files in DIR when unchanged), but either I had the syntax
> wrong, or didn't understand it as it didn't seem to do what I
> wanted: cp'ing the new files in dir1 into the orig dir).
> 
> Does rsync have an option to just "copy" over the new
> files via a hardlink?
> 
> Tnx!
> 
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Speeding up rsync

[Kevin Korb via rsync]

Most of the speedup is due to ssh being bound to a single CPU.  You will
get a little more speed out of a single CPU system but the speedup is
unlikely to go past 2 in parallel.  Even with that you may end up adding
more latency via disk seeks to accomplish much.

On 8/24/21 5:12 PM, hancooper wrote:
> ‐‐‐ Original Message ‐‐‐
> On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync 
>  wrote:
> 
>> In my experience backing up multiple hosts to a single host can speed
>> things up. Especially if using rsync over ssh with multiple CPUs on the
>> single host. You would need to do some experimentation to determine the
>> best number for your hardware and network. Also, if you exceed that
>> number you can put more stress on the backup machine lessening the
>> impact to the hosts being backed up.
>>
>> On 8/24/21 3:58 PM, hancooper via rsync wrote:
>>
>>> It makes sense to we to run multiple rsync commands to speed up a backup.
>>> At work, some have argued that if I sync the data all to the same host,
>>> there is no advantage
>>> in parallelization.   Whether you sync 3x 1G in parallel or 1x 3G makes
>>> no noticeable difference.
>>> Speed in general does not increase by increasing the number of sync
>>> processes.
>>> However, if one want to sync to multiple hosts, the effort may be
>>> worthwhile.
>>> What is the current advice about speeding up data transfer ?
>>
>> --
>> Kevin Korb Phone: (407) 252-6853
> 
> 
> Hi Kevin.  What about a single host to another with multiple CPU's ?
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Speeding up rsync

[Kevin Korb via rsync]

In my experience backing up multiple hosts to a single host can speed
things up.  Especially if using rsync over ssh with multiple CPUs on the
single host.  You would need to do some experimentation to determine the
best number for your hardware and network.  Also, if you exceed that
number you can put more stress on the backup machine lessening the
impact to the hosts being backed up.

On 8/24/21 3:58 PM, hancooper via rsync wrote:
> It makes sense to we to run multiple rsync commands to speed up a backup.
> 
> At work, some have argued that if I sync the data all to the same host,
> there is no advantage
> in parallelization.   Whether you sync 3x 1G in parallel or 1x 3G makes
> no noticeable difference.
> 
> Speed in general does not increase by increasing the number of sync
> processes.
> 
> However, if one want to sync to multiple hosts, the effort may be
> worthwhile.
> 
> What is the current advice about speeding up data transfer ?
> 
> 
> 
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: config

[Kevin Korb via rsync]

Rsync by default displays nothing.  There are more than one options that
tell it to display the files it is touching.  There are other (and
duplicate) options that tell it to show everything.  You didn't say what
options you are using so we have no idea.  Except that you aren't using
--itemize-changes which is the only option that will tell you why it is
saying what it is saying.

On 6/25/21 9:48 AM, Gerhard Obermayr via rsync wrote:
> Hello everyone,
> 
> Briefly to my problem - or wish: By lucky coincidence I came into
> possession of an HP Proliant.
> My junior installed OpenMediaVault for me.
> I've been using it as a daily backup for my TeraStation ever since.
> Rsync is set up for this.
> Every day I get two emails from this backup, in which the complete
> directory is listed that is due to be backed up.
> But that's pretty confusing, because that's currently more than 8000
> lines in total.
> And why? Because, among other things, I back up my directories with the
> photos from one NAS to another.
> And that's quite a lot of directories from the last 20 years ...
> In between, individual files appear in individual directories that have
> changed and have been saved as a result.
> Isn't it possible that only those files appear in this mail that were
> actually backed up because of the change?
> That would be easier and clearer ...
> Maybe there is already a solution - an additional option button - *to
> only display the copied files in the report* ...
> 
> What can i do? What parameter work right? Is there an option for that?
> 
> -- 
> Liebe Grüße aus Haag - und Xund bleim!
> 
> Gerhard Obermayr
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Permission denied errors with --append-verify run as non-root user

[Kevin Korb via rsync]

Short answer: Don't use --append[-verify]

If you aren't syncing files that aren't tagged with 'chattr +a' you
don't want --append.  --append-verify is essentially "I think --append
is a good idea to use on general files but rsync keeps corrupting
stuff!" but it still allows files to be out of sync since it ignores any
change to a file that doesn't make it a bigger file.  Essentially it is
for things like append only filesytem images and non-rotated log files.

All of that being said, rsync by default only transfers what is
different between the source and the target.  If a file has gotten
bigger only by adding stuff at the end rsync will figure that out and
transfer only the stuff that matters.  Any performance improvement you
are getting from --append-verify is either because it is ignoring file
changes that didn't result in a bigger file and because it causes file
updates to work like --inplace (which is what is causing your problem).

On 6/4/21 11:50 AM, Knight, Dave via rsync wrote:
> I noticed that my latest  "syncHome" script log scontained a number of
> unexpected "Permission denied" errors for some modified files.  The
> cause seems to be that those files were read-only at the target.  
> 
> The syncHome script is used to maintain a partial mirror of my home
> directory tree on another host.
> 
> Further investigation and some experiments led me to suspect the
> --append-verify functionality.  Specifically, those files were not
> rsync'd when the --append-verify rsync flag was specified but were
> properly rsync'd when --append-verify was NOT specified.  
> 
> Other rsync scripts that use --append-verify and are run as root seem
> not to exhibit this behavior.
> 
> FYI, I downloaded and built the "latest" version (rsync  version 3.2.3
>  protocol version 31) and with this version of rsync, the modified files
> are silently NOT rsync'd for both root and non-root users when
> --append-verify is specified.  When --append-verify is not specified,
> the updated files are rsync'd correctly.
> 
> That seems to me to be a serious bug, IMHO, because I "LIKE' to use
> --append-verify to ensure that large files are copied entirely without
> having to resync the entire file if there is an error.  If the
> --append-verify option */silently fails/* to rsync some updated files
> that */should/* be rsync'd, I cannot use --append-verify!
> 
> Please advise whether I am misunderstanding the purpose of, or misusing
> the  --append-verify flag or if this is a bug.   It is really easy to
> reproduce, BTW... 
> 
> Dave Knight 
> Radford, VA
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync: [generator] failed to set permissions : Operation not supported (95)

[Kevin Korb via rsync]

Those aren't really rsync errors they are just rsync telling you what
the kernel told it.  The only thing I see wrong in your paste is that
you didn't use a trailing / on the source+target.  Those do mean
something to rsync.  The only thing I see wrong in the instructions is
the use of dd to make a file of a specific size.  The truncate command
does that much better.

Anyway, check the mount options on the file system (use the mount
command it will show them) and the options on the filesystem itself with
tune2fs -l.  My guess is that either ACLs or XATTRs are not enabled and
they probably are in use since the instructions told you to tell rsync
to copy those.  You can also try running rsync without the -A and -X to
confirm.

On 4/6/21 9:50 AM, Mario Marietto via rsync wrote:
> Hello.
> 
> I want to virtualize correctly Android 10 on top of my Jetson nano
> (arm64) using qemu and kvm on ubuntu 18.04. This is the tutorial that
> I'm following :
> 
> https://github.com/antmicro/kvm-aosp-jetson-nano
> 
> 
> everything went good until this command :
> 
> |sudo rsync -avxHAX system-r{o,w}/
> 
> |
> 
> |something is not good because I get a lot of errors when I transfer the
> files and permissions from the source to the destination path (both are
> on the same disk and on the same ext4 partition. You can see the full
> log with the errors here :
> 
> |
> 
> https://pastebin.ubuntu.com/p/W9GjPCt8G4/
> 
> 
> the consequence of these errors is that when I try to emulate android
> with qemu like this :
> 
> |qemu-system-aarch64 \ -enable-kvm \ -smp 4 \ -m 2048 \ -cpu host \ -M
> virt \ -device virtio-gpu-pci \ -device usb-ehci \ -device usb-kbd \
> -device virtio-tablet-pci \ -usb \ -serial stdio \ -display sdl,gl=on \
> -kernel aosp/Image \ -initrd aosp/ramdisk.img \ -drive
> index=0,if=none,id=system,file=aosp/system.img \ -device
> virtio-blk-pci,drive=system \ -drive
> index=1,if=none,id=vendor,file=aosp/vendor.img \ -device
> virtio-blk-pci,drive=vendor \ -drive
> index=2,if=none,id=userdata,file=aosp/userdata.img \ -device
> virtio-blk-pci,drive=userdata \ -full-screen \ -append
> "console=ttyAMA0,38400 earlycon=pl011,0x0900 drm.debug=0x0 rootwait
> rootdelay=5 androidboot.hardware=ranchu androidboot.selinux=permissive
> security=selinux selinux=1 androidboot.qemu.hw.mainkeys=0
> androidboot.lcd.density=160" |
> 
> 
> this is the error that I get :
> 
> [ 2.532754] init: init first stage started!
> [ 2.535936] init: [libfs_mgr]ReadFstabFromDt(): failed to read fstab
> from dt
> [ 2.540632] init: [libfs_mgr]ReadDefaultFstab(): failed to find device
> default fstab
> [ 2.546246] init: Failed to fstab for first stage mount
> [ 2.549616] init: Using Android DT directory
> /proc/device-tree/firmware/android/
> [ 2.555116] init: [libfs_mgr]ReadDefaultFstab(): failed to find device
> default fstab
> [ 2.560762] init: First stage mount skipped (missing/incompatible/empty
> fstab in device tree)
> [ 2.566906] init: Skipped setting INIT_AVB_VERSION (not in recovery mode)
> [ 2.571227] init: execv("/system/bin/init") failed: No such file or
> directory
> [ 2.593768] init: #00 pc 000e90a0 /init
> [ 2.599958] reboot: Restarting system with command 'bootloader'
> 
> -- 
> Mario.
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync support in authprogs - feedback requested

[Kevin Korb via rsync]

You should both look into rrsync.  It comes with rsync and is designed
to do exactly this.  Unfortunately some Linux distros are maintained by
insane people who install rrsync as if it was documentation (compressed
and not executable) instead of a helper script which is what it is.

On 2/18/21 10:28 AM, Karl O. Pinc via rsync wrote:
> On Wed, 17 Feb 2021 21:52:06 -0800
> Bri Hatch via rsync  wrote:
> 
>> I recently added initial rsync support to authprogs.
> 
>> I'd be very interested in feedback 
> 
> For some 15 years+ (?) I've had a /root/.ssh/authorized keys line
> that starts with:
> 
> "no-pty,no-agent-forwarding,no-port-forwarding,no-user-rc,no-X11-forwarding,command="rsync
>  --server --daemon ."
> 
> Occasionally I frob the ssh restrictions as new ones are
> introduced.
> 
> The remote end uses rsync to backup (with --link-dest) the
> entire file system.  The idea (iirc) was to restrict
> the given key so that it would only run rsync.
> And I think this also forces the local end to use
> /etc/rsyncd.conf, where there's an additional layer
> of security via a secrets file and read-only can
> be set to provide some control.
> 
> The remote end always runs rsync -- the direction of 
> transfer is static, per-host-pair, but can be either
> in or out. (Push or pull backups.) The above authorized_keys 
> line does not enforce direction, which might be useful.
> 
> I only rarely think about tweaking the authorized_keys line, 
> and the rsync options used haven't changed since I got them to work.
> Without really thinking about it it seems that your
> authprogs development might be useful.  
> 
> My purpose with this email is to let you do all the 
> thinking and tell me of all the wonderful utility
> your authprogs work can provides, either now or
> in the future.  ;-)  Or at least give you some
> background in case you want to develop in a direction
> that you think would helpful to me.  If something comes
> of this I might even turn my brain on again and
> modify my systems.  :)
> 
> Regards,
> 
> Karl 
> Free Software:  "You don't pay back, you pay forward."
>  -- Robert A. Heinlein
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync failed to set times (on temporary files)

[Kevin Korb via rsync]

I am not sure if I have missed information here but...  Rsync only sets
the mtime on files that it completely transferred.  The back-dating of a
file is rsync's guarantee that that file was completely transferred and
verified.

On 2/2/21 9:03 PM, Leon Vanderploeg via rsync wrote:
> Windows Cygwin rsync is version 3.1.1
> 
> Ubuntu version is 3.1.2
> 
> Ubuntu file system is ext4
> 
> Rsync command main options -rtlg0D ( see attachment for full view)
> 
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Preserve Windows Symlinks

[Kevin Korb via rsync]

Unison doesn't use rsync it only uses librsync (rsync's file diffing
algorithm).  Also, unison has a native Windows version so no *nix
translation is needed.

On 12/29/20 12:48 PM, Edwin Bradford wrote:
> Sorry, I should have realized, that makes complete sense in hindsight.
> I deleted the original source volume symlink and restored from the
> external volume symlink and the same problem exists... Windows doesn't
> recognise it as a symlink or appear to know what kind of file it is.
> 
> Previously I was using Unison File Sync which itself uses rsync and
> doesn't support preserving symlinks between Windows and Unix so I'm
> wondering if it's just not possible in which case maybe I'll just have
> to live with following symlinks --copy-links.
> 
> 
> 
> + + + + + + + + + + + + + + + + + +
> 
> Email: edwinbradf...@gmail.com
> 
> Tel.: +44 (0)7981 873 771
> 
> + + + + + + + + + + + + + + + + + +
> 
> 
> On Tue, 29 Dec 2020 at 16:39, Kevin Korb  wrote:
>>
>> Not quite.  I meant rsync a symlink then delete the source symlink and
>> use rsync to restore it from what rsync made.  If the target of the
>> symlink doesn't exist on the other system (in the same place) then the
>> symlink shouldn't work on the remote but rsync can still backup it up
>> and restore it.  At least that is how it works in *nix.
>>
>> On 12/29/20 11:34 AM, Edwin Bradford wrote:
>>> Yes I found --checksum is much slower so following your feedback I'll do
>>> some more reading to see if I really need it thanks. Assuming I
>>> understand correctly I did the following test and also replaced absolute
>>> symlink paths with relative symlink paths which worked this time:
>>>
>>> 1. I deleted and recreated the Windows symlinks on the source volume
>>> using relative paths.
>>> 2. I deleted and recreated the same symlinks directly on the destination
>>> volume.
>>> 3. I ran the same rsync command with the --archive flag (preserve symlinks).
>>>
>>> The result was the symlinks on the destination volume were preserved or
>>> at least not destroyed by the subsequent sync. If this test is what you
>>> intended then it shows rsync can preserve the symlink between NTFS
>>> volumes but I would have to manually recreate the symlinks on the
>>> destination NTFS volume once.
>>>
>>> Have I understood correctly?
>>>
>>>
>>> On Tue, 29 Dec 2020 at 13:53, Kevin Korb via rsync
>>> mailto:rsync@lists.samba.org>> wrote:
>>>
>>> If you delete the link then restore it does it start working again when
>>> in the correct place?
>>>
>>> Also, don't use --checksum unless you are really certain you understand
>>> how terribly slow it is and how it doesn't actually accomplish much of
>>> anything (certainly not any kind of data verification).
>>>
>>> On 12/29/20 7:29 AM, Edwin Bradford via rsync wrote:
>>> > Is it possible to preserve Windows symlinks when backing up from
>>> NTFS to
>>> > NTFS volumes or any other for that matter?
>>> >
>>> > I am running rsync in Ubuntu in Windows Subsytem for Linux on
>>> Windows 10
>>> > backing up a local NTFS volume to an external NTFS volume (and
>>> also to a
>>> > remote Unix server).
>>> >
>>> > The source NTFS volume has symlinks created in Windows using:
>>> >
>>> > mklink the\link the\file
>>> > mklink /d the\link the\directory
>>> >
>>> > I am running rsync within a long shell script with the command:
>>> >
>>> > rsync
>>> > --verbose --archive --checksum --itemize-changes --human-readable
>>> > --delete --delete-excluded --partial --progress --stats
>>> > --exclude-from='ignore.txt'
>>> > source/ destination
>>> >
>>> > The symlinks are copied to the external NTFS partition and have
>>> the same
>>> > name but no functionality. Windows doesn't recognise them as symlinks,
>>> > doesn't know what to do with them and there is no apparent difference
>>> > between symlinked directories and files.
>>> >
>>> > I tried creating the symlinks with relative paths on the source volume
>>> > but rsync generated an error and refused to complete. Using
>>> --copy-links
>>> > inste

Re: Preserve Windows Symlinks

[Kevin Korb via rsync]

Not quite.  I meant rsync a symlink then delete the source symlink and
use rsync to restore it from what rsync made.  If the target of the
symlink doesn't exist on the other system (in the same place) then the
symlink shouldn't work on the remote but rsync can still backup it up
and restore it.  At least that is how it works in *nix.

On 12/29/20 11:34 AM, Edwin Bradford wrote:
> Yes I found --checksum is much slower so following your feedback I'll do
> some more reading to see if I really need it thanks. Assuming I
> understand correctly I did the following test and also replaced absolute
> symlink paths with relative symlink paths which worked this time:
> 
> 1. I deleted and recreated the Windows symlinks on the source volume
> using relative paths.
> 2. I deleted and recreated the same symlinks directly on the destination
> volume.
> 3. I ran the same rsync command with the --archive flag (preserve symlinks).
> 
> The result was the symlinks on the destination volume were preserved or
> at least not destroyed by the subsequent sync. If this test is what you
> intended then it shows rsync can preserve the symlink between NTFS
> volumes but I would have to manually recreate the symlinks on the
> destination NTFS volume once.
> 
> Have I understood correctly? 
>  
> 
> On Tue, 29 Dec 2020 at 13:53, Kevin Korb via rsync
> mailto:rsync@lists.samba.org>> wrote:
> 
> If you delete the link then restore it does it start working again when
> in the correct place?
> 
> Also, don't use --checksum unless you are really certain you understand
> how terribly slow it is and how it doesn't actually accomplish much of
> anything (certainly not any kind of data verification).
> 
> On 12/29/20 7:29 AM, Edwin Bradford via rsync wrote:
> > Is it possible to preserve Windows symlinks when backing up from
> NTFS to
> > NTFS volumes or any other for that matter?
> >
> > I am running rsync in Ubuntu in Windows Subsytem for Linux on
> Windows 10
> > backing up a local NTFS volume to an external NTFS volume (and
> also to a
> > remote Unix server).
> >
> > The source NTFS volume has symlinks created in Windows using:
> >
> >     mklink the\link the\file
> >     mklink /d the\link the\directory
> >
> > I am running rsync within a long shell script with the command:
> >
> >     rsync
> >     --verbose --archive --checksum --itemize-changes --human-readable
> >     --delete --delete-excluded --partial --progress --stats
> >     --exclude-from='ignore.txt'
> >     source/ destination
> >
> > The symlinks are copied to the external NTFS partition and have
> the same
> > name but no functionality. Windows doesn't recognise them as symlinks,
> > doesn't know what to do with them and there is no apparent difference
> > between symlinked directories and files.
> >
> > I tried creating the symlinks with relative paths on the source volume
> > but rsync generated an error and refused to complete. Using
> --copy-links
> > instead of --links (implicit in --archive) gives the expected
> behaviour
> > but I would prefer to preserve symlinks rather than replace them with
> > their linked contents.
> >
> > At this point I'm not sure if it's a limitation of rsync and NTFS
> > compatibility or whether I'm missing something.
> >
> > + + + + + + + + + + + + + + + + + +
> >
> > Email: edwinbradf...@gmail.com <mailto:edwinbradf...@gmail.com>
> <mailto:edwinbradf...@gmail.com <mailto:edwinbradf...@gmail.com>>
> >
> > Tel.: +44 (0)7981 873 771
> >
> > + + + + + + + + + + + + + + + + + +
> >
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>         Kevin Korb                      Phone:    (407) 252-6853
>         Systems Administrator           Internet:
>         FutureQuest, Inc.               ke...@futurequest.net  (work)
>         Orlando, Florida                k...@sanitarium.net
> <mailto:k...@sanitarium.net> (personal)
>         Web page:                       https://sanitarium.net/
> <https://sanitarium.net/>
>         PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing
> list.
> To unsubscribe or change options:
> https://lists.samba.org/mail

Re: Preserve Windows Symlinks

[Kevin Korb via rsync]

If you delete the link then restore it does it start working again when
in the correct place?

Also, don't use --checksum unless you are really certain you understand
how terribly slow it is and how it doesn't actually accomplish much of
anything (certainly not any kind of data verification).

On 12/29/20 7:29 AM, Edwin Bradford via rsync wrote:
> Is it possible to preserve Windows symlinks when backing up from NTFS to
> NTFS volumes or any other for that matter?
> 
> I am running rsync in Ubuntu in Windows Subsytem for Linux on Windows 10
> backing up a local NTFS volume to an external NTFS volume (and also to a
> remote Unix server).
> 
> The source NTFS volume has symlinks created in Windows using:
> 
>     mklink the\link the\file
>     mklink /d the\link the\directory
> 
> I am running rsync within a long shell script with the command:
> 
>     rsync
>     --verbose --archive --checksum --itemize-changes --human-readable
>     --delete --delete-excluded --partial --progress --stats
>     --exclude-from='ignore.txt'
>     source/ destination
> 
> The symlinks are copied to the external NTFS partition and have the same
> name but no functionality. Windows doesn't recognise them as symlinks,
> doesn't know what to do with them and there is no apparent difference
> between symlinked directories and files.
> 
> I tried creating the symlinks with relative paths on the source volume
> but rsync generated an error and refused to complete. Using --copy-links
> instead of --links (implicit in --archive) gives the expected behaviour
> but I would prefer to preserve symlinks rather than replace them with
> their linked contents.
> 
> At this point I'm not sure if it's a limitation of rsync and NTFS
> compatibility or whether I'm missing something.
> 
> + + + + + + + + + + + + + + + + + +
> 
> Email: edwinbradf...@gmail.com 
> 
> Tel.: +44 (0)7981 873 771
> 
> + + + + + + + + + + + + + + + + + +
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync checksum issue

[Kevin Korb via rsync]

Either that or you have a hardware problem causing the checksum routine
to return the wrong result.

On 12/15/20 11:47 PM, Dan Stromberg via rsync wrote:
> 
> On Tue, Dec 15, 2020 at 3:25 AM Laurent B via rsync
> mailto:rsync@lists.samba.org>> wrote:
> 
> Dear all,
> 
> I'm encountering a problem with one of my backup. For some files, the
> checksum calculation is failing leading to the following error :
> 
> It sounds a little like a bug, but perhaps if you share the command
> you're using for the backup?
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: A strange problem with my daily backups performed via rsync

[Kevin Korb via rsync]

A few things here...
What filesystem is on /mnt/sony?
Add a --itemize-changes to see what it says is going on.
There really isn't a good reason to exclude lost+found. It should always
be empty and take up no space.  If there is something in it that is a
sign that there is a problem with your filesystem and you may as well
have rsync bring it to your attention.

On 11/2/20 3:30 AM, Manish Jain via rsync wrote:
> 
> Hi,
> 
> I am facing a strange situation with my daily backups performed via
> rsync. I primarily use Manjaro KDE Linux (LTS kernel), but also have
> FreeBSD and Windows 10 bare-metal installations.
> 
> I have an all-OS-writable ext2 partition /dev/sda2 mounted at /mnt/wall
> 
> My USB backup device is a Sony SSD mounted at /mnt/sony
> 
> This is the relevant line from my script which backs up /mnt/wall :
> 
> rsync -aAHXv --delete \
> --exclude "lost+found" --exclude "System Volume Information" \
> /mnt/wall /mnt/sony/
> 
> Whenever the line above goes into action, it recursively deletes
> /mnt/sony/wall and then copies /mnt/wall all over again.
> 
> Any ideas why this might be happening ? Since /mnt/wall is 13GB, it
> costs me time and SSD-wear to have /mnt/sony/wall recreated all over
> again every time I do a backup.
> 
> 
> Thanks,
> Manish Jain
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: pattern geeks?

[Kevin Korb via rsync]

You need to include the dirs leading to there...
--include=httpd/, --include=httpd/conf.d/
Otherwise it will never look inside to see the cluster.d

(I am assuming you are right about lsyncd using the same patterns as rsync).


On 10/21/20 9:01 AM, lejeczek via rsync wrote:
> hi guys,
> 
> I'm trying lsyncd with some patterns which should be just
> rsync's own patterns and I have these:
> 
> rsync = {  _extra = {
>     "--include=samba**",
>     "--include=postfix**",
>     "--include=openvpn**",
>     "--include=strongswan**",
>     "--include=httpd/conf.d/cluster.d**",
>     "--exclude=**",
>     }
> }
> 
> in my lsyncd.conf
> The rest of the config says the source dir, as well as the
> target, is '/etc'
> Now, stuff gets excluded and things as listed get included
> and go into target except for:
> "httpd/conf.d/cluster.d**"
> 
> Obviously it isn't as simple as I presumed.
> How to make such sub-folder path included would you know?
> many thanks, L.
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Syncing multiple sub-directories to one directory

[Kevin Korb via rsync]

Interesting idea.  It isn't something I have ever wanted to do.  BTW, if
your find is recent use + instead of \;.  + replaces {} with however
many entries fit in the command line length limit instead of running
individual rsync processes for each entry.

On 9/29/20 7:46 PM, Rob Campbell via rsync wrote:
> Thanks for your help.  What you provided didn't work for me because that
> still placed things in subdirectories.
> 
> I figured it out.  This puts it all in the images directory.
> 
> find /my/phone/root/dir/ \( -path '*Duo*' -o -path '*DCIM*' -o -path
> '*Pictures*' -o -path '*Camera*' -o -path "*Download*" -o -path
> "*textgram*" -o -path "*WhatsApp*" ! -path '*.textgram*' ! -path
> '*.thumbnails*' \) -type f \( -iname '*.jp*g' -o -iname '*.png' -o
> -iname '*.dng' -o -iname '*.raw' -o -iname '*.raw' \) -exec rsync -uz
> --progress "{}" /my/backup/directory/for/images/ \;
> 
> ~
> When you are principled, set standards and stick to them some people will 
> fall out of your life; let them!
> 
> 
> On Tue, Sep 29, 2020 at 1:33 PM Wayne Davison  > wrote:
> 
> On Tue, Sep 29, 2020 at 7:38 AM Rob Campbell wrote:
> 
> I would like to sync many subdirectories into one directory with
> no subdirectories. I've tried
> 
> rsync -rv --progress --include '*.jp*g' --include '*.png'
> --include '*.dng' --include '*.raw' --include '*.nef' --include
> 'Duo' --include 'DCIM' --include 'WhatsApp' --exclude '*'
> /my/phone/root/dir/ /my/backup/directory/for/images/
> 
> 
> You didn't specify what isn't working as you expect. That command
> works fine for the 3 listed dir names as long as you either don't
> have matching files inside /my/phone/root/dir (or as long as it's ok
> to also copy those files) and as long as there aren't subdirs or
> files named the same as the 3 top dirs you included. The latter can
> be fixed by anchoring your dir names and making them only match a
> dir (e.g. --include '/Duo/').  If you want to have the copy avoid
> files in the top dir you can either change all the file-based
> includes to have a "*/" prefix (such as "--include '*/*.png' ...")
> or you can change the dir includes into args and exclude '*/*'
> instead of "*".  The last option would look like this (I also tossed
> in -i):
> 
> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
> '*.dng' --include '*.raw' --include '*.nef' --exclude '*/*'
> /my/phone/root/dir/{Duo,DCIM,WhatsApp} /my/backup/directory/for/images/
> 
> That assumes you've got bash to do the brace expansion, but you
> could change that into 3 arg paths if you need to. You may also want
> to add --del if you want rsync to delete inside the 3 listed dirs.
> 
> I'm not sure why you listed an extra command with "NewDir" when it's
> not mentioned in the first command.  If that is an indication that
> you really want to copy all dirs under root/dir (not just the 3
> named dirs in the first command) then you could use an include of
> "/*/" to match any dir in the root of the transfer, like this:
> 
> rsync -riv --progress --include '*.jp*g' --include '*.png' --include
> '*.dng' --include '*.raw' --include '*.nef' --include '/*/'
> --exclude '*' /my/phone/root/dir/ /my/backup/directory/for/images/
> 
> One last suggestion, I like to make the args shorter by using -f
> (filter) commands, so an include example is -f '+ *.png' and an
> exclude example is -f '- /*/' (those are identical to the equivalent
> include/exclude args, so that's just personal preference).
> 
> ..wayne..
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Back-up differences. Raspberry Pi and Fedora

[Kevin Korb via rsync]

On 9/29/20 7:07 AM, joe--- via rsync wrote:
> unknown module Linux1
This would imply that "Linux1" is not defined in the rsyncd.conf file on
the backup server.  Of course since it is a NAS appliance I don't know
if you have any access to the config file.

1 check you could run is 'rsync root@server::'.  This would list the
modules the server is configured with (assuming listing is allowed).


-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Meaning of "failed verification -- update retained (will try again)."

[Kevin Korb via rsync]

Mostly it means that the file rsync ended up with on the target end
didn't match the file on the source it started with.  This is mostly
caused by the file being modified on the source while rsync is copying
it but it can also be a memory corruption problem.

On 8/11/20 1:13 PM, Peng Yu via rsync wrote:
> Hi,
> 
> I see some warnings like the following. Could anybody explains what
> they mean? Thanks.
> 
> 19/31274477.pdf
> 257,169,119   0%1.26MB/s0:03:15 (xfr#121,
> to-chk=848309/1043298)WARNING: 18/32281577.pdf failed verification --
> update retained (will try again).
> WARNING: 19/28879866.pdf failed verification -- update retained (will
> try again).
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync Daemon Remote Pull

[Kevin Korb via rsync]

user@x.x.x.x:port/ModuleName is not correct syntax.  You may have
something in your shell config translating that for you.  Correct syntax
is --port  and user@host::module

On 6/25/20 6:01 PM, Chandrasekar Natarajan via rsync wrote:
> Hi,
> 
> I am trying to pull folders from a windows remote machine using daemon
> without SSH.
>  Below is the command that used:
> rsync -vrtz --delete user@x.x.x.x:port/ModuleName '/cygdrive/d/backup/'
> 
> No issue running the same command in Command Prompt.While executing this
> command via c#.net Process startup, facing some issue.
> 
> The exception is ': Unexpected remote arg: user@x.x.x.x:port/module
> rsync error: syntax or usage error (code 1) at main.c(1361) [sender=3.1.2]
> 
> 
> Kindly help me to resolve this.
> 
> -- 
> Thanks,
> 
> *Chandrasekar N*
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: High memory usage - any way around it other than splitting jobs?

[Kevin Korb via rsync]

Unfortunately the hard links are the problem.  In order to keep them
straight rsync has to remember the details of every file it finds with a
link count >1 making it grow and grow.  Of course without -H rsync will
end up duplicating them.

On 6/25/20 10:30 AM, Andy Smith via rsync wrote:
> Hi,
> 
> I have a virtual machine with 2G of memory. On this VM there is a
> directory tree with 33.3 million files in it. When attempting to
> rsync (rsync -PSHav --delete /source /dest) this tree from one
> directory to another on the same host, rsync uses all the memory and
> is killed by oom-killer.
> 
> This host is Debian oldstable so has
> 
> $ rsync --version
> rsync  version 3.1.2  protocol version 31
> 
> The normal operation of this VM does not require more than 2G of
> memory, but I doubled it to 4G anyway. Unfortunately rsync still
> uses all the memory and is killed.
> 
> Most advice I can find on decreasing rsync memory usage advises to
> split the job up into batches. By issuing one rsync for each
> directory within /source I was able to make this work.
> 
> The interesting thing is though, the split of file numbers between
> sub-directories is very uneven with the majority of them (31.5
> million of the 33.3 million) being in just one of the sub-directory
> trees. I am kind of surprised that rsync has such a problem going
> just that little bit further with the last 2 million. Is there any
> scope for improvement with the incremental recursion code?
> 
> If I upgraded the version of rsync could I expect this to work any
> better?
> 
> I could also give the host a massive swap file. It currently has
> just 1G of swap, which all gets used in the failure case. I could
> add more but I fear that the job will go so slow it will not
> complete in a reasonable time.
> 
> I don't know if the -H option is causing extra memory usage here;
> unfortunately it is necessary as there are hardlinks in there.
> 
> Some years old advice says to disable incremental recursion with
> --no-i-r. As incremental recursion was added to reduce memory usage
> this seems counter-intuitive to me, but this advice is all over the
> Internet…
> 
> These are all things I will investigate before settling for the
> "split into multiple jobs" approach; just wondered if anyone has any
> shortcuts for me.
> 
> Thanks,
> Andy
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync sometimes decides to delete all data

[Kevin Korb via rsync]

The only time I have seen something like that was when I was using rsync
to backup a snapshot and something went wrong in the snapshot creation
resulting in me rsyncing an empty dir instead of a mounted filesystem.
Does the --stats output imply that rsync saw an empty dir?

On 6/9/20 2:46 AM, freebsd--- via rsync wrote:
> Hello,
> 
> I run into some really strange case I haven't seen so far over the years.
> I syncing a lot of different macines, raspberry pis to a backup server
> nightly.
> Lately one of them misbehaves.
> 
> Server rsync:  rsync  version 3.1.2  protocol version 31
> Client rsync:  rsync  version 3.1.3  protocol version 31
> 
> Command used:
> rsync --numeric-ids --delete --delete-excluded --delete-after -f "merge
> ${BUFFER}" --stats  -e "${SSH_CMD}" root@${BACKUP_HOST}:/ $BACKUPROOT/
> 
> Sometimes when I run this script from the server it just decides to
> delete ALL files what it already have and sync is done.
> 
> I run it again couple of minutes after it syncs everything.
> 
> What can cause such a behavior? I can of course remove the delete
> directives but then the local copy will become messy over time and this
> perfectly works with all other machines.
> 
> It is not even that it cannot connect with rsync timeout and then it
> decides to delete the data. It can connect then decides to delete all
> data and done.
> 
> Rsync is run by root in any case so it's not a privilege problem.
> 
> Any ideas?
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to diff two directories?

[Kevin Korb via rsync]

On 5/6/20 11:49 AM, Peng Yu wrote:
> On 5/5/20, Kevin Korb via rsync  wrote:
>> Rsync != diff.
> 
> What do you mean? I only need to know what files are not the same, but
> I don't need to know what the differences are with the files.

See diff -q

> 
>> However, if that command lists a file something is different about it.
>> add --itemize-changes to find out what.
> 
> I see this output. What does ">f.s 2.txt" mean? Thanks.

There is a whole section in the man page that explains what all of the
itemize output means.  This means the file size is different.

> 
> $ rsync -avun --delete --itemize-changes /tmp/tmp.GH6z0oN8rn/a/
> /tmp/tmp.GH6z0oN8rn/b
> sending incremental file list
>> f.s 2.txt
> 
> sent 73 bytes  received 19 bytes  184.00 bytes/sec
> total size is 4  speedup is 0.04 (DRY RUN)
> 
> 
> ==> /tmp/tmp.GH6z0oN8rn/a/1.txt <==
> 1
> 
> ==> /tmp/tmp.GH6z0oN8rn/a/2.txt <==
> 2
> 
> ==> /tmp/tmp.GH6z0oN8rn/b/1.txt <==
> 1
> 
> ==> /tmp/tmp.GH6z0oN8rn/b/2.txt <==
> 22
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to diff two directories?

[Kevin Korb via rsync]

Rsync != diff.

However, if that command lists a file something is different about it.
add --itemize-changes to find out what.

On 5/5/20 8:47 PM, Peng Yu via rsync wrote:
> Hi,
> 
> I use this command. But it also shows the files that are the same. Is
> there an option that can be specified so that only the differences are
> printed? Thanks.
> 
> rsync -avun --delete src dir
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Would you expect --perms -M--fake-super to set the file mode to the original one?

[Kevin Korb via rsync]

I don't believe it is possible.  I think the misunderstanding stems from
the fact that the permissions are even stored in the xattr.  They don't
need to be there but they may as well be.  They don't take much space.
The real question would be when rsync reads the file to restore it and
the file perms are different than the ones in the xattr which set does
it use?

On 3/16/20 10:01 AM, Dimitrios Apostolou via rsync wrote:
> Thanks. This is a bit counter-intuitive to me. So how would you tell
> rsync to store the original permissions in the xattr, but do not touch
> the real file mode?
> 
> On Thursday, March 12, 2020 6:26:18 PM CET, Kevin Korb via rsync wrote:
>> I would expect that the sending rsync would only send the perms provided
>> modified by the --chmod.  I wouldn't expect the receiver to even know
>> the other permissions.
>>
>> On 3/12/20 1:23 PM, Dimitrios Apostolou via rsync wrote:
>>> Thank you for the feedback, I'm glad to see that different people see
>>> the issue
>>> differently. As a followup question, what would you expect this to do:
>>>
>>> rsync --perms --chmod g+rX -M--fake-super src dst
>>>
>>> I would expect it to store the original permissions in the xattr,
>>> while ...
>>
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Would you expect --perms -M--fake-super to set the file mode to the original one?

[Kevin Korb via rsync]

I would expect that the sending rsync would only send the perms provided
modified by the --chmod.  I wouldn't expect the receiver to even know
the other permissions.

On 3/12/20 1:23 PM, Dimitrios Apostolou via rsync wrote:
> Thank you for the feedback, I'm glad to see that different people see
> the issue
> differently. As a followup question, what would you expect this to do:
> 
> rsync --perms --chmod g+rX -M--fake-super src dst
> 
> I would expect it to store the original permissions in the xattr, while
> modifying the real file mode according to the chmod.
> 
> On Thursday, March 12, 2020 6:06:34 PM CET, Kevin Korb via rsync wrote:
>> Permissions don't require super.  Any place where permissions can't be
>> stored certainly can't handle xattrs either.  So, I wouldn't expect
>> --fake-super to affect --perms at all.
>>
>> On 3/12/20 12:46 PM, Dimitrios Apostolou via rsync wrote:
>>> rsync --perms -M--fake-super src dst
>>>
>>> For me, this command means that rsync should save the original perms
>>> in the
>>> xattr, and leave the real file mode to the umask default. Currently
>>> it also
>>> modifies the real file mode, and there is no way to store something
>>> different ...
>>
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Would you expect --perms -M--fake-super to set the file mode to the original one?

[Kevin Korb via rsync]

Permissions don't require super.  Any place where permissions can't be
stored certainly can't handle xattrs either.  So, I wouldn't expect
--fake-super to affect --perms at all.

On 3/12/20 12:46 PM, Dimitrios Apostolou via rsync wrote:
> rsync --perms -M--fake-super src dst
> 
> For me, this command means that rsync should save the original perms in the
> xattr, and leave the real file mode to the umask default. Currently it also
> modifies the real file mode, and there is no way to store something
> different
> in the xattr.
> 
> According to an old bug report that I found, more people would like
> --fake-super to be a complete attribute emulation layer.
> 
> https://bugzilla.samba.org/show_bug.cgi?id=7112
> 
> Do you agree? I'm in the process of implementing this as a bug fix to
> rsync,
> and would like to know if everybody agrees with this behaviour. The patch
> would also modify the man page to document it under --perms.
> 
> 
> Regards,
> Dimitris
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Question/comment about -n (dry run) flag of rsync

[Kevin Korb via rsync]

If you used -v then the very last line rsync outputs is:

total size is ###  speedup is ### (DRY RUN)


--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

On Tue, 10 Mar 2020, T. Shandelman via rsync wrote:


Date: Tue, 10 Mar 2020 17:26:41 -0500
From: T. Shandelman via rsync 
To: rsync@lists.samba.org
Subject: Question/comment about -n (dry run) flag of rsync

Rsync is a remarkably handy tool that I use virtually every day.

But there is one thing about rsync that drives me totally crazy.

Under the -n (dry run) flag, rsync seems to produce exactly the same output
as without that flag.

I cannot tell you how many times I sit and scratch my head long and
hard,after I discover that my intended rsync operations did not actually
happen. Until I finally remember that I ran rsync in dry-run mode. That's
why! This is especially a problem for very long-running rsync jobs run in
dry-run mode.

It seems to me that when run in dry-run mode, rsync should display a
warning at the very, very end, something like:

*WARNING: None of the above operations have been actually performed, *
*because you ran rsync in dry-run mode.*


Or does rsync already have such a feature, and I am not aware of it?

But if not, that is my vote for the next feature to be added. It should be
a very, very easy fix.

Todd S.
Austin, Texas, USA



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Exclude-from file format?

[Kevin Korb via rsync]

Either format is correct.  However, there is no quoting or escaping in
the file.  So:
- filename1
- file name 1
or without the -

On 1/3/20 1:45 PM, @lbutlr via rsync wrote:
> I have seen two main styles in examples for using rsync with exclude-from.
> 
> The first is simply a list of filename, one per line.
> 
> The second is a list of filenames, one per line, prefixed with either a - or 
> a + to indicate exclude/include. Which is correct?
> 
> Also, none show the correct style with a filname containing spaces, though I 
> assumed quoting is OK
> 
> Exclude1.txt:
> - filename1
> - “file name 1”
> 
> Exclude.txt:
> filename1
> “file name 1”
> 
> Which of these will exclude both filename1 and “file name 1” anywhere they 
> appear in the source?
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Excluding a . directory in a subdirectory

[Kevin Korb via rsync]

Order matters.  --include overrides --exclude listed later.  So, your
exclude is nullified by both of  your includes.  So move the exclude
before the includes.  Of course then your includes will be irrelevant so
just remove them unless you are leaving out details.

On 12/28/19 9:15 PM, H via rsync wrote:
> I am having problems getting a . directory in a subdirectory to be excluded 
> when rsyncing. I am using rsync 3.1.2 on CentOS 7 and the command line is:
> 
> #    all files from ~/bin/ and I use want to exclude the directory 
> ~/bin/docker/.git
> rsync -vHrltDium -c --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --stats 
> --include='*' --include='*/' --exclude=".git/" -e "ssh -y -p 22" 
> /home/user/bin/ user@1.2.3.4:~/bin
> 
> What am I doing wrong? I thought the exclusion pattern above would exclude 
> any .git subdirectory regardless of level found?
> 
> Thank you.
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Skip creating files in --backup-dir if content has not changed

[Kevin Korb via rsync]

The reason for that is pretty simple...  Rsync isn't reading the
existing file to find out that it is the same.  Doing so would normally
be a waste of rsync's time.  A better question is why are your files
changing timestamps when the data is the same.

On 12/18/19 6:47 PM, Wayne Piekarski via rsync wrote:
> I am using rsync with --backup --backup-dir to keep copies of files
> which have changed as part of an incremental backup system. However, if
> only the timestamp has changed, it creates a copy of the file in
> --backup-dir, and if thousands of large files have their timestamps
> changed, this can waste a lot of disk space on something which hasn't
> really changed.
> 
> Interestingly, if you use --checksum, rsync will not create a file in
> --backup-dir unless the contents are truly different, but it will fix up
> the timestamp on the remote end to match. This is what I want, but I
> just don't want to pay the performance penalty of running --checksum all
> the time.
> 
> Here is an example that shows the problem:
> 
> mkdir ./SRC
> echo hello > ./SRC/a
> echo hello > ./SRC/b
> rsync -av ./SRC/ ./DEST/
> touch ./SRC/*
> ls -al --full-time ./SRC/ ./DEST/
> # Creates copies in BACKUP, even though contents are the same
> rsync -av --backup-dir=`pwd`/BACKUP/ ./SRC/ ./DEST/
> 
> After this run, the BACKUP directory will contain copies of both a and b
> even though neither actually changed. If you add --checksum, then it
> avoids creating a copy, but still syncs the timestamps correctly.
> 
> touch ./SRC/*
> # Does not create any copies in BACKUP since nothing changed
> rsync -av --checksum --backup-dir=`pwd`/BACKUP/ ./SRC/ ./DEST/
> 
> The problem with --checksum is that for hundreds of gigabytes of data,
> it can be very slow to run over every file, especially if the timestamps
> are mostly actually the same. But without it, the delta algorithm in
> rsync has already decided to make a backup copy before it realizes later
> that nothing has changed.
> 
> Is there a flag I can add to rsync that will tell it to only create a
> backup file if something actually changed, saving lots of wasted backup
> space?
> 
> thanks,
> Wayne
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How Can I save the original permissions while setting g+rX for all files

[Kevin Korb via rsync]

When I try it the chmod works on both the real permissions and the
permissions in the xatttr.  Maybe the behavior has changed since
whatever version you have (3.1.3 here) but this probably wouldn't help
you either.


On 12/8/19 7:56 PM, Dimitrios Apostolou via rsync wrote:
> Hello list!
> 
> Combining -M--fake-super with --chmod ends up changing the permissions
> stored in the fake-super xattrs. I.e. the permissions stored in the
> xattr, are
> affected by --chmod.
> 
> The desirable behaviour for me would be for --chmod to modify the real
> permissions of the destination files. The use case is writing a backup as a
> non-root user, having it readable by the group (g+rX), while also saving
> the
> original permissions in the xattr.
> 
> Is it a bug? Any suggestions on how I can achieve what I want?
> 
> 
> Thanks in advance,
> Dimitris
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Seemingly impossible bug: -v not always listing every copied file

[Kevin Korb via rsync]

It does seem impossible.  I would suggest adding --itemize-changes (-v
isn't really all that useful without it anyway).  If entries are still
missing then I would suspect that either log files are missing (maybe
duplicate file names replacing the occasional log file?) or something
other than rsync is doing things in the same dir.

On 10/29/19 9:00 PM, raf via rsync wrote:
> Hi,
> 
> debian-9, rsync-3.1.2 (both ends)
> 
> I have a task that rsyncs files from a list of
> candidate files (--files-from=). It's verbose (-v) and
> its stdout is captured to a file which is then sent to
> the receiving host. The captured verbose output is
> examined on the receiving host to know which files were
> actually copied so that notification emails can be sent
> to various people.
> 
> The problem is that, sometimes, not all copied files
> are listed in the verbose output, and so some
> notification emails don't get sent out. At first, I
> thought there was something wrong with the notification
> emails not arriving, but the files in question, that
> had definitely been copied, did not appear in any of
> the captured verbose output files.
> 
> This seems like an impossible bug but it really seems
> to be happening. Has anyone else encountered behaviour
> like this? I didn't have much luck searching the
> internet for it. It would probably be hard to notice
> if the verbose output wasn't being used for something
> like triggering notifications whose absence might be
> noticed.
> 
> cheers,
> raf
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: difference in behaviour of --filter --delete: cmdline vs. file

[Kevin Korb via rsync]

It is a little confusing but in order to have an exclude dir merge
filter do both things (prevent transfer and prevent deletion) then it
needs to be on both ends of the transfer.  The file on each end only has
partial effect on what rsync does.

Think of it like a file you (as a regular user) can make in your home
dir telling the backup system to not backup a file.  If that file has
already been backed up rsync will also remove it from the backup on the
next run.

On 8/7/19 1:04 PM, Madhu via rsync wrote:
> Hello,
> 
> I want to avoid transferring a file which is present both on the source
> tree and on the target tree when calling rsync with -a --delete.
> 
> mkdir -pv src/ dst/
> echo file1 > src/file
> echo file2 > dst/file
> 
> rsync -navi --delete --exclude file src/ dst/
> 
> works as expected. The file is not transferred nor is it deleted at
> the receiver.
> 
> rsync -navi --delete --filter '- file' src/ dst/
> 
> also works as expected. The file is not transferred nor is it deleted at
> the receiver.
> 
> echo '- file' > src/.rsync-filter
> rsync -navi --delete -F -F src/ dst/
> 
> file is now deleted at the receiver, which is unexpected.  I expect this
> to be the same as the earlier two invocations.
> 
> Am I missing something - or is this a bug?
> 
> I think the manual could be clearer on the s and r modifiers with regard
> to --delete.  For now neither of the following works to protect file
> from being deleted at the receiver
> 
> echo '-s file' > src/.rsync-filter
> rsync -navi --delete -c -F -F src/ dst/
> 
> echo '-r file' > src/.rsync-filter
> rsync -navi --delete -c -F src/ dst/
> 
> ---Madhu
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Advanced rsync includes and excludes

[Kevin Korb via rsync]

I believe you can shorten that to:

+ /some/very/
+ /some/very/deep/
+ /some/very/deep/path/
+ /some/very/deep/path/to/
+ /some/very/deep/path/to/save/***
- /some/*

You could also exclude /some and then use /some/very/deep/path/to/save
as an additional source.  I don't know if rsnapshot can handle multiple
sources in 1 rsync but rsync itself can.

On 8/7/19 9:54 AM, Hans-Peter Jansen via rsync wrote:
> Hi,
> 
> I'm a happy camper @ rsync (and rsnapshot) since years. Thanks for this major 
> piece of software.
> 
> In an attempt to reorganize my rsnapshot backups, I stumbled across an issue, 
> that I'm trying to seeking a more sophisticated solution here.
> 
> Given, I have a deeply branched tree, where I would like to include a 
> specific 
> directory deep under, while excluding anything else on that path, I find 
> myself doing:
> 
> + /some/
> + /some/very/
> + /some/very/deep/
> + /some/very/deep/path/
> + /some/very/deep/path/to/
> + /some/very/deep/path/to/save/
> - /some/very/deep/path/to/*
> - /some/very/deep/path/*
> - /some/very/deep/*
> - /some/very/*
> - /some/*
> 
> While it works, it feels rather awkward and gets very complicated, if you 
> have 
> more of such items to deal with. Sure, I could run this separately, but this 
> isn't the real McCoy either with a complicated tree structure.
> 
> What are your favorite ways of doing such tasks with rsync?
> 
> Thanks in advance,
> Pete
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync many processes and slow backup

[Kevin Korb via rsync]

In addition to the cron problems that others have pointed out there are
rsync problems...

#1  rsync's -c is almost always a bad idea and is extremely slow
#2  if rsync isn't networking then -z is a waste of time

On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote:
> Hi all! :-)
>  
> I have a small rsync script to sync my data to a usb-disk. It works
> fine, when I start it in console. I get 3 rsync processes (look in top)
> and the backup takes ~25 min. But, when I add the script in cron to
> start it at 1am at night it takes 7 - 9 hours and I see up to
> 180 processes. When I look in top I see a hight load of 60 - 80 and 40 -
> 60 waits. Why? Can someone explain why it takes so long when it starts
> with cron?
>  
> This is my rsync command:  rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" 
> This is the entry in cron (crontab -e): * 2 * * *
> /root/backupscript/backup.sh
> Data to sync: 18 Gb, 185.000 files.
>  
> When I look in the log files I see errors like this: 
>  
> rsync:
> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c"
> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 
> 
> rsync error: some files/attrs were not transferred (see previousrsync error:
> 
> some files/attrs were not transferred (see previous errors) (code 23) at
> 
> main.c(1196) [sender=3.1.2]
> 
> directory (2)
> 
>  
> 
> Can you help me to solve this problem?
> 
>  
> 
> regards,
> 
> Hannes Hutmacher
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: "WARNING: failed verification -- update retained (will try again)" message

[Kevin Korb via rsync]

Usually it means that the file changed while rsync was running.  But it
can also mean that a hardware problem (usually RAM) is causing the
hashing to return bogus results.

On 6/3/19 6:09 AM, sambalist--- via rsync wrote:
> Hi there!
> 
> On a daily backup script, I am getting a lot of these messages:
> 
>   WARNING:  -- update retained (will try again).
> 
> I've tried googling, but i can't figure out what these messages exactly
> mean and how i can solve them.
> 
> Any hint?
> 
> Thank you
> 
> Gabriele
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --delete-missing-args doesn't delete

[Kevin Korb via rsync]

--files-from will delete files from the destination that are listed but
not existing in the source.  It isn't delete what isn't listed.

On 4/23/19 5:29 AM, MI via rsync wrote:
> I'm generating a list of files to sync, and would like all the files not
> in my list to be deleted from the destination. I thought that
> --delete-missing-args would do just that, but it doesn't delete anything.
> 
> Would someone have an idea of what I'm doing wrong?
> 
> This is what I tried :
> 
> ~$ mkdir -p /tmp/source /tmp/dest
> 
> ~$ for i in {1..3}; do echo "keep $i"      > /tmp/source/keep$i; done
> 
> ~$ for i in {4..5}; do echo "to remove $i" > /tmp/source/to-delete-$i; done
> 
> ~$ rsync -va /tmp/source/ /tmp/dest/
> sending incremental file list
> ./
> keep1
> keep2
> keep3
> to-delete-4
> to-delete-5
> 
> sent 410 bytes  received 114 bytes  1,048.00 bytes/sec
> total size is 45  speedup is 0.09
> 
> ~$ for i in {1..3}; do touch /tmp/source/keep$i; done
> 
> ~$ cd /tmp/source/
> 
> /tmp/source$ find . -name "keep*" | tee /tmp/source/file-list
> ./keep2
> ./keep1
> ./keep3
> 
> Now, using my file-list which does not include the "to-delete*" files, I
> hoped that these would be deleted. The new files were correctly copied,
> but nothing was deleted:
> 
> /tmp/source$ rsync -vva --delete --delete-missing-args --force
> --files-from=/tmp/source/file-list . /tmp/dest/
> building file list ... done
> delta-transmission disabled for local transfer or --whole-file
> ./
> keep1
> keep2
> keep3
> total: matches=0  hash_hits=0  false_alarms=0 data=21
> 
> sent 259 bytes  received 143 bytes  804.00 bytes/sec
> total size is 21  speedup is 0.05
> 
> /tmp/source$ ls -Al /tmp/dest/
> total 20
> -rw-rw-r-- 1 mi mi  7 Apr 22 23:43 keep1
> -rw-rw-r-- 1 mi mi  7 Apr 22 23:43 keep2
> -rw-rw-r-- 1 mi mi  7 Apr 22 23:43 keep3
> -rw-rw-r-- 1 mi mi 12 Apr 22 23:42 to-delete-4
> -rw-rw-r-- 1 mi mi 12 Apr 22 23:42 to-delete-5
> 
> 
> The files not in my --files-from list are still at the destination.
> 
> Thanks for any help,
> 
> MI
> 
> 
> PS: my version of rsync :
> 
> ~$ rsync --version
> rsync  version 3.1.1  protocol version 31
> Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
> Web site: http://rsync.samba.org/
> Capabilities:
>     64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
>     socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
>     append, ACLs, xattrs, iconv, symtimes, prealloc
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Improved output (dry run)

[Kevin Korb via rsync]

Having dry-run not act like the same command without --dry-run would
make less sense than people occasionally running it and discovering that
they need more options.

-v doesn't show things it didn't touch.  So, it touched them for some
reason.  Without --itemize-changes you are left in the dark about what
is happening or why.  But what you probably want is --omit-dir-times

That being said, -v is virtually useless without --itemize-changes anyway.

On 4/14/19 6:30 AM, darkdragon via rsync wrote:
> Please improve (default) output for /--dry-run/.
> 
> Specifically, it would be nice to imply /-v/ since dry runs without
> displaying anything make no sense.
> 
> Further (independently from dry run), it would be nice to have an option
> to hide directories which did *not* change. The need to /grep/ results
> is really frustrating and error prone.
> 
> Thanks a lot!
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Possible not expected behaviour with "delete" option

[Kevin Korb via rsync]

It doesn't seem like the trailing / on the target should matter.  It
only means something special on the source.

However, if there is any error then rsync will abort the deletions for
safety.

On 4/3/19 2:10 PM, lugarci1 via rsync wrote:
> Hi, I was trying to make a mirror backup of a folder in my machine with
> the followin|g command:|
> |rsync -avh --delete /source/ /destination|
> |
> |
> |But the extra files that where on "destination" didn't got deleted.
> Where as doing:|
> ||rsync -avh --delete /source/ /destination/||
> ||worked as expected.||
> ||
> ||
> ||I wonder if this is correct, or it shouldn't be like that.||
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync rewrites all blocks of large files although it uses delta transfer

[Kevin Korb via rsync]

It can't do what you want.  The closest thing would be --compare-dest.

On 2/13/19 5:26 PM, Delian Krustev wrote:
> On Wednesday, February 13, 2019 11:29:44 AM EET Kevin Korb via rsync 
>  wrote:
>> With --backup in order to end up with 2 files it has to write out a
>> whole new file.
>> Sure, it only sent the differences (normally that means
>> over the network but there is no network here) but the writing end was
>> told to duplicate the file being updated before updating it.
> 
> The copy is needed for the comparison of the blocks as "--inplace" overwrites 
> the destination file. I've tried without "--backup" but then the delta 
> transfers too much data - close to the size of the backed-up files.
> 
> The copy is in a temp file system which is discarded after the backup (by "rm 
> -rf"). This temp filesystem is not log structured or copy-on-write so having 
> a 
> copy there is not a big problem. Although I don't want a backup of all files 
> which are modified but rather a TMPDIR.
> 
> The ideal workflow would be to compare SRC and DST and write changed blocks 
> to 
> the TMPDIR, then read them from TMPDIR and apply it to DST.
> 
> 
> 
>  
> Cheers
> --
> Delian
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync rewrites all blocks of large files although it uses delta transfer

[Kevin Korb via rsync]

With --backup in order to end up with 2 files it has to write out a
whole new file.  Sure, it only sent the differences (normally that means
over the network but there is no network here) but the writing end was
told to duplicate the file being updated before updating it.

On 2/13/19 10:47 AM, Delian Krustev via rsync wrote:
>   Hi All,
> 
> For a backup purpose I'm trying to transfer only the changed blocks of
> large files. Thus I've run "rsync" with the appropriate options:
> 
>   RSYNC_BKPDIR=`mktemp -d`
>   rsync \
>   --archive \
>   --no-whole-file \
>   --inplace \
>   --backup \
>   --backup-dir="$RSYNC_BKPDIR" \
>   --verbose \
>   --stats \
>   /var/backups/mysql-dbs/. \
>   /mnt/bkp/var/backups/mysql-dbs/.
> 
> The problem is that although "rsync" shows that delta transfer is used(when 
> run with -vv) and only small amount if data is transferred, the target files 
> look to be overwritten in full.
> 
> Here is the output of "rsync" and some more debugging info:
> 
> 
> 
> sending incremental file list
> ./
> horde.data.sql
> horde.schema.sql
> LARGEDB.data.sql
> LARGEDB.schema.sql
> mysql.data.sql
> mysql.schema.sql
> phpmyadmin.data.sql
> phpmyadmin.schema.sql
> 
> Number of files: 9 (reg: 8, dir: 1)
> Number of created files: 0
> Number of deleted files: 0
> Number of regular files transferred: 8
> Total file size: 1,944,522,704 bytes
> Total transferred file size: 1,944,522,704 bytes
> Literal data: 21,421,681 bytes
> Matched data: 1,923,101,023 bytes
> File list size: 0
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 21,612,218
> Total bytes received: 323,302
> 
> sent 21,612,218 bytes  received 323,302 bytes  259,591.95 bytes/sec
> total size is 1,944,522,704  speedup is 88.65
> 
> # du -m 1.9G /tmp/tmp.8gBzjNQOQZ
> 1.9G /tmp/tmp.8gBzjNQOQZ
> 
> # tree -a /tmp/tmp.8gBzjNQOQZ
> /tmp/tmp.8gBzjNQOQZ
> ├── horde.data.sql
> ├── horde.schema.sql
> ├── LARGEDB.data.sql
> ├── LARGEDB.schema.sql
> ├── mysql.data.sql
> ├── mysql.schema.sql
> ├── phpmyadmin.data.sql
> └── phpmyadmin.schema.sql
> 
> 0 directories, 8 files
> 
> Free space at the beginning and end of the backup:
> Filesystem 1M-blocks   Used Available Use% Mounted on
> /dev/mapper/bkp   102392  76872 20400  80% /mnt/bkp
> /dev/mapper/bkp   102392  78768 18504  81% /mnt/bkp
> 
> 
> 
> As can be seen "rsync" has sent about 20M and received 300K of data. However 
> the filesystem has allocated almost 2G, which is the total size of the files 
> being backed up.
> 
> The filesystem mounted on "/mnt/bkp" is of type "nilfs2", which is a log 
> structured filesystem. I'm using its snapshotting feature to keep backups for 
> past dates.
> 
> 
> Is there anything that can be done in order "rsync" to overwrite only the 
> changed blocks ?
> 
> 
> 
> 
> P.S. I guess that it will be the same for copy-on-write filesystems, e.g. 
> BTRFS or ZFS.
> 
> 
> 
> Cheers
> --
> Delian
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --server options

[Kevin Korb via rsync]

I don't remember your original question but if I didn't suggest looking
at rrsync I should have.  It comes with rsync in the contrib dir and it
knows which options to allow/deny and it can restrict the transfer to a
specific dir and read-only or write-only.  Since it comes with rsync it
is kept current with the options rsync uses including the undocumented ones.

Even if it doesn't do exactly what you are trying to do, it is a perl
script so it should be easy enough to modify for your needs.  Just keep
a diff so you can easily analyze changes in new versions.

On 2/7/19 10:57 PM, Richard Hector via rsync wrote:
> On 21/01/19 8:07 PM, Richard Hector via rsync wrote:
>> Hi all,
>>
>> I see a couple of earlier threads, particularly this one:
>>
>> https://www.mail-archive.com/rsync@lists.samba.org/msg32328.html
>>
>> partly answer my question, but not fully.
>>
>> I also am writing a wrapper, so that I can run multiple dirvish runs
>> against the same server, using forced commands, but without using
>> multiple keys to get multiple commands, which is my current workaround.
>>
>> But in the interests of locking things down, I'd like to know what the
>> options are in the -eSTRING section, so I can choose which to
>> allow/force. Is that documented anywhere? Or do I have to resort to the
>> source?
>>
>> And/or can I safely assume that anything in the -eSTRING won't let a
>> client get any more than I intended? Does the STRING just run from after
>> the -e to the next whitespace?
> 
> Anybody? Are these options/flags deliberately kept obscure?
> 
> Cheers,
> Richard
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --link-dest. Time to 'building file list' incrementing

[Kevin Korb via rsync]

It does normally take some time to analyze large trees of files.  It has
to call stat() on each file to get the size and timestamp.

However, 15 hours seems a bit excessive even though I have never tried
to do this on Windows or a NAS system.  Just to be clear, is your
--link-dest parameter a single directory or are you trying to tell it to
use all of the previous backups?

Also, have you deleted a backup yet?  In my experience that takes a lot
longer than running one so if you need 15 hours to run a backup I would
expect deleting one to take about a week.

On 1/3/19 4:23 AM, John Simpson via rsync wrote:
> 
> 
> I've been running rsync as a cygwin task on Windows Server 2008 for about two 
> months now. I'm using the --link-dest option to do a daily 'snapshot' of the 
> contents of a server containing about 10TB of data, about 13 million files, 
> to a Linux based NAS server. Things started out great but I soon noticed that 
> the time take to complete was slowly incrementing. It started at around three 
> hours, but is now around fifteen.
> 
> The command is as follows...
> 
> rsync -rlptDhPR \
> --password-file=password \
> --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \
> --Stats \
> --delete \
> --log-file=logfilename \
> --link-dest=linkdestpath \
> sourceDirectory \
> rsync@192.168.1.2::destinationDirectory
> 
> I'm not using the full -a option as the differences between the Windows and 
> Linux ownership stuff messed things up.
> 
> The first log file looked like this...
> 
> 2018/10/01 23:00:14 [2164] building file list
> ...transfer file list here
> 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 
> 1,499,821, link: 1)
> 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11)
> 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310
> 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes
> 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes
> 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes
> 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes
> 2018/10/02 02:11:30 [2164] File list size: 10.42M
> 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds
> 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds
> 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M
> 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M
> 2018/10/02 02:11:30 [2164] sent 235.68M bytes  received 7.51M bytes  21.17K 
> bytes/sec
> 2018/10/02 02:11:30 [2164] total size is 10.40T  speedup is 42,753.79
> 
> the most recent looks like this...
> 
> 2018/11/24 23:00:15 [2924] building file list
> 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/
> 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 
> 1,501,780, link: 1)
> 2018/11/25 13:21:16 [2924] Number of created files: 0
> 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0
> 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes
> 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes
> 2018/11/25 13:21:16 [2924] Literal data: 0 bytes
> 2018/11/25 13:21:16 [2924] Matched data: 0 bytes
> 2018/11/25 13:21:16 [2924] File list size: 10.35M
> 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds
> 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds
> 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M
> 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M
> 2018/11/25 13:21:16 [2924] sent 236.55M bytes  received 7.51M bytes  4.72K 
> bytes/sec
> 2018/11/25 13:21:16 [2924] total size is 10.49T  speedup is 42,996.96
> 
> As you can see the start time is 11:00PM (23:00) in both cases. The first log 
> shows that identifying the files to transfer took about three hours (I've 
> omitted the file list - it's quite long), the second log takes fourteen hours 
> to do the same job (in this case this was done at the weekend and I've 
> include the whole log file which correctly identifies that no files have 
> changed)
> 
> The number of files is as might be expected as is everything else. It's just 
> the time taken "building file list" is significantly larger.
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --partial does not "unhide" the files

[Kevin Korb via rsync]

I can't say I have any idea why rsync would just skip that step and I
can't duplicate it myself.

Your only recourse might be to use --inplace on that system.

On 12/31/18 12:33 PM, Heiko Schlittermann via rsync wrote:
> Kevin Korb via rsync  (So 30 Dez 2018 23:56:44 CET):
>> I think --partial might be a red herring here.  It only applies to what
>> happens when rsync is aborted in the middle of a file.  What happens
>> without -P?
> 
> Same happens w/o --partial. I append 2 logs:
> - a from localhost to remote server, exposing the missing "rename"
> - b from localhost to localhost
> 
> In both cases the file itself is empty, and didn't exist on the
> receivers side.
> 
> The unified diff looks like this:
> 
> --- /tmp/a2018-12-31 18:30:38.747209487 +0100
> +++ /tmp/b2018-12-31 18:30:45.623719009 +0100
> @@ -1,6 +1,6 @@
> -cmd= machine=platte user= path=/tmp/y
> -cmd[0]=ssh cmd[1]=platte cmd[2]=rsync cmd[3]=--server 
> cmd[4]=-e.LsfxC cmd[5]=. cmd[6]=/tmp/y
> -opening connection using: ssh platte rsync --server -e.LsfxC . 
> /tmp/y  (7 args)
> +cmd= machine=localhost user= path=/tmp/y
> +cmd[0]=ssh cmd[1]=localhost cmd[2]=rsync cmd[3]=--server 
> cmd[4]=-e.LsfxC cmd[5]=. cmd[6]=/tmp/y
> +opening connection using: ssh localhost rsync --server -e.LsfxC . 
> /tmp/y  (7 args)
>  msg checking charset: UTF-8
>  (Client) Protocol versions: remote=31, negotiated=31
>  [sender] make_file(x,*,0)
> @@ -10,7 +10,7 @@
>  [sender] flist_eof=1
>  file list sent
>  send_files starting
> -server_recv(2) starting pid=16534
> +server_recv(2) starting pid=18896
>  recv_file_name(x)
>  received 1 names
>  [Receiver] flist_eof=1
> @@ -18,7 +18,7 @@
>  [Receiver] i=0 1 x mode=0100644 len=0 flags=1000
>  recv_file_list done
>  get_local_name count=1 /tmp/y
> -generator starting pid=16534
> +generator starting pid=18896
>  delta-transmission enabled
>  recv_generator(y,0)
>  send_files(0, ./x)
> @@ -30,10 +30,11 @@
>  false_alarms=0 hash_hits=0 matches=0
>  sender finished ./x
>  generate_files phase=1
> -recv_files(1) starting
>  send_files phase=1
> +recv_files(1) starting
>  recv_files(y)
>  got file_sum
> *   +renaming .y.Oq45md to y
>  recv_files phase=1
>  generate_files phase=2
>  send_files phase=2
> @@ -43,9 +44,9 @@
>  recv_files finished
>  generate_files phase=3
>  generate_files finished
> -client_run waiting on 18876
> +client_run waiting on 18878
> 
> -sent 75 bytes  received 533 bytes  1,216.00 bytes/sec
> +sent 75 bytes  received 561 bytes  424.00 bytes/sec
>  total size is 0  speedup is 0.00
>  [sender] _exit_cleanup(code=0, file=main.c, line=1196): entered
>  [sender] _exit_cleanup(code=0, file=main.c, line=1196): about to call 
> exit(0)
> 
> 
> Best regards from Dresden/Germany
> Viele Grüße aus Dresden
> Heiko Schlittermann
> --
>  SCHLITTERMANN.de  internet & unix support -
>  Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
>  gnupg encrypted messages are welcome --- key ID: F69376CE -
>  ! key id 7CBF764A and 972EAC9F are revoked since 2015-01  -
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Aw: Re: rsync remote raw block device with --inplace

[Kevin Korb via rsync]

Rsync would only benefit you in terms of reducing network bandwidth
(assuming it is even networking).  The resulting file will be the size
of the block device just like you get with dd[rescue].  You can tell
rsync to write the file sparsely which would mean that any parts of the
disk that have never been written to would probably still contain nulls
and would not be allocated but you will still be copying any deleted
file data that still exists.

Ntfsclone and partimage understand the filesystem.  They will just skip
the blocks that don't contain any current data.  This will keep your
backup file as small as possible.  If you ever have to do a restore they
will put all the file data back where it was and either skip the parts
they didn't save or fill them with nulls.

On 12/30/18 3:50 PM, devz...@web.de wrote:
>> There have been addons to rsync in the past to do that but rsync really
>> isn't the correct tool for the job.
> 
> why not correct tool ?
> 
> if rsync can greatly keep two large files in sync between source and 
> destination
> (using --inplace), why should it (generally spoken) not also be used to keep 
> two 
> blockdevices in sync ? 
> 
> maybe these links are interesting in that context:
> 
> https://lists.samba.org/archive/rsync/2010-June/025164.html
> 
> https://github.com/dop251/diskrsync
> 
> roland
> 
>> Gesendet: Sonntag, 30. Dezember 2018 um 19:53 Uhr
>> Von: "Kevin Korb via rsync" 
>> An: rsync@lists.samba.org
>> Betreff: Re: rsync remote raw block device with --inplace
>>
>> There have been addons to rsync in the past to do that but rsync really
>> isn't the correct tool for the job.  Neither is dd.
>>
>> The right tool is something that understands the filesystem within the
>> block device such as ntfsclone (what I use) or partimage (if you have
>> ever used Clonezilla this is what it uses).  These will know how to skip
>> all the empty parts of the filesystem and will still be capable of
>> restoring a complete image in a bare metal restore.  You can still use
>> dd to snag a copy of the MBR since that is outside of any filesystems.
>>
>> Also, if you do have to resort to a plain image use ddrescue instead of
>> dd.  It has a status screen and it can resume as long as you used a log
>> file when you ran it.
>>
>> On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
>>> It would be very nice to be able to rsync the raw data content of, e.g.,
>>> a non-mounted disk partition, particularly in combination with --inplace.
>>>
>>> Our reality: several dual-boot machines running Windows during the day
>>> and Linux at night, during backups.  Windows is very tedious and iffy to
>>> re-reinstall without a raw disk image to start from.  Disks fail, and
>>> the ensuing downtime must be minimized. 
>>>
>>> We're using dd for this.  Most of the nightly work is redundant and
>>> wasteful of elapsed time and storage.  Storage is cheap, but it's not
>>> *that* cheap.  Elapsed time is priceless.
>>>
>>> Rsync refuses to back up raw devices, and even raw character devices,
>>> with the message "skipping non-regular file" (I think the relevant
>>> message is in generator.c). 
>>>
>>> In Linux, anyway, the "raw" command allows a block device to be bound as
>>> a character device, and then even a "cat" command can read the raw data
>>> of the block device.  So why does rsync refuse to copy such content, or
>>> why is it a bad idea, or what rsync doctrine conflicts with it?  I agree
>>> there are security concerns here, but rsync already disallows some of
>>> its functions unless the super user is requesting them. 
>>>
>>>
>>
>> -- 
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>  Kevin Korb  Phone:(407) 252-6853
>>  Systems Administrator   Internet:
>>  FutureQuest, Inc.   ke...@futurequest.net  (work)
>>  Orlando, Floridak...@sanitarium.net (personal)
>>  Web page:   https://sanitarium.net/
>>  PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>> -- 
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options: 
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Ke

Re: rsync remote raw block device with --inplace

[Kevin Korb via rsync]

There have been addons to rsync in the past to do that but rsync really
isn't the correct tool for the job.  Neither is dd.

The right tool is something that understands the filesystem within the
block device such as ntfsclone (what I use) or partimage (if you have
ever used Clonezilla this is what it uses).  These will know how to skip
all the empty parts of the filesystem and will still be capable of
restoring a complete image in a bare metal restore.  You can still use
dd to snag a copy of the MBR since that is outside of any filesystems.

Also, if you do have to resort to a plain image use ddrescue instead of
dd.  It has a status screen and it can resume as long as you used a log
file when you ran it.

On 12/30/18 1:45 PM, Steve Newcomb via rsync wrote:
> It would be very nice to be able to rsync the raw data content of, e.g.,
> a non-mounted disk partition, particularly in combination with --inplace.
> 
> Our reality: several dual-boot machines running Windows during the day
> and Linux at night, during backups.  Windows is very tedious and iffy to
> re-reinstall without a raw disk image to start from.  Disks fail, and
> the ensuing downtime must be minimized. 
> 
> We're using dd for this.  Most of the nightly work is redundant and
> wasteful of elapsed time and storage.  Storage is cheap, but it's not
> *that* cheap.  Elapsed time is priceless.
> 
> Rsync refuses to back up raw devices, and even raw character devices,
> with the message "skipping non-regular file" (I think the relevant
> message is in generator.c). 
> 
> In Linux, anyway, the "raw" command allows a block device to be bound as
> a character device, and then even a "cat" command can read the raw data
> of the block device.  So why does rsync refuse to copy such content, or
> why is it a bad idea, or what rsync doctrine conflicts with it?  I agree
> there are security concerns here, but rsync already disallows some of
> its functions unless the super user is requesting them. 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --dry-run won't say where the files are going to go

[Kevin Korb via rsync]

/tmp/Maildir/new/1546157908.1392_1.jidanni7

Also, don't use -z on a local copy.  Rsync is dumb enough to do what you
tell it to.

On 12/30/18 3:40 AM, 積丹尼 Dan Jacobson via rsync wrote:
> Try as we may (--dry-run), it still won't tell us where it intends to put the 
> files,
> 
> $ rsync --dry-run --remove-source-files --relative --verbose 
> --itemize-changes -Cavz Maildir/new /tmp/
> sending incremental file list
> delta-transmission disabled for local transfer or --whole-file
> cd+ Maildir/
> cd+ Maildir/new/
>> f+ Maildir/new/1546157908.1392_1.jidanni7
> total: matches=0  hash_hits=0  false_alarms=0 data=0
> 
> sent 155 bytes  received 94 bytes  498.00 bytes/sec
> total size is 320  speedup is 1.29 (DRY RUN)
> 
> Guess we'll just have to cross our fingers...
> 
> rsync  version 3.1.2  protocol version 31
> 
> (In fact there is no option to show where the files will go,
> --dry-run or not.)
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html