Re: Trying to elevate rsync privileges when connecting over ssh without using NOPASSWD in sudoers

[raf via rsync]

On Fri, Mar 11, 2022 at 10:36:49PM -0800, Bri Hatch via rsync 
 wrote:

> On Fri, Mar 11, 2022 at 10:22 PM Kevin Korb via rsync 
> wrote:
> 
> > Rsync includes a script named rrsync that handles this perfectly.
> 
> And authprogs provides similar functionality, though you use yaml to define
> what is/isn't allowed. However it does allow you to use one SSH identity
> for potentially many different source dirs rather than requiring a separate
> authorized_key entry for each forced command.
> 
> example:
> 
> - rule_type: rsync
>   allow_donwload: true
>   allow_recursive: true
>   paths:
> - /etc
> - /srv/freezeray
>   path_startswith:
> - /srv/web
> 
> https://github.com/daethnir/authprogs/blob/main/doc/authprogs.md#rsync-subrules

And there's sshdo as well: https://github.com/raforg/sshdo
Like authprogs, it also works with any command, not just rsync.
And it almost configures itself with a learning mode to monitor
commands that need to be allowed. And it can relearn if commands
need to change over time, and unlearn old commands that are no
longer needed.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to manage root<-->root rsync keeping permissions?

[raf via rsync]

On Sat, Aug 07, 2021 at 11:44:34AM +0100, Chris Green via rsync 
 wrote:

> L A Walsh via rsync  wrote:
> > On 2021/08/03 07:09, Chris Green via rsync wrote:
> > > I already have an rsync daemon server running elsewhere, I can add
> > > this requirement to that I think.  Thank you.
> > >   
> > 
> > 
> > It seems to me, a safer bet would be to generate an ssh-cert
> > that allows a passwdless login from your sys to the remote.
> > 
> The trouble with that is that it leaves a big security hole.
> 
> If (for example) I leave my laptop turned on somewhere, or someone
> wanders into my study where my desktop machine is they have instant,
> passwordless access to the remote backup machine.
> I try very hard to make my backups secure from attack so that if my
> desktop or laptop is compromised somehow the (remote) backups are
> still secure.

You can use my sshdo program (https://github.com/raforg/sshdo)
to solve this problem. It limits the commands that a
key is allowed to execute to just those commands that
it has been trained on and nothing else. It makes it OK
to have an unencrypted ssh key used for automated or
scripted actions such as backups. The key can only be
used to execute the exact commands that you need it to.

It works by allowing everything during a brief learning
period, and then you tell it to add the commands that
have been encountered to its policy and disallow
anything else. You can also periodically unlearn or
relearn as needed. It does all the heavy lifting. You
just have to examine what it's seeing, and decide if it
should be added to the policy.

> The backup system that runs the rsync daemon has its rsync configured
> with 'refuse options = delete' so not only does someone with access to
> my desktop/laptop need to know the rsyncd username and password but
> they also cannot delete my existing backups.  It runs incremental
> backups so nothing is ever overwritten either.
> 
> -- 
> Chris Green

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: How to manage root<-->root rsync keeping permissions?

[raf via rsync]

On Tue, Aug 03, 2021 at 10:51:34AM +, Andy Smith via rsync 
 wrote:

> Hi Chris,
> 
> On Tue, Aug 03, 2021 at 09:48:37AM +0100, Chris Green via rsync wrote:
> > But how do you handle the other end to restore the root ownership etc.?
> > The script has to do something like:-
> > 
> > rsync -a /etc/ chris@remote:backups/etc/
> > 
> > So at the remote end it only has chris' privileges.
> 
> A couple of options:
> 
> 
> https://strugglers.net/~andy/blog/2021/04/10/rsync-and-sudo-without-x-forwarding/
> 
> Since you want to automate it I'd go with letting root log in by ssh
> key only, and force the key to work only with a specific script.
> 
> Here is an example forced command that only allows rsync
> 
> https://www.guyrutenberg.com/2014/01/14/restricting-ssh-access-to-rsync/
> 
> This is still vulnerable to doing anything that rsync can do. You
> can secure it further by making a script that only does the specific
> things you need rsync to do, e.g. the exact parameters and paths,
> and force that script instead.
> 
> Cheers,
> Andy

An easy way to restrict rsync (or anything else that
operates over ssh) to only the precise commands you
need it to do, without allowing it free reign, is my
sshdo program (https://github.com/raforg/sshdo). You
can use it to learn what rsync is doing, then lock it
down to only what it has seen happen and nothing else.
That might help.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is there a default for 'pid file' for rsync in daemon mode?

[raf via rsync]

On Sat, Jan 02, 2021 at 11:41:52AM -0800, Wayne Davison  
wrote:

> On Fri, Jan 1, 2021 at 2:13 PM raf wrote:
> 
> > maybe the pidfile should be locked
> >
> 
> https://download.samba.org/pub/rsync/NEWS#3.2.0
> 
> ..wayne..

lovely. it must have been an old version.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Is there a default for 'pid file' for rsync in daemon mode?

[raf via rsync]

On Fri, Jan 01, 2021 at 11:31:11AM +, Chris Green via rsync 
 wrote:

> Francis.Montagnac--- via rsync  wrote:
> > 
> > Hi.
> > 
> > On Fri, 01 Jan 2021 09:57:38 + Chris Green via rsync wrote:
> > 
> > > My backup system crashed a couple of nights ago due to a power cut
> > > (can't really blame it!) and I went and restarted it after the power
> > > came back.  However, as I note above, files in /home/chris/tmp/pid
> > > aren't cleared out so rsync refused to run when systemd attempted to
> > > start it up.
> > 
> > > I guess the pid file should either be in /tmp or somewhere in
> > > /run/user.  However shouldn't there be some sort of sensible default
> > > if it isn't set in rsyncd.conf?
> > 
> > Since rsyncd is launched by systemd, I would say that no pid file is
> > needed: systemd will prevent simultaneous launch of rsyncd.
> > 
> > Can you try without spicifying "pid file" in rsyncd.conf ?
> > 
> > What is the content of the rsyncd.service file on your system ?
> > 
> > On Fedora it is a system (not user) service:
> > 
> > cat /lib/systemd/system/rsyncd.service 
> > [Unit]
> > Description=fast remote file copy program daemon
> > ConditionPathExists=/etc/rsyncd.conf
> > Wants=network-online.target
> > After=network-online.target
> > 
> > [Service]
> > EnvironmentFile=/etc/sysconfig/rsyncd
> > ExecStart=/usr/bin/rsync --daemon --no-detach "$OPTIONS"
> > 
> > [Install]
> > WantedBy=multi-user.target
> > 
> You're quite right, simply removing the 'pid file' setting from
> rsynd.conf seems to work perfectly.
> 
> Maybe this should be noted somwhere in the rsyncd.conf man page.
> 
> -- 
> Chris Green

And maybe the pidfile should be locked so that a new
rsyncd process can tell if it's still in use by a
running rsyncd process. That would fix the problem on
systems that don't have systemd.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: "-f ._filterlist" syntax

[raf via rsync]

On Sun, Sep 27, 2020 at 03:19:17PM -0700, Will McGinnis via rsync 
 wrote:

> Hello, A GUI program uses rsync for network transfers. It is able to
> load  filters from a file by putting a dot and underscore before it.
> Example:mkdir 1 2; touch 1/a .empty; rsync -f ._empty 1/ 2/
> 
> Is the dot underscore specific to rsync or is it a general bash (or
> similar?) feature to load arguments from a file? And is it documented
> anywhere for rsync?

>From the manpage:

  FILTER RULES
[...]
Rsync builds an ordered list of filter rules as
specified on the command-line. Filter rules have
the following syntax:

  RULE [PATTERN_OR_FILENAME]
  RULE,MODIFIERS [PATTERN_OR_FILENAME]

You have your choice of using either short or long
RULE names, as described below. If you use a
short-named rule, the ',' separating the RULE
from the MODIFIERS is optional. The PATTERN or
FILE- NAME that follows (when present) must come
after either a single space or an underscore (_).
Here are the available rule prefixes:

merge, '.'
  specifies a merge-file to read for more rules.

So the "." means load rules from a file, and the "_"
is like an alternative to a space for separating the
"." and the filename. It seems that the space isn't
optional when using the short form, like the comma is.

I think this is rsync-specific.

I have seen programs use a convention of an "@" prefix
to an option argument to indicate that the rest of the
option argument is a filename to read the real option
arguments from. And I have seen libraries that provide
command line option functionality with support for
config files for default options.

A general purpose way of getting command line options
from a file, if you have a shell available, is to store
command line options in a file like this:

  --optiona ...
  --optionb ...
  --optionc ...

And then use them with on the command line like this:

  cmd `cat cmd-options` args...

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Copying TBs -> error -> work around

[raf via rsync]

On Fri, Sep 11, 2020 at 10:53:14AM +1000, Philip Rhoades via rsync 
 wrote:

> Roland,
> 
> 
> On 2020-09-10 21:27, Roland wrote:
> > > with rsync hanging - after breakout on /home for writing I then get:
> > > "Read-only file system"
> > 
> > if your filesystem switches to read-only, you have a serious problem
> > with your system/storage, not with rsync.
> > 
> > rsync (or the workload) is simply triggering the problem.
> 
> 
> Thanks for the response . .
> 
> Hmm . . but the drive that goes read-only is being read FROM not TO . . it
> is hard to see how that should be an issue?
> The backstory is that a relatively recent internal 8TB Seagate Barracuda had
> its 7.2TB sda5 (home) partition corrupted - which itself was suspicious but
> not impossible of course - so I had to switch temporarily to an external USB
> 4TB drive (which was a backup drive and was already up-to-date) for /home.
> So now this exercise is rsyncing back to a NEW internal 8TB Seagate
> Barracuda (sda5 again) . .
> 
> If you are correct about rsync simply triggering an existing problem on the
> 4TB USB drive, would that problem going to be recognised by a fsck (ext4)?
> I will check this out after I switch over to the new internal sda5 for
> /home.
> 
> Thanks,
> Phil.

file systems can be remounted read only when there are
too many errors. perhaps that applies to read errors as
well, not just write errors. check logs for i/o errors.
if it were i/o errors that caused the kernel to remount
the file system read only, it should have logged those
errors. and you should be able to use fsck with a usb
drive.

cheers,
raf

> > regards
> > roland
> > 
> > 
> > Am 10.09.20 um 07:30 schrieb Philip Rhoades via rsync:
> > > People,
> > > 
> > > When I did:
> > > 
> > >   rsync -av /home/ /mntb5/  # about 4TB
> > > 
> > > I got errors like:
> > > 
> > >   'rsync [sender] expand file_list pointer array to xxx bytes, "did
> > > move"'
> > > 
> > > with rsync hanging - after breakout on /home for writing I then get:
> > > 
> > >   "Read-only file system"
> > > 
> > > So after unmounting and remounting /home I did:
> > > 
> > >   cd /home
> > >   find /home/ -type d | sort > ./home_dirs_sorted.txt
> > > 
> > > delete first line "/home/" of ./home_dirs_sorted.txt then:
> > > 
> > >   while read dir ; do echo $dir ; rsync -lptgod "$dir" /mntb5/"$dir" ;
> > > done < ./home_dirs_sorted.txt
> > > 
> > > and:
> > > 
> > >   while read dir ; do echo $dir ; rsync -lptgoD "$dir"/\.[a-zA-Z0-9]*
> > > /mntb5/"$dir"/ ; done < ./home_dirs_sorted.txt
> > > 
> > > and finally with no problems:
> > > 
> > >   rsync -av --exclude-from=/usr/local/bin/nfb_caches.txt /home/
> > > /mntb5/
> > > 
> > > If there was a more sensible / efficient way of getting this done I
> > > would like to know about it!
> > > 
> > > Thanks,
> > > 
> > > Phil.
> > > 
> 
> -- 
> Philip Rhoades
> 
> PO Box 896
> Cowra  NSW  2794
> Australia
> E-mail:  p...@pricom.com.au
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: osx permission issue

[raf via rsync]

Henri Shustak via rsync wrote:

> If you are talking about SIP, then at present SSH will have full disk access 
> (as much as it can) by default.
> 
> Take a look at this LBackup page which discusses this in more detail : 
> http://www.lbackup.org/developer/dealing_with_sip
> 
> Hope that helps.

I think that that only applies to the sshd binary that
comes with macOS. If you install a more recent version
of openssh via macports or similar, it doesn't apply to
that. It certainly doesn't apply to a
macports-installed ssh client. In that case, you need
to grant full disk access to Terminal.app or similar,
and invoke the ssh client from there. It's not possible
to grant full disk access to an arbitrary binary
executable via System Preferences. You can only do that
for "applications". I'm not sure what aplication you
would need to grant full disk access to in order to give
a macports-installed sshd daemon full disk access.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsyncing via LAN deadly slow

[raf via rsync]

Stephane Ascoet via rsync wrote:

> > Is there iowait? If so, that might be the problem. Also,
> 
> Where can I see this information?

in the header section of top:

  top - 12:25:13 up 43 days, 22:31,  4 users,  load average: 0.18, 0.18, 0.12
  Tasks: 366 total,   1 running, 365 sleeping,   0 stopped,   0 zombie
  %Cpu(s):  3.2 us,  1.1 sy,  1.6 ni, 91.6 id,  2.5 wa,  0.0 hi,  0.0 si,  0.0 
st
  KiB Mem :  7905444 total,   623888 free,   584760 used,  6696796 buff/cache
  KiB Swap: 19543036 total, 18603192 free,   939844 used.  6345040 avail Mem

PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND

On the "%Cpu" line, where it says "wa", that's the percentage of
CPU time spent waiting for an I/O device. When it's high, things
are bad, and there will be processes below that show "D" in the
"S" column. Those processes are the ones waiting on the I/O.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsyncing via LAN deadly slow

[raf via rsync]

f27ae.9000...@free.fr>

Stephane Ascoet via rsync wrote:

> Hi, I'm backing up contents of my 
> GOBook(http://sascoet.mutu.fdn.fr/monalbum/albums/sa-12082010fujitsugigamos2300scsifleury-les-aubraisrecusdemonsauveurcomputermagneticsdmc-lx2/sa-1208201012h39fujitsugigamo2300scsiconnecteagobookfleury-les-aubraisrecusdemonsauveurcomputermagneticsdmc-lx2.jpg)
> before it dies completely on a dell laptop via my LAN(netstat indicates a
> 11.8MB/s speed between the two computers via this one).
> 
> On the GOBook I launch "rsync --daemon"(3.1.1-3 version)
> 
> On the dell: rsync -abSbHXz --progress(3.1.2-1 version)
> 
> CPUs are doing almost nothing. So does the GOBook HDD. The destination is on
> a mounted logical volume in Ext2 on a nvme SSD.
> 
> And the indicated speed transfer is around 50kB/s!!! I tried -W option: it's
> even slower. Same thing when I put off the "z"(anyway, CPUs aren't doing
> nothing).
> 
> If I netstat during the backup, it appears that there is about 4MB/s used,
> so almost 100 times more than the actual data written on destination! And
> why not 11MB/s since my network can do it?
> 
> I spent hours searching an explanation on the Web, nothing explaining what's
> happening in a sufficient satisfaying way for me. Does somenone on this list
> have a clue?
> 
> -- 
> Sincerely, Stephane Ascoet

Hi, I probably can't help. The only time I had rsync
being slow enough to bother me, it was being slowed
down by ssh's encryption, but that was a long time ago,
and you're not even using ssh. The only thing I can
suggest is that your network bandwidth is being used up
by something else (whatever is using up that 4MB/s) but
that sounds unlikely, and even so, there should still
be another 8MB/s to spare. Wireshark could help you
identify what the traffic is.

I just saw the picture. It looks super old. Perhaps the
problem is the disk I/O speed on the gobook itself. Is
that possible? I have no idea how old it is or what old
I/O speeds were like. Can you run top on the gobook? Is
there iowait? If so, that might be the problem. Also,
if the gobook is really old, its network interface
might only be 10Mbps, but even that should mean 1.25MB/s
(if the disk I/O speed can keep up).

My advice is to let it run now while it still can. :-)
Or you could try scp just to see if the transfer speed
is any different.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync to Network Share works from cmd but fails from Scheduled Task

[raf via rsync]

Hi Leon,

No worries. I'm glad you managed to get something working.

cheers,
raf

Leon Vanderploeg wrote:

>   Hi raf,
> 
> I was not aware the network maps were "user experience".  Thanks for
> your tips.  I have tried everything I can come up with, but cannot get it
> to work.  I figured out another solution, but thank you very much for your
> help.
> 
> Best,
> Leon
> 
> On Sun, Mar 29, 2020 at 9:23 PM raf via rsync  wrote:
> 
> > Leon Vanderploeg via rsync wrote:
> >
> > > Greetings,
> > >
> > > I have written a C program that runs on Windows servers and does overhead
> > > and reporting for backup administration.  The program calls rsync for the
> > > actual data copy function.  It copies data from a remote system to a
> > local
> > > drive and that works fine.  However, I have recently made changes to copy
> > > the data to a mapped drive on a NAS device instead of the local drive.
> > >
> > > The changes work fine when run from a command line, but fail when run
> > from
> > > a scheduled task or cron job:
> > > rsync: mkdir "/cygdrive/q/restores" failed: No such file or directory
> > > (2)
> > > rsync error: error in file IO (code 11) at main.c(675)
> > [Receiver=3.1.2]
> > > Note:  The folder already exists and permissions are open for
> > Administrator.
> > >
> > > Command line that works is bash command to call the C program under
> > > Administrator login.
> > > Scheduled task built as Administrator with or without highest privileges
> > > fails.
> > > Tried Cron entry built under Administrator and it fails.
> > >
> > > Here is the Rsync command (abbreviated) called by the C program:
> > > rsync  -rltgoD --chmod=Du+w,u+r,u+x -e "ssh -p 33222 -i
> > > \"/cygdrive/c/Users/administrator/.ssh/id_rsa\"" --stats
> > --files-from=/...
> > > -vv --exclude-from=/ @remote:/  /cygdrive/q/restores
> > >
> > > Note: q is the mapped drive to the NAS share.
> > >
> > > So, what can be done to track this down?  Why would running as a
> > scheduled
> > > task or cron job fail when command line works?  I am willing to do any
> > kind
> > > of tracing if someone can provide directions on how to do that.
> > >
> > > Thanks in advance,
> > > Leon
> >
> > Hi,
> >
> > I think the most likely reason is that when you execute
> > the command interactively, you are logged into Windows
> > locally and your cached credentials are used to connect
> > to the mapped drive. But that's not the case when the
> > task is scheduled, and so the mapped drive isn't mapped
> > when the scheduled task is attempted.
> >
> > Similarly, with Cygwin, you can only see
> > /cygdrive/q/... while you are accessing it locally. If
> > you ssh to a Windows host with Cygwin and sshd running,
> > you can't see any mapped drives that are accessible to
> > the local user, even if you ssh in as the same user.
> >
> > You will have to map the drive in the scheduled task
> > before invoking rsync, and then unmap the drive
> > afterwards. I don't know how to do that or handle the
> > credentials that would be needed, but I think that's
> > where to look.
> >
> > cheers,
> > raf
> >
> >
> > --
> > Please use reply-all for most replies to avoid omitting the mailing list.
> > To unsubscribe or change options:
> > https://lists.samba.org/mailman/listinfo/rsync
> > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> >

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync to Network Share works from cmd but fails from Scheduled Task

[raf via rsync]

Leon Vanderploeg via rsync wrote:

> Greetings,
> 
> I have written a C program that runs on Windows servers and does overhead
> and reporting for backup administration.  The program calls rsync for the
> actual data copy function.  It copies data from a remote system to a local
> drive and that works fine.  However, I have recently made changes to copy
> the data to a mapped drive on a NAS device instead of the local drive.
> 
> The changes work fine when run from a command line, but fail when run from
> a scheduled task or cron job:
> rsync: mkdir "/cygdrive/q/restores" failed: No such file or directory
> (2)
> rsync error: error in file IO (code 11) at main.c(675) [Receiver=3.1.2]
> Note:  The folder already exists and permissions are open for Administrator.
> 
> Command line that works is bash command to call the C program under
> Administrator login.
> Scheduled task built as Administrator with or without highest privileges
> fails.
> Tried Cron entry built under Administrator and it fails.
> 
> Here is the Rsync command (abbreviated) called by the C program:
> rsync  -rltgoD --chmod=Du+w,u+r,u+x -e "ssh -p 33222 -i
> \"/cygdrive/c/Users/administrator/.ssh/id_rsa\"" --stats --files-from=/...
> -vv --exclude-from=/ @remote:/  /cygdrive/q/restores
> 
> Note: q is the mapped drive to the NAS share.
> 
> So, what can be done to track this down?  Why would running as a scheduled
> task or cron job fail when command line works?  I am willing to do any kind
> of tracing if someone can provide directions on how to do that.
> 
> Thanks in advance,
> Leon

Hi,

I think the most likely reason is that when you execute
the command interactively, you are logged into Windows
locally and your cached credentials are used to connect
to the mapped drive. But that's not the case when the
task is scheduled, and so the mapped drive isn't mapped
when the scheduled task is attempted.

Similarly, with Cygwin, you can only see
/cygdrive/q/... while you are accessing it locally. If
you ssh to a Windows host with Cygwin and sshd running,
you can't see any mapped drives that are accessible to
the local user, even if you ssh in as the same user.

You will have to map the drive in the scheduled task
before invoking rsync, and then unmap the drive
afterwards. I don't know how to do that or handle the
credentials that would be needed, but I think that's
where to look.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: osx permission issue

[raf via rsync]

Hi,

I don't think it's about file flags. macOS has SIP
(System Integrity Protection) which, among other
things, restricts the ability of processes to access
certain parts of the file system.

Usually people ask about how to override this locally
which can be done by disabling SIP altogether, or for
file access only, adding the relevant app in:

  System Preferences -> Secutiy & Privacy -> Full Disk Access

But It's not great. I think you can only add "apps" to
the list, not arbitrary executables. I've added
Terminal, iTerm.app and XQuartz.app but it doesn't work
for XQuartz/X11 for some reason, so I need to rsync
some parts from the file system from Terminal/iTerm
instead of from an xterm under XQuartz.

But the OP is asking about the remote end of the
connection. If you are allowed to manipulate the
security settings on the remote end, you could apply
the above ideas there but you'd need to know which app
needs to be added to the list so that rsync can
function. It's not obvious. It's probably the app that
starts sshd. It might be launchd. I can't imagine that
giving that full disk access is a great idea.

It might be possible to get the remote sshd to run the
remote side of rsync via an intermediary app that can
be added to the list of apps with full disk access,
such as Automator.

Maybe you could use (via rsync) ssh's -e / --rsh option
to specify what's needed. See this page for ideas:

  How to Give Full Disk Access to a Binary in MacOS Mojave
  
https://n8henrie.com/2018/11/how-to-give-full-disk-access-to-a-binary-in-macos-mojave/

I haven't read it, just googled "full disk access launchd"
so not sure it'll help.

good luck,
raf

Perry Hutchison via rsync wrote:

> Roland via rsync  wrote:
> > does somebody know how to circumvent that "extra file access restriction
> > feature" introduced in osx some time ago ?
> 
> It may not be possible.
> 
> Based on experience with FreeBSD, from which much of OSX is derived,
> I suspect you may be running into issues with "file flags"; check the
> OSX documentation for the chflags system call.  In FreeBSD, I've seen
> that same error message when rsync attempts an operation that would
> violate chflags restrictions.
> 
> > i already tried adding rsync binary to programms with "full disk access"
> > privilege ( system-prefs -> security & privacy -> privacy -> full disk
> > access)?? , since running as root is not sufficient - but it does not work.
> 
> "full disk access" likely refers to the ability to read the disk
> directly, bypassing the filesystem.  It's used by maintenance
> programs like fsck and fsdb, and by some full-disk backup programs,
> but it won't help rsync.
> 
> > i want to make sure that every file on osx is getting backup
> >
> > i run rsync on linux to remotely backup osx system (via ssh).
> >
> > regards
> > roland
> >
> > rsync:
> > readlink_stat("/private/var/folders/_p/ky_w_lyj6ps7jcnkjkl5ss0mgn/0/com.apple.routined")
> > failed: Operation not permitted (1)
> > ...
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Question/comment about -n (dry run) flag of rsync

[raf via rsync]

raf via rsync wrote:

> T. Shandelman via rsync wrote:
> 
> > Rsync is a remarkably handy tool that I use virtually every day.
> > 
> > But there is one thing about rsync that drives me totally crazy.
> > 
> > Under the -n (dry run) flag, rsync seems to produce exactly the same output
> > as without that flag.
> > 
> > I cannot tell you how many times I sit and scratch my head long and
> > hard,after I discover that my intended rsync operations did not actually
> > happen. Until I finally remember that I ran rsync in dry-run mode. That's
> > why! This is especially a problem for very long-running rsync jobs run in
> > dry-run mode.
> > 
> > It seems to me that when run in dry-run mode, rsync should display a
> > warning at the very, very end, something like:
> > 
> > *WARNING: None of the above operations have been actually performed, *
> > *because you ran rsync in dry-run mode.*
> > 
> > Or does rsync already have such a feature, and I am not aware of it?
> > 
> > But if not, that is my vote for the next feature to be added. It should be
> > a very, very easy fix.
> > 
> > Todd S.
> > Austin, Texas, USA
> 
> Bear in mind that it shouldn't happen by default as
> that could break scripts that parse rsync's output. You
> should need to request this behaviour in the config
> file somehow.

Oops. Please ignore that.


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Question/comment about -n (dry run) flag of rsync

[raf via rsync]

T. Shandelman via rsync wrote:

> Rsync is a remarkably handy tool that I use virtually every day.
> 
> But there is one thing about rsync that drives me totally crazy.
> 
> Under the -n (dry run) flag, rsync seems to produce exactly the same output
> as without that flag.
> 
> I cannot tell you how many times I sit and scratch my head long and
> hard,after I discover that my intended rsync operations did not actually
> happen. Until I finally remember that I ran rsync in dry-run mode. That's
> why! This is especially a problem for very long-running rsync jobs run in
> dry-run mode.
> 
> It seems to me that when run in dry-run mode, rsync should display a
> warning at the very, very end, something like:
> 
> *WARNING: None of the above operations have been actually performed, *
> *because you ran rsync in dry-run mode.*
> 
> Or does rsync already have such a feature, and I am not aware of it?
> 
> But if not, that is my vote for the next feature to be added. It should be
> a very, very easy fix.
> 
> Todd S.
> Austin, Texas, USA

Bear in mind that it shouldn't happen by default as
that could break scripts that parse rsync's output. You
should need to request this behaviour in the config
file somehow.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [draft PATCH] whitelist support for refuse options

[raf via rsync]

Hi Nick,

Just in case you aren't already aware of it, there is a tool
that limits rsync to particular directories:

  rrsync - Restricts rsync to subdirectory declared in .ssh/authorized_keys
  https://www.samba.org/ftp/unpacked/rsync/support/rrsync

There is also a generic tool (I wrote this one) for limiting
incoming ssh commands to a fixed list of specific actual
commands (with semi-automatic learning of commands based on
observed behaviour during training mode):

  sshdo - controls which commands may be executed via incoming ssh
  http://raf.org/sshdo
  https://github.com/raforg/sshdo

And another (less easy to use) generic alternative:

  authprogs - SSH Command Authenticator
  https://github.com/daethnir/authprogs

I don't know if any of the above commands would give you
what you want but they might.

Suggestion: If the above won't do what you need, I'd suggest
different syntax. I don't like a directive that starts with
"refuse options" and then is mostly followed by the options
that are allowed. It's really saying refuse all options except
the following. I think it would be more intuitive if it
looked like:

  allowed options = verbose archive

and the presence of "allowed options" in rsyncd.conf causes
all other options (not present in that or any other "allowed
options" directive) to be disallowed. Just a thought.

A potential problem with your approach I imagine is that it
would apply equally to all local users but different users
might have different use cases. The tools mentioned above
would support handling different users distinctly.

cheers,
raf

Nick Cleaton via rsync wrote:

> This adds support for whitelisting the acceptable options in the
> "refuse options" setting in rsyncd.conf. It introduces "!" as a
> special option string that refuses most options and interprets
> any following strings as patterns of options to allow.
> 
> For example, to allow only verbose and archive:
> 
>   refuse options = ! verbose archive
> 
> The "!" does't refuse no-iconv, but you can still refuse it and
> use a whitelist if you want:
> 
>   refuse options = no-iconv ! verbose archive
> 
> It's not finished (needs tests and doc) I just wanted to see if
> there'd be any interest in merging something of this shape
> before I put more work into it.
> 
> My use case is setting up a restricted trust relationship by
> allowing host A to ssh to host B with a forced command of
> "rsync --server --daemon --config=/path/to/rsyncd.conf ." and
> configuring the restictions in rsyncd.conf. I know what options
> I want to use, it'd be nice to enforce that on the server side
> without listing every other option in "refuse options".

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Stale pid file problem, and a proposed solution

[raf via rsync]

Joseph C. Sible via rsync wrote:

> Today, rsyncd manages its pid file by open()ing it with O_CREAT|O_EXCL
> at startup, and then unlink()ing it at shutdown. If the open() fails
> at startup because the file already exists, then rsyncd will assume
> another instance of itself is already running and not start.
> 
> However, there's a problem with this approach: if rsyncd is terminated
> without being able to clean up (e.g., kill -9, or the server losing
> power), then the stale pid file will prevent rsyncd from ever
> restarting until an administrator manually intervenes.
> 
> I propose a solution to this problem: open the file without O_EXCL,
> then try to take an exclusive lock on the whole file (we already use
> file locks to limit max connections, so this change wouldn't add any
> new requirements to rsyncd). If we can't get the lock, then abort, and
> if we can, then truncate the file and write our PID into it. Since
> locks never outlive the process that took them, this fixes the stale
> pid file problem.
> 
> Does this seem like a reasonable idea? If so, I'll write and submit a
> patch that implements it.
> 
> Joseph C. Sible

I think that's very sensible. It's what my daemon program does
(libslack.org/daemon) to ensure a single instance of a daemon.
It probably means that the pidfile shouldn't be on an NFS-mounted
file system but hopefully that won't be a problem for anyone.
Or could that be a problem?

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Removing folder at destination

[raf via rsync]

Fabian Cenedese via rsync wrote:

> >Why not exclude 2_ASSETS from the first run, and then do a second run to
> >rsync the _contents_ of 2_ASSETS to the target 01_us_eng directory?
> >
> >Add /01_us_eng/2_ASSETS/ to the exclude.txt file.
> >
> >rsync -rvz --exclude-from=exclude.txt source destination
> >rsync -rvz source/01_us_eng/2_ASSETS/ destination/01_us_eng/
> >
> >(perhaps add some variation of the exclude.txt file if you're excluding
> >stuff under the 2_ASSETS directory)
> >
> >> My humble question to samba.org is this, can I add an rsync command to move
> >> folders 3 and 4 into 01... and delete the 2_ASSETS folder so I dont have to
> >> manually do this 200 times each day?
> >
> >You can't use rsync to perform remote rename operations, which is what
> >you're essentially asking for.
> 
> You can execute remote commands using ssh/rsh. If the names
> stay the same then you can add the rename commands to a script
> file being executed after the rsync run.
> 
> bye  Fabi

But if you do that, subsequent uses of rsync won't work
any more efficiently than the first one did. It would
have to re-copy the files that have not kept their
original names. That's probably why rsync doesn't do
renaming operations.

You might be better off renaming or moving things at
the source and rsyncing that. If that's not an option,
and the source and destination really need to differ,
you could construct the desired directory layout using
symlinks to the originals, and rsync the alternate
symlink version.

I've done a similar thing when rsyncing files
originally from a macOS host whose names contained
colon characters (representing forward slashes) to a
Windows host where colon characters aren't allowed in
file names. A script would construct the
Windows-friendly view of the files, then rsync that,
then delete the Windows-friendly view afterwards.

That won't work if you need to rsync any symlinks as
symlinks but otherwise, it should be OK.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Speed up rsync with many of excluded files

[raf via rsync]

raf via rsync wrote:

> Georg Limbach via rsync wrote:
> 
> > Hello,
> > 
> > I have a problem while rsyncing a directory with perhaps 1.000.000
> > files. Before this I generate some files locally and add them to temp
> > files. So this files should be excluded from syncing.
> > 
> > My rsync call looks like this:
> > 
> > rsync --exclude='tmp/' \
> > --exclude-from=/tmp/tmp.GF7SsFPnS3 \
> > --exclude-from=/tmp/tmp.8SjJNCHyaI \
> > --exclude-from=/tmp/tmp.CxZXEoPjgV \
> > --exclude-from=/tmp/tmp.G3g2iMo4bs \
> > --exclude-from=/tmp/tmp.H9KJYPMfMS \
> > --exclude-from=/tmp/tmp.PNi7cJaREP \
> > --exclude-from=/tmp/tmp.S4N9H4lsU7 \
> > --exclude-from=/tmp/tmp.a5Zlgh6pUK \
> > --exclude-from=/tmp/tmp.eiUlMluAe8 \
> > --exclude-from=/tmp/tmp.ma0S1YSewc \
> > --exclude-from=/tmp/tmp.sLR95oVbVD \
> > --exclude-from=/tmp/tmp.zbfeLpezMX \
> > -ax --info=progress2 \
> > -e 'ssh -x -T -o Compression=no' \
> > '/srv/project/shared/uploads/' \
> > 'project@host:/srv/project/shared/uploads/'
> > 
> > In every temp file are 1000s of lines with files like this:
> > 
> > orders/scan/file/2234480/scan.pdf
> > 
> > When I started rsync it only should transfer perhaps 1000 files and
> > delete 20 ones. But it takes 15 minutes to do that. I think the big
> > count of excluded files tear down the compare speed of files.
> > 
> > What can I do to speed up this process?
> > 
> > Thanks for your advise!
> > 
> > Georg
> 
> I found a big speed up when I starting providing a list of
> candidate files to rsync with the --files-from option.
> If you have some way of identifying which files might need
> to be rsynced that's quicker than rsync itself checking
> everything, it can make a big difference. That's not why
> I did it, but it was a nice bonus.

But it might not help if doing --delete at the same
time. Can the --delete be done occasionally instead of
every time? Unless the files to be deleted can be added
to the list of files handed to rsync. I don't know if
that'll do what I think but it might.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Speed up rsync with many of excluded files

[raf via rsync]

Georg Limbach via rsync wrote:

> Hello,
> 
> I have a problem while rsyncing a directory with perhaps 1.000.000
> files. Before this I generate some files locally and add them to temp
> files. So this files should be excluded from syncing.
> 
> My rsync call looks like this:
> 
> rsync --exclude='tmp/' \
> --exclude-from=/tmp/tmp.GF7SsFPnS3 \
> --exclude-from=/tmp/tmp.8SjJNCHyaI \
> --exclude-from=/tmp/tmp.CxZXEoPjgV \
> --exclude-from=/tmp/tmp.G3g2iMo4bs \
> --exclude-from=/tmp/tmp.H9KJYPMfMS \
> --exclude-from=/tmp/tmp.PNi7cJaREP \
> --exclude-from=/tmp/tmp.S4N9H4lsU7 \
> --exclude-from=/tmp/tmp.a5Zlgh6pUK \
> --exclude-from=/tmp/tmp.eiUlMluAe8 \
> --exclude-from=/tmp/tmp.ma0S1YSewc \
> --exclude-from=/tmp/tmp.sLR95oVbVD \
> --exclude-from=/tmp/tmp.zbfeLpezMX \
> -ax --info=progress2 \
> -e 'ssh -x -T -o Compression=no' \
> '/srv/project/shared/uploads/' \
> 'project@host:/srv/project/shared/uploads/'
> 
> In every temp file are 1000s of lines with files like this:
> 
> orders/scan/file/2234480/scan.pdf
> 
> When I started rsync it only should transfer perhaps 1000 files and
> delete 20 ones. But it takes 15 minutes to do that. I think the big
> count of excluded files tear down the compare speed of files.
> 
> What can I do to speed up this process?
> 
> Thanks for your advise!
> 
> Georg

I found a big speed up when I starting providing a list of
candidate files to rsync with the --files-from option.
If you have some way of identifying which files might need
to be rsynced that's quicker than rsync itself checking
everything, it can make a big difference. That's not why
I did it, but it was a nice bonus.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Seemingly impossible bug: -v not always listing every copied file

[raf via rsync]

Unfortunately, I can't replicate the problem at will.
But I'll investigate using iwatch/inotify to log
everything from the file system's point of view.
That should shed some light on what's happening.

The file names don't contain any control characters
(but good thinking!).

thanks,
raf

Joe via rsync wrote:

> If you are doing a small test run that duplicates the problem, you could
> use inotifywait or tail -f to watch the log in real time on another
> terminal. Maybe you'd see something like a line being overwritten, etc.

> I don't know what rsync would do with a file name that ends in a
> carriage return (not a newline). It's probably smart enough to handle
> it, but if it isn't, the current log line might be overwritten by the
> next one.
> 
> Joe
> 
> On 10/30/19 1:24 AM, raf via rsync wrote:
> > Thanks. I'll try that. But I agree that it'll be
> > something else. It's unlikely that whole trace files
> > are being overwritten, because there's locking in place
> > to prevent that sort of thing, but it's more likely
> > than anything else. I'll check that the locking is
> > working properly.
> >
> > Although, when I last investigated it, it really did
> > look as though, in a single run of this task, three
> > files were copied for one directory but only two were
> > listed. I need to verify that more thoroughly, perhaps
> > with directory listings on both sides before and after
> > each run.
> >
> > cheers,
> > raf
> >
> > Kevin Korb via rsync wrote:
> >
> >> It does seem impossible.  I would suggest adding --itemize-changes (-v
> >> isn't really all that useful without it anyway).  If entries are still
> >> missing then I would suspect that either log files are missing (maybe
> >> duplicate file names replacing the occasional log file?) or something
> >> other than rsync is doing things in the same dir.
> >>
> >> On 10/29/19 9:00 PM, raf via rsync wrote:
> >>> Hi,
> >>>
> >>> debian-9, rsync-3.1.2 (both ends)
> >>>
> >>> I have a task that rsyncs files from a list of
> >>> candidate files (--files-from=). It's verbose (-v) and
> >>> its stdout is captured to a file which is then sent to
> >>> the receiving host. The captured verbose output is
> >>> examined on the receiving host to know which files were
> >>> actually copied so that notification emails can be sent
> >>> to various people.
> >>>
> >>> The problem is that, sometimes, not all copied files
> >>> are listed in the verbose output, and so some
> >>> notification emails don't get sent out. At first, I
> >>> thought there was something wrong with the notification
> >>> emails not arriving, but the files in question, that
> >>> had definitely been copied, did not appear in any of
> >>> the captured verbose output files.
> >>>
> >>> This seems like an impossible bug but it really seems
> >>> to be happening. Has anyone else encountered behaviour
> >>> like this? I didn't have much luck searching the
> >>> internet for it. It would probably be hard to notice
> >>> if the verbose output wasn't being used for something
> >>> like triggering notifications whose absence might be
> >>> noticed.
> >>>
> >>> cheers,
> >>> raf
> >> -- 
> >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >>Kevin Korb  Phone:(407) 252-6853
> >>Systems Administrator   Internet:
> >>FutureQuest, Inc.   ke...@futurequest.net  (work)
> >>Orlando, Floridak...@sanitarium.net (personal)
> >>Web page:   https://sanitarium.net/
> >>PGP public key available on web site.
> >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >>
> >
> >
> >
> >> -- 
> >> Please use reply-all for most replies to avoid omitting the mailing list.
> >> To unsubscribe or change options: 
> >> https://lists.samba.org/mailman/listinfo/rsync
> >> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> >
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Seemingly impossible bug: -v not always listing every copied file

[raf via rsync]

The complete rsync command at the client end is approximately:

  rsync --files-from=/tmp/rptaudit.list.4958 -zltv --compress-level=9 \
  -e 'ssh' --rsync-path 'umask 002; rsync' --chmod=D2770,F640 --exclude=archive 
\
  ./ user@host:/var/thing/report >> /var/log/rpt2web/rpt2web.trace

cheers,
raf

Paul Slootman via rsync wrote:

> On Wed 30 Oct 2019, raf via rsync wrote:
> > 
> > I have a task that rsyncs files from a list of
> > candidate files (--files-from=). It's verbose (-v) and
> 
> It would be helpful to show the complete rsync command line.
> 
> Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Seemingly impossible bug: -v not always listing every copied file

[raf via rsync]

Kyle Bassett via rsync wrote:

> Hi raf,
> 
> Curious issue you have.  A few things:
> 
> What distro(s) are you using?

debian-9

> Same rsync version on both ends?

Yes, rsync-3.1.2

> Hash of files look correct before and after the rsync?

I didn't check but I expect so. I don't expect rsync to
ever do the wrong thing. :-)

> Have you tried using inotify to monitor for changes at the fs level?  You
> should see a "read" on the sender and a "read" + "write" on the receiver.

No, but that's a good idea. I'll investigate that.

> On Tue, Oct 29, 2019, 11:25 PM raf via rsync  wrote:
> 
> > Thanks. I'll try that. But I agree that it'll be
> > something else. It's unlikely that whole trace files
> > are being overwritten, because there's locking in place
> > to prevent that sort of thing, but it's more likely
> > than anything else. I'll check that the locking is
> > working properly.
> >
> > Although, when I last investigated it, it really did
> > look as though, in a single run of this task, three
> > files were copied for one directory but only two were
> > listed. I need to verify that more thoroughly, perhaps
> > with directory listings on both sides before and after
> > each run.
> >
> > cheers,
> > raf
> >
> > Kevin Korb via rsync wrote:
> >
> > > It does seem impossible.  I would suggest adding --itemize-changes (-v
> > > isn't really all that useful without it anyway).  If entries are still
> > > missing then I would suspect that either log files are missing (maybe
> > > duplicate file names replacing the occasional log file?) or something
> > > other than rsync is doing things in the same dir.
> > >
> > > On 10/29/19 9:00 PM, raf via rsync wrote:
> > > > Hi,
> > > >
> > > > debian-9, rsync-3.1.2 (both ends)
> > > >
> > > > I have a task that rsyncs files from a list of
> > > > candidate files (--files-from=). It's verbose (-v) and
> > > > its stdout is captured to a file which is then sent to
> > > > the receiving host. The captured verbose output is
> > > > examined on the receiving host to know which files were
> > > > actually copied so that notification emails can be sent
> > > > to various people.
> > > >
> > > > The problem is that, sometimes, not all copied files
> > > > are listed in the verbose output, and so some
> > > > notification emails don't get sent out. At first, I
> > > > thought there was something wrong with the notification
> > > > emails not arriving, but the files in question, that
> > > > had definitely been copied, did not appear in any of
> > > > the captured verbose output files.
> > > >
> > > > This seems like an impossible bug but it really seems
> > > > to be happening. Has anyone else encountered behaviour
> > > > like this? I didn't have much luck searching the
> > > > internet for it. It would probably be hard to notice
> > > > if the verbose output wasn't being used for something
> > > > like triggering notifications whose absence might be
> > > > noticed.
> > > >
> > > > cheers,
> > > > raf
> > >
> > > --
> > > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> > >   Kevin Korb  Phone:(407) 252-6853
> > >   Systems Administrator   Internet:
> > >   FutureQuest, Inc.   ke...@futurequest.net  (work)
> > >   Orlando, Floridak...@sanitarium.net (personal)
> > >   Web page:   https://sanitarium.net/
> > >   PGP public key available on web site.
> > > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Seemingly impossible bug: -v not always listing every copied file

[raf via rsync]

Thanks. I'll try that. But I agree that it'll be
something else. It's unlikely that whole trace files
are being overwritten, because there's locking in place
to prevent that sort of thing, but it's more likely
than anything else. I'll check that the locking is
working properly.

Although, when I last investigated it, it really did
look as though, in a single run of this task, three
files were copied for one directory but only two were
listed. I need to verify that more thoroughly, perhaps
with directory listings on both sides before and after
each run.

cheers,
raf

Kevin Korb via rsync wrote:

> It does seem impossible.  I would suggest adding --itemize-changes (-v
> isn't really all that useful without it anyway).  If entries are still
> missing then I would suspect that either log files are missing (maybe
> duplicate file names replacing the occasional log file?) or something
> other than rsync is doing things in the same dir.
> 
> On 10/29/19 9:00 PM, raf via rsync wrote:
> > Hi,
> > 
> > debian-9, rsync-3.1.2 (both ends)
> > 
> > I have a task that rsyncs files from a list of
> > candidate files (--files-from=). It's verbose (-v) and
> > its stdout is captured to a file which is then sent to
> > the receiving host. The captured verbose output is
> > examined on the receiving host to know which files were
> > actually copied so that notification emails can be sent
> > to various people.
> > 
> > The problem is that, sometimes, not all copied files
> > are listed in the verbose output, and so some
> > notification emails don't get sent out. At first, I
> > thought there was something wrong with the notification
> > emails not arriving, but the files in question, that
> > had definitely been copied, did not appear in any of
> > the captured verbose output files.
> > 
> > This seems like an impossible bug but it really seems
> > to be happening. Has anyone else encountered behaviour
> > like this? I didn't have much luck searching the
> > internet for it. It would probably be hard to notice
> > if the verbose output wasn't being used for something
> > like triggering notifications whose absence might be
> > noticed.
> > 
> > cheers,
> > raf
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   https://sanitarium.net/
>   PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 




> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Seemingly impossible bug: -v not always listing every copied file

[raf via rsync]

Hi,

debian-9, rsync-3.1.2 (both ends)

I have a task that rsyncs files from a list of
candidate files (--files-from=). It's verbose (-v) and
its stdout is captured to a file which is then sent to
the receiving host. The captured verbose output is
examined on the receiving host to know which files were
actually copied so that notification emails can be sent
to various people.

The problem is that, sometimes, not all copied files
are listed in the verbose output, and so some
notification emails don't get sent out. At first, I
thought there was something wrong with the notification
emails not arriving, but the files in question, that
had definitely been copied, did not appear in any of
the captured verbose output files.

This seems like an impossible bug but it really seems
to be happening. Has anyone else encountered behaviour
like this? I didn't have much luck searching the
internet for it. It would probably be hard to notice
if the verbose output wasn't being used for something
like triggering notifications whose absence might be
noticed.

cheers,
raf


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html